ECE302 — Probability and Applications

Introduction to basic concepts in probability theory. Pre-requisite to many upper-year machine learning courses.

We used the Probability, Statistics, and Random Processes for Electrical Engineering textbook, by Alberto Leon-Garcia.

Concepts covered

Basic probability theory
- Mutual exclusivity
- Events
- Basic set operations ( $\land$ , $\lor$ , $\$ ) and De Morgan’s laws
- Finite and infinite sets (sample spaces)
- Tree diagrams
- Conditional probability
  - Bayes’ theorem
- Classification metrics (accuracy)
Combinatorics
- Rule of products, rule of sums
- Sampling {with, without} replacement, {with, without} ordering
- Permutations
- Combinations
Markov chain
Random variable
- Functions of a random variable
- Multiple random variables
  - Joint PMFs, PDFs, CDFs
  - Joint moments
  - Correlation
    - Pearson correlation coefficient
  - Covariance
- Functions of multiple random variables
- Vectors of $n$ random variables
- Inequalities
  - Markov’s inequality
  - Chebyshev’s inequality
Probability distribution
- Probability mass function (PMF)
- Probability density function (PDF)
- Cumulative distribution function (CDF)
- Discrete probability distributions
  - Bernoulli distribution
    - Bernoulli trials
  - Binomial distribution
    - Binomial probability law
    - Binomial theorem
  - Geometric distribution
  - Poisson distribution
  - Zipf distribution
- Continuous probability distributions
Metrics
- Expected value (mean)
- Variance
- Standard deviation
Transform methods
Shannon entropy (and uncertainty)
- Huffman compression
Estimation
- Maximum a posteriori estimation
- Maximum likelihood estimation
- Mean-squared error estimation
Inequalities