Introduction to basic concepts in probability theory. Pre-requisite to many upper-year machine learning courses. We used the Probability, Statistics, and Random Processes for Electrical Engineering textbook by Alberto Leon-Garcia.
Concepts covered
- Basic probability theory
- Mutual exclusivity
- Events
- Basic set operations (, , ) and De Morgan’s laws
- Finite and infinite sets (sample spaces)
- Tree diagrams
- Conditional probability (Bayes’ theorem)
- Classification metrics (accuracy)
- Combinatorics
- Rule of products, rule of sums
- Sampling {with, without} replacement, {with, without} ordering
- Permutations
- Combinations
- Markov chain
- Random variable
- Functions of a random variable
- Multiple random variables
- Joint PMFs, PDFs, CDFs
- Joint moments
- Correlation
- Covariance
- Vector random variable
- Functions of multiple random variables
- Inequalities
- Probability distribution
- Probability mass function (PMF)
- Probability density function (PDF)
- Cumulative distribution function (CDF)
- Discrete probability distributions
- Bernoulli distribution
- Bernoulli trials
- Binomial distribution
- Binomial probability law
- Binomial theorem
- Geometric distribution
- Poisson distribution
- Zipf distribution
- Bernoulli distribution
- Continuous probability distributions
- Metrics
- Transform methods
- Information theory
- Shannon entropy (and uncertainty)
- Huffman compression
- Estimation
- Maximum a posteriori estimation
- Maximum likelihood estimation
- Mean squared error estimation
- Machine learning
- Law of large numbers
- Central limit theorem