neural nets a single decision boundary is not enough to solve many problems, i.e., we shouldn’t have a single NN layer → XOR function needs two boundaries to solve
at least one hidden NN layer is used in between the input and output layers the limit of an infinitely wide NN with at least one hidden layer → NN as a universal function approximator
Discriminative models
Discriminative models are able to learn the decision boundary between different classes. They’re specifically meant for classification tasks, and maximise the conditional probability . Oftentimes this means an input image, and figuring out the output label.
Given an input , maximise the probability of label .
Many machine learning models are explicitly meant to learn the decision boundary:
- Logistic regression
- Random forests
- Decision tree
- Traditional MLPs
These are contrasted with generative models. Discriminative models don’t possess generative properties, but the vice versa is true. Generative models learn the underlying probability distribution.