Regularisation

In statistical learning. regularisation is “any modification [made] to a learning algorithm that is intended to reduce its generalisation error but not its training error”.¹ Regularisation is an important objective of ML, as important as optimisation.

Some key ideas:

• Weight decay (or ${\mathsf{L}}^2$ regularisation) prevents weights from growing too much.
• Neuron dropout forces a neural network to learn more robust features.

Footnotes

1. From Deep Learning, by Goodfellow, Bengio, Courville, and Bach. ↩