Optimiser

In deep learning, optimisers are used to determine how each parameter (i.e., the weights) in a model should change based on the value of the loss function.

Many common NN optimisers are based on gradient descent.

Types

• Stochastic gradient descent
• Adaptive moment estimation (Adam)

In code

During model training, we should define our optimiser:

optimiser = optim.fn(model.parameters(), lr, ...)

PyTorch has several optimiser functions contained within the torch.optim subclass.