Adaptive moment estimation

Adaptive moment estimation (Adam) is an efficient optimisation algorithm for deep learning. It builds on stochastic gradient descent by using both momentum and RMS prop.

$$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)\left(\frac{\partial E}{\partial w_{ji}}\right)$$ $$v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)(\frac{\partial E}{\partial w_{ji}}{)}^2$$ $$w_{ji}^{t+1}=w_{ji}^t-\frac{\gamma }{{\sqrt{v}}_t+ϵ}{m}_t$$

In code

In PyTorch, we access Adam with the torch.optim.Adam() function.