Gradient

In multivariable calculus, assuming a function $f$ is differentiable at the point $(x,y)$, the gradient of a function $f$ is the vector field of the partial derivatives, denoted with $\nabla$:

$$\nabla f(x,y)=⟨\frac{\partial f}{\partial x},\frac{\partial f}{\partial y},\frac{\partial f}{\partial z}⟩=f_x\hat{i}+f_y\hat{j}+f_z\hat{k}$$

The directional derivative of $f$ at $(a,b)$ in the direction of the unit vector can be written:

$$D_{\text{u}}f(a,b)=\nabla f(a,b)\cdot \text{u}=|\nabla f(a,b)|\cos \theta$$

Properties

Like regular derivatives, the gradient satisfies sum, product, and quotient rules and is distributive.

$$\nabla (uv)=u\nabla (v)+v\nabla (u)$$

The gradient generalises past ${\mathbb{R}}^3$ to:

$$\nabla f(\vec{x})=\frac{1}{n}\sum _{i=1}^n\nabla f_i(\vec{x})$$

i.e., it is the average of each gradient term. One key problem with this: if we use gradient descent, the time complexity of computing the gradient for each independent variable iteration is $O(n)$, which can be prohibitively expensive for large deep learning datasets.

Interpretations

Just looking at how the gradient interacts with the directional derivative:

• When $\theta =0$ and the gradient and unit vector point in the same direction, the directional derivative has its maximum value, and $f$ has its greatest rate of increase.
• When $\theta =\pi$ and both point in opposite directions, $f$ has its greatest rate of decrease and the directional derivative has its minimum value.
• The directional derivative is zero in any direction orthogonal to the gradient vector.

So what this means is that the gradient points in the direction of steepest ascent at $(a,b)$. The negative gradient points in the direction of steepest descent.

By theorem, the line tangent to the level curve of $f$ at $(a,b)$ is orthogonal to the gradient at the point provided $\nabla f(a,b)\ne 0$.

See also

• Gradient descent