KL divergence

The Kullback-Leibler divergence (KL divergence) is able to measure the difference between two probability distributions $P(X)$ and $Q(X)$.

$$D_{KL}(P\Vert Q)=\sum _{x\in X}p(x)\log \left(\frac{p(x)}{q(x)}\right)$$

Cross entropy is a special case of KL divergence.

For two multivariable Gaussian distributions, we get:

$$D_{KL}(p|q)=\frac{1}{2}\sum _{i=1}^N[{\mu }_i^2+{\sigma }_i^2-(1+\log ({\sigma }_i^2))]$$