Cross entropy

Cross entropy (CE) is an error function mostly used for classification problems. It’s given by:

$$\text{CE}=-\frac{1}{N}\sum _{n=1}^N\sum _{k=1}^K{t}_{n,k}\log (y_n,k)$$

where:

• $N$ is the number of training samples.
• $K$ is the number of classes.
• $t_{n,k}$ is the ground truth label.
• $y_{n,k}$ is the prediction.

Does the $\log$ base matter? No, not really. Historically base 2 has been used because of origins in information theory, where information is stored in bits and bytes.

This is actually just negative log likelihood if we divide by the number of samples.

We can implement this in PyTorch with torch.nn.CrossEntropy().

Binary cross-entropy

For a Bernoulli distribution, i.e., where we have a binary classification problem, we can use a specialised case of CE called binary cross entropy (BCE).

$$\text{BCE}=-\frac{1}{N}\sum _{n=1}^N[t_n\log (y_n)+(1-t_n)\log (1-y_n)]$$

where the term $t_n$ and $(1-t_n)$ relate to the corresponding class in the Bernoulli distribution.

In PyTorch, we use torch.nn.BCELoss(). A variation on this is torch.nn.BCEWithLogitsLoss(), which is more numerically stable and combines a sigmoid layer with BCE.