Shannon entropy

In information theory, we define the uncertainty of an event $A$ as how sure we are that $A$ is going to happen. An uncertainty of 0 is that $A$ is definitely going to happen. An uncertainty of $\infty$ means it’ll never happen. Numerically, it’s given by:

$$\text{uncertainty}={\log }_2\left(\frac{1}{P(A)}\right)$$

For a random variable, we often want the average uncertainty, i.e., the expected value. In the discrete case, this is given by:

$$E(\text{uncertainty of }X)=E\left({\log }_2\frac{1}{P(X)}\right)=-\sum _{\forall x}{P}_X(x){\log }_2{P}_X(x)$$

This is the Shannon entropy, which describes how much uncertainty there is in a random variable. This refers to a theoretical lower bound on the minimum number of bits we need to use to losslessly express the data (might not be achievable practically).

We use the base-2 logarithm to keep in line with other concepts in information theory, especially when problems involve the number of bits. The $\log$ base is determined by the number of values it can take (for bits, 2). The unit for the natural logarithm are nats.

The differential entropy is defined by:

$$H_X=-{\int }_{-\infty }^{\infty }{f}_X(x)\ln (f_X(x))dx=-E(\ln (f_X(X)))$$

which describes the entropy for continuous random variables.

See also

• Thermodynamic entropy, an analogous concept in statistical thermodynamics