In information theory, the binary entropy function, is defined as the entropy of a Bernoulli process (i.i.d. binary variable with probability of one of two values, and is given by the formula:

The base of the logarithm corresponds to the choice of units of information; base (natural logarithm) corresponds to nats and is mathematically convenient, while base 2 (binary logarithm) corresponds to shannons and is conventional (as shown in the graph); explicitly:

(by L'Hôpital's rule); and that "binary" refers to two possible values for the variable, not the units of information.

, the binary entropy function attains its maximum value of 1 shannon (1 binary unit of information); this is the case of an unbiased coin flip, the binary entropy is 0 (regardless of the unit), corresponding to no information, since there is no uncertainty in the variable.

Notation

, the entropy function is distinguished from the general entropy function in that the former takes a single real number as a parameter whereas the latter takes a distribution or random variable as a parameter. Thus the binary entropy (of .

Writing the probability of each of the two values being and , this corresponds to

. However, it is different from and should not be confused with the Rényi entropy.

Explanation

In terms of information theory, entropy, there is still a measure of uncertainty on the outcome, but one can still predict the outcome correctly more often than not, so the uncertainty measure, or entropy, is less than 1 full bit.

Properties

Derivative

The derivative of the binary entropy function may be expressed as the negative of the logit function:

1. .

where denotes the given base of the logarithm.

Convex conjugate

The convex conjugate (specifically, the Legendre transform) of the binary entropy (with base ) is the negative softplus function. This is because (following the definition of the Legendre transform: the derivatives are inverse functions) the derivative of negative binary entropy is the logit, whose inverse function is the logistic function, which is the derivative of softplus.

Softplus can be interpreted as logistic loss, so by duality, minimizing logistic loss corresponds to maximizing entropy. This justifies the principle of maximum entropy as loss minimization.

Taylor series

The Taylor series of the binary entropy function at 1/2 is

.

Bounds

:

and

denotes natural logarithm.

See also

• Metric entropy

• Information theory

• Information entropy

• Quantities of information

References

Further reading

• MacKay, David J. C. Information Theory, Inference, and Learning Algorithms Cambridge: Cambridge University Press, 2003.

zh-yue:二元熵函數