Differential entropy (also referred to as continuous entropy) in information theory is a property of absolutely continuous probability distributions which generalizes the Shannon entropy to continuous probability distributions.

In terms of measure theory, the differential entropy of a probability measure is the negative relative entropy from that measure to the Lebesgue measure, where the latter is treated as if it were a probability measure, despite being unnormalized.

Definition

be a random variable with a probability density function whose support. The differential entropy is defined as

For probability distributions which do not have an explicit density function expression, but have an explicit quantile function as

As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2 (i.e., the units are bits). See logarithmic units for logarithms taken in different bases. Related concepts such as joint, conditional differential entropy, and relative entropy. For example, the differential entropy of a quantity measured in millimeters will be more than the same quantity measured in meters; a dimensionless quantity will have differential entropy of more than the same quantity divided by 1000.

One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, the uniform distribution has negative as shown now

which has zero differential entropy. Thus, differential entropy does not share all properties of discrete entropy.

The continuous mutual information has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of partitions as these partitions become finer and finer. Thus it is invariant under non-linear homeomorphisms (continuous and uniquely invertible maps), including linear, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.

For the direct analogue of discrete entropy extended to the continuous space, see limiting density of discrete points.

Properties of differential entropy

• , the Kullback–Leibler divergence almost everywhere with equality if and only if are independent.

• The chain rule for differential entropy holds as in the discrete case

• .

• Differential entropy is in general not invariant under arbitrary invertible maps. and an invertible (square) matrix

• is the Jacobian..

• has mean zero and covariance is jointly gaussian (see below).

However, differential entropy does not have other desirable properties:

• It is not invariant under change of variables, and is therefore most useful with dimensionless variables.

• It can be negative.

A modification of differential entropy that addresses these drawbacks is the relative information entropy, also known as the Kullback–Leibler divergence, which includes an invariant measure factor (see limiting density of discrete points).

Maximization in the normal distribution

Theorem

With a normal distribution, differential entropy is maximized for a given variance. A Gaussian random variable has the largest entropy amongst all random variables of equal variance, or, alternatively, the maximum entropy distribution under constraints of mean and variance is the Gaussian.

Proof

be a Gaussian PDF with mean an arbitrary PDF as

Consider the Kullback–Leibler divergence following from the properties of Kullback–Leibler divergence.

Alternative proof

This result may also be demonstrated using the calculus of variations. A Lagrangian function with two Lagrangian multipliers may be defined as:

where is some function with mean . When the entropy of , are both satisfied, then a small variation about will produce a variation about which is equal to zero:

Since this must hold for any small , the term in brackets must be zero, and solving for yields:

Using the constraint equations to solve for and yields the normal distribution:

Example: Exponential distribution

be an exponentially distributed, that is, with probability density function

to make it explicit that the logarithm was taken to base , to simplify the calculation.

Relation to estimator error

The differential entropy yields a lower bound on the expected squared error of an estimator the following holds:.

Differential entropies for various distributions

is the gamma function is the digamma function is the beta function, and is Euler's constant.Many of the differential entropies are from.

Variants

Differential entropy is also not invariant under continuous coordinate transformations. It also does not share all properties of discrete entropy. For example, it may be negative. Instead of differential entropy, one may deduce the limit of discrete entropy as calculated on an increasingly dense set of points. The expression thus attained will be non-negative, and allow for coordinate-free descriptions of entropy.

. The expression is as follows:

is a probability density, this is the Kullback–Leibler divergence are transformed. The value calculated will then remain constant.

Riemann integrable. This gives a quantized is

. Note that this procedure suggests that the entropy in the discrete sense of a continuous random variable.

See also

• Information entropy

• Self-information

• Entropy estimation

References

External links