↑ arvau

Wikipedia / Entropy rate

In the mathematical theory of probability, the entropy rate or source information rate of a stochastic process is, informally, the time density of the average information in a stochastic process. For stochastic processes with a countable index, the entropy is the limit of the joint entropy tends to infinity:

when the limit exists. An alternative, related quantity is:

For strongly stationary. The entropy rate can be thought of as a general property of stochastic sources; this is the asymptotic equipartition property.

The entropy rate may be used to estimate the complexity of stochastic processes. It is used in diverse applications ranging from characterizing the complexity of languages, blind source separation, through to optimizing quantizers and data compression algorithms. For example, a maximum entropy rate criterion may be used for feature selection in machine learning.

Definition

with a countable index gives rise to the sequence of its joint entropies. If the limit exists, the entropy rate is defined as

, by telescoping going to infinityth entropy change is itself the conditional entropyth variable once the previous ones are known. The behaviour of joint entropies from one index to the next is also explicitly subject in some characterizations of entropy.

Discussion

represents the average entropy change per one random variable, in the long term.

It can be thought of as a general property of stochastic sources - this is the subject of the asymptotic equipartition property.

For strongly stationary processes

A stochastic process also gives rise to a sequence of conditional entropies, comprising more and more random variables. For strongly stationary stochastic processes, the entropy rate equals the limit of that sequence

, which is motivated to the extent that here this is then again a rate associated with the process, in the above sense.

For Markov chains

Since a stochastic process defined by a Markov chain that is irreducible and aperiodic has a stationary distribution, the entropy rate is independent of the initial distribution.

For example, consider a Markov chain defined on a countable number of states. Given its right stochastic transition matrix and an entropy

associated with each state, one finds

is the asymptotic distribution of the chain.

In particular, it follows that the entropy rate of an i.i.d. stochastic process is the same as the entropy of any individual member in the process.

For hidden Markov models

, both sides converge to the middle.

See also

References