↑ arvau

Wikipedia / Information content

In information theory, the information content, self-information, surprisal, or Shannon information is a basic quantity derived from the probability of a particular event occurring from a random variable. It can be thought of as an alternative way of expressing probability, much like odds or log-odds, but which has particular mathematical advantages in the setting of information theory.

The Shannon information can be interpreted as quantifying the level of "surprise" of a particular outcome. As it is such a basic quantity, it also appears in several other settings, such as the length of a message needed to transmit the event given an optimal source coding of the random variable.

The Shannon information is closely related to entropy, which is the expected value of the self-information of a random variable, quantifying how surprising the random variable is "on average". This is the average amount of self-information an observer would expect to gain about a random variable when measuring it.

The information content can be expressed in various units of information, of which the most common is the "bit" (more formally called the shannon), as explained below.

The term 'perplexity' has been used in language modelling to quantify the uncertainty inherent in a set of prospective events.

Definition

Claude Shannon's definition of self-information was chosen to meet several axioms:

  • An event with probability 100% is perfectly unsurprising and yields no information.

  • The less probable an event is, the more surprising it is and the more information it yields.

  • If two independent events are measured separately, the total amount of information is the sum of the self-informations of the individual events.

and an event with probability, the information content is defined as the negative log probability corresponds to the scaling factor above. Different choices of b, the unit is the shannon (symbol Sh, the unit is the natural unit of information (symbol nat, the unit is the hartley (symbol Hart).

with probability mass function as outcome is defined as: is also often used for the related quantity of mutual information for the entropy.

Properties

Monotonically decreasing function of probability

For a given probability space, the measurement of rarer eventss are intuitively more "surprising", and yield more information content than more "common" events. Thus, self-information is a strictly decreasing monotonic function of the probability, or sometimes called an "antitonic" function.

, self-information values are non-negative extended real numbers. Specifically:

  • . Its occurrence is perfectly unsurprising and reveals no new information.

  • by convention. This reflects that observing an event believed to be impossible would be infinitely surprising.

This monotonic relationship is fundamental to the use of information content as a measure of uncertainty. For example, learning that a one-in-a-million lottery ticket won provides far more information than learning it lost (See also Lottery mathematics.) This also establishes an intuitive connection to concepts like statistical dispersion; events that are far from the mean or typical outcome (and thus have low probability in many common distributions) have high self-information.

Relationship to log-odds

The Shannon information is closely related to the log-odds, is defined as the logarithm of the odds denotes the event not.

This expression can be interpreted as the amount of information gained (or surprise) from learning the event did not occur, minus the information gained from learning it did occur. This connection is particularly relevant in statistical modeling where log-odds are the core of the logit function and logistic regression.

Additivity of independent events

The information content of two independent events is the sum of each event's information content. This property is known as additivity in mathematics. Consider two independent random variables with probability mass functions. The joint probability is given by the product of the individual probabilities due to independenceThis additivity makes information content a more mathematically convenient measure than probability in many applications, such as in coding theory where the amount of information needed to describe a sequence of independent symbols is the sum of the information needed for each symbol.

The corresponding property for likelihoods is that the log-likelihood of independent events is the sum of the log-likelihoods of each event. Interpreting log-likelihood as "support" or negative surprisal (the degree to which an event supports a given model: a model is supported by an event to the extent that the event is unsurprising, given the model), this states that independent events add support: the information that the two events together provide for statistical inference is the sum of their independent information.

Relationship to entropy

The Shannon entropy is defined asby definition equal to the expected.

The expectation is taken over the discrete values over its support.

is the mutual information with itself.

For continuous random variables the corresponding concept is differential entropy.

Notes

This measure has also been called surprisal, as it represents the "surprise" of seeing the outcome (a highly improbable outcome is very surprising). This term (as a log-probability measure) was introduced by Edward W. Samson in his 1951 report "Fundamental natural concepts of information theory". An early appearance in the Physics literature is in Myron Tribus' 1961 book Thermostatics and Thermodynamics.

When the event is a random realization (of a variable) the self-information of the variable is defined as the expected value of the self-information of the realization.

Examples

Fair coin toss

Consider the Bernoulli trial of tossing a fair coin. The probabilities of the events (see fair coin and obverse and reverse) are one half. Upon measuringso the information gain of a fair coin landing as heads is 1 shannon.

Fair die roll

Suppose we have a fair six-sided die. The value of a die roll is a discrete uniform random variable with probability mass functionof information.

Two independent, identically distributed dice

Suppose we have two independent, identically distributed random variables each corresponding to an independent fair 6-sided dice roll. The joint distribution

The information content of the random variate and can also be calculated by additivity of events

Information from frequency of rolls

If we receive information about the value of the dice without knowledge and the counts have the multinomial distribution

and a total probability of combinations correspond to one die rolling one number and the other die rolling a different number, each having probability , as required.

.

Information from sum of dice

The probability mass or density function (collectively probability measure) of the sum of two independent random variables is the convolution of each probability measure represents the discrete convolution. The outcome

General discrete uniform distribution

Generalizing the example above, consider a general discrete uniform random variable. The probability mass functionIn general, the values of the DURV need not be integers, or for the purposes of information theory even uniformly spaced; they need only be equiprobable.

Special case: constant random variable

degenerates to a constant random variable and probability measure the Dirac measure can take is deterministicallyIn general, there is no information gained from measuring a known value.

Categorical distribution

Generalizing all of the above cases, consider a categorical discrete random variable with support and probability mass function given by

do not have to be numbers; they can be any mutually exclusive events on a measure space of finite measure that has been normalized to a probability measure. Without loss of generality, we can assume the categorical distribution; the mathematical structure is isomorphic in terms of probability theory and therefore information theory as well.

is given

From these examples, it is possible to calculate the information of any set of independent DRVs with known distributions by additivity.

Derivation

By definition, information is transferred from an originating entity possessing the information to a receiving entity only when the receiver had not known the information a priori. If the receiving entity had previously known the content of a message with certainty before receiving the message, the amount of information of the message received is zero. Only when the advance knowledge of the content of the message by the receiver is less than 100% certain does the message actually convey information.

For example, quoting a character (the Hippy Dippy Weatherman) of comedian George Carlin:Weather forecast for tonight: dark. ]

Continued dark overnight, with widely scattered light by morning. Assuming that one does not reside near the polar regions, the amount of information conveyed in that forecast is zero because it is known, in advance of receiving the forecast, that darkness always comes with the night.

contained in a message conveying an occurrence of event.

Further, by definition, the measure is the intersection of two independent occurring is the sumThis is a functional equation with this property are the logarithm is shannon. This is the most common practice. When using the natural logarithm, the unit will be the nat. For the base 10 logarithm, the unit of information is the hartley.

shannons. See above for detailed examples.

See also

References

Further reading