In probability theory, particularly information theory, the conditional mutual information is, in its most basic form, the expected value of the mutual information of two random variables given the value of a third.

Definition

with support sets, we define the conditional mutual information as

.

.

) Kullback–Leibler divergence. Compare with the definition of mutual information.

In terms of PMFs for discrete distributions

with support sets is as follows

where the marginal, joint, and/or conditional probability mass functions with the appropriate subscript. This can be simplified as

.

In terms of PDFs for continuous distributions

with support sets is as follows

where the marginal, joint, and/or conditional probability density functions with the appropriate subscript. This can be simplified as

.

Some identities

Alternatively, we may write in terms of joint and conditional entropies as

This can be rewritten to show its relationship to mutual information

usually rearranged as the chain rule for mutual information

or

Another equivalent form of the above is

Like mutual information, conditional mutual information can be expressed as a Kullback–Leibler divergence:

Or as an expected value of simpler Kullback–Leibler divergences:

1. ,

2. .

More general definition

A more general definition of conditional mutual information, applicable to random variables with continuous or other arbitrary distributions, will depend on the concept of regular conditional probability.

be a probability space to some state space endowed with a topological structure.

Consider the Borel measure (on the σ-algebra generated by the open sets) in the state space of each random variable defined by assigning each Borel set. This is called the pushforward measure The support of a random variable is defined to be the topological support

Now we can formally define the conditional probability measure given the value of one (or, via the product topology Then, using the disintegration theorem:

, as they are allowed to become arbitrarily smaller with respect to set inclusion.

Finally we can define the conditional mutual information via Lebesgue integration:

where the integrand is the logarithm of a Radon–Nikodym derivative involving some of the conditional probability measures we have just defined.

Note on notation

need not necessarily be restricted to representing individual random variables, but could also represent the joint distribution of any collection of random variables defined on the same probability space. As is common in probability theory) to separate the principal arguments of the mutual information symbol. (No such distinction is necessary in the symbol for joint entropy, since the joint entropy of any number of random variables is the same as the entropy of their joint distribution.)

Properties

Nonnegativity

It is always true that

1. ,

. This result has been used as a basic building block for proving other inequalities in information theory, in particular, those known as Shannon-type inequalities. Conditional mutual information is also non-negative for continuous random variables under certain regularity conditions.

Interaction information

, called the interaction information (Y here is the xor of X and Z so Z acts as the "secret key" for "plaintext" X and ciphertext "Y")

Chain rule for mutual information

:

The data processing inequality is closely related to conditional mutual information and can be proven using the chain rule.

Interaction information

The conditional mutual information is used to inductively define the interaction information, a generalization of mutual information, as follows:

where

Because the conditional mutual information can be greater than or less than its unconditional counterpart, the interaction information can be positive, negative, or zero, which makes it hard to interpret.

References