In probability theory and information theory, the interaction information is a generalization of the mutual information for more than two variables.

There are many names for interaction information, including amount of information, information correlation, co-information, and simply mutual information. Interaction information expresses the amount of information (redundancy or synergy) bound up in a set of variables, beyond that which is present in any subset of those variables. Unlike the mutual information, the interaction information can be either positive or negative. These functions, their negativity and minima have a direct interpretation in algebraic topology.

Definition

The conditional mutual information can be used to inductively define the interaction information for any finite number of variables as follows:

where

Some authors define the interaction information differently, by swapping the two terms being subtracted in the preceding equation. This has the effect of reversing the sign for an odd number of variables.

is given by

is the conditional mutual information. The interaction information is symmetric, so it does not matter which variable is conditioned on. This is easy to see when the interaction information is written in terms of entropy and joint entropy, as follows:

, the interaction information can be written in the following form (compare with Kirkwood approximation):

are independent but not conditionally independent inhibits (i.e., accounts for or explains facilitates or enhances the correlation.

Properties

Interaction information is bounded. In the three variable case, it is bounded by

If three variables form a Markov chain. Therefore

Examples

Positive interaction information

Positive interaction information seems much more natural than negative interaction information in the sense that such explanatory.

Negative interaction information

, and the result is negative interaction information.

Difficulty of interpretation

The possible negativity of interaction information can be the source of some confusion. Many authors have taken zero interaction information as a sign that three or more random variables do not interact, but this interpretation is wrong.

To see how difficult interpretation can be, consider a set of eight independent binary variables. Agglomerate these variables as follows:

bits, which it does. However, consider now the agglomerated variables

bit, indicating less redundancy. This is correct in the sense that

but it remains difficult to interpret.

Uses

• Jakulin and Bratko (2003b) provide a machine learning algorithm which uses interaction information.

• Killian, Kravitz and Gilson (2007) use mutual information expansion to extract entropy estimates from molecular simulations.

• LeVine and Weinstein (2014) use interaction information and other N-body information measures to quantify allosteric couplings in molecular simulations.

• Moore et al. (2006), Chanda P, Zhang A, Brazeau D, Sucheston L, Freudenheim JL, Ambrosone C, Ramanathan M. (2007) and Chanda P, Sucheston L, Zhang A, Brazeau D, Freudenheim JL, Ambrosone C, Ramanathan M. (2008) demonstrate the use of interaction information for analyzing gene-gene and gene-environmental interactions associated with complex diseases.

• Pandey and Sarkar (2017) use interaction information in Cosmology to study the influence of large-scale environments on galaxy properties.

• A python package for computing all multivariate interaction or mutual informations, conditional mutual information, joint entropies, total correlations, information distance in a dataset of n variables is available .

See also

• Mutual information

• Total correlation

• Dual total correlation

• Partial Information Decomposition

References

• Bell, A J (2003), The co-information lattice http://www.kecl.ntt.co.jp/icl/signal/ica2003/cdrom/data/0187.pdf

• Fano, R M (1961), Transmission of Information: A Statistical Theory of Communications, MIT Press, Cambridge, MA.

• Garner W R (1962). Uncertainty and Structure as Psychological Concepts, JohnWiley & Sons, New York.

• Hu Kuo Tin (1962), On the Amount of Information. Theory Probab. Appl.,7(4), 439–44. PDF

• Jakulin A & Bratko I (2003a). Analyzing Attribute Dependencies, in N Lavra\quad{c}, D Gamberger, L Todorovski & H Blockeel, eds, Proceedings of the 7th European Conference on Principles and Practice of Knowledge Discovery in Databases, Springer, Cavtat-Dubrovnik, Croatia, pp.229–240.

• Jakulin A & Bratko I (2003b). Quantifying and visualizing attribute interactions https://arxiv.org/abs/cs/0308002v1.

• Moore JH, Gilbert JC, Tsai CT, Chiang FT, Holden T, Barney N, White BC (2006). A flexible computational framework for detecting, characterizing, and interpreting statistical patterns of epistasis in genetic studies of human disease susceptibility, Journal of Theoretical Biology 241, 252–261. https://www.ncbi.nlm.nih.gov/pubmed/16457852?ordinalpos=19&itool=EntrezSystem2.PEntrez.Pubmed.Pubmed_ResultsPanel.Pubmed_DefaultReportPanel.Pubmed_RVDocSum

• Nemenman I (2004). Information theory, multivariate dependence, and genetic network inference https://arxiv.org/abs/q-bio.QM/0406015.

• Pearl, J (1988), Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference, Morgan Kaufmann, San Mateo, CA.

• Tsujishita, T (1995), On triple mutual information, Advances in applied mathematics 16, 269–274.

• LeVine MV, Weinstein H (2014), NbIT - A New Information Theory-Based Analysis of Allosteric Mechanisms Reveals Residues that Underlie Function in the Leucine Transporter LeuT. PLoS Computational Biology. http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003603

• https://www3.nd.edu/~jnl/ee80653/Fall2005/tutorials/sunil.pdf

• Yeung R W (1992). A new outlook on Shannon's information measures. in IEEE Transactions on Information Theory. https://ieeexplore.ieee.org/document/79902