Source-linked AI summary
Information processing in living systems
Gašper Tkačik, William Bialek
TL;DR
The paper addresses how the flow of information in living systems can be made precise and studied across biological scales. It reviews information-theoretic foundations, applications to biological data, and evidence concerning information gathering and representation under physical constraints. Across these examples, information theory offers a framework for analyzing biological networks and possible optimization strategies, while biological coding mechanisms can involve tradeoffs such as fidelity versus energetic cost.
Problem
The paper asks how information flow in biological systems can be quantified and whether organisms extract and represent information efficiently under physical constraints.
Method
The review introduces information theory and synthesizes applications including neural inference, population models, sensory coding, and goal-directed behavior.
Results
The reviewed studies show that information-theoretic analysis can reveal biological network structure, efficient coding, and optimization principles across scales.
Takeaways & Limitations
Information theory provides a common language for connecting biological information processing with experimental analysis and physical constraints.
Abstract
from arXiv · showhide
Life depends as much on the flow of information as on the flow of energy. Here we review the many efforts to make this intuition precise. Starting with the building blocks of information theory, we explore examples where it has been possible to measure, directly, the flow of information in biological networks, or more generally where information theoretic ideas have been used to guide the analysis of experiments. Systems of interest range from single molecules (the sequence diversity in families of proteins) to groups of organisms (the distribution of velocities in flocks of birds), and all scales in between. Many of these analyses are motivated by the idea that biological systems may have evolved to optimize the gathering and representation of information, and we review the experimental evidence for this optimization, again across a wide range of scales.
I. INTRODUCTION
The review frames biological information flow as a problem for information theory because biological systems span diverse scales and often lack simple physical organizing variables. It develops this framework to analyze experiments and ask whether living systems optimize information flow under physical constraints.
- I. INTRODUCTION: Biological systems sense, compute, decide, and allocate limited resources across widely varying temporal, spatial, and dimensional scales.Examples include bacterial chemotactic signaling and collective behavior among neighboring organisms.
- I. INTRODUCTION: The review formalizes biological information flows using information theory to clarify what they teach about the physics of living systems.It introduces information-theoretic concepts and applies them to biological data.
- I. INTRODUCTION: The review examines whether biological mechanisms can be understood as optimizing information flow subject to physical constraints.This extends the framework from describing information to studying possible optimization in living systems.
- I. INTRODUCTION: Information-theoretic ideas provide a general alternative when biological networks lack symmetries, locality, or a usable order parameter.Neural networks with thousands of connections per cell illustrate why familiar physical summaries may be unavailable.
II. BUILDING BLOCKS OF INFORMATION THEORY
The review develops information-theoretic tools for quantifying biological information, from entropy and mutual information to noisy-channel capacity and rate–distortion tradeoffs. These tools formalize dependencies, information loss, and performance under resource constraints.
- Information measures: Entropy measures information associated with possible answers and also the minimum space needed to represent long sequences of answers.Shannon’s framework connects entropy with uncertainty and representation length.
- Information measures: Mutual information measures dependence between variables, including nonlinear relationships that ordinary linear correlation can miss.It is especially useful when measured variables are nonlinear transformations of underlying variables.
- Communication channels: Noise limits information transmission through a channel, while channel capacity specifies the maximum rate of reliable transmission.Finding capacity can be difficult for detailed input/output and noise models; Gaussian channels remain especially tractable.
- Information loss: The data processing inequality states that sequential processing cannot create information: I(x; y) ≤ I(x; z) when x reaches y through z.Both component-channel capacities must be at least as large as the information transmitted through the combined channel.
- Information and performance: Rate–distortion theory links performance to the minimum mutual information required between inputs and outputs, making resource-limited information gathering experimentally relevant.For cellular regulation, achieving a target average growth rate requires a minimum I(c, e).
- Information and representation: A remaining issue is how much total biological information is explicitly accessible through simple, biologically plausible decoding mechanisms.The review identifies this distinction as especially relevant to neural coding.
III. INFORMATION AND DATA ANALYSIS
Information theory is useful for complex biological systems because it provides general measures even when the relevant dynamical features and biologically meaningful similarity metrics are unknown.
- III. INFORMATION AND DATA ANALYSIS: Information theory supplies general measures for analyzing complex systems when relevant features and biologically meaningful similarity metrics are uncertain.It can detect informative relationships even when a particular correlation function vanishes.
A. Quantifying real information flows
Information theory provides general tools for quantifying information in biological data, but reliable estimation is limited by sampling, measurement, and distributional complexity. These methods have been applied to neural signals, gene regulation, and other biological measurements.
- A. Quantifying real information flows: Entropy and mutual information are difficult to estimate because they depend on full probability distributions and can require many samples.The sampling burden grows with the number of possible states, and mutual information is harder because the state count multiplies across variables.
- A. Quantifying real information flows: Systematic estimation errors arise from the convexity of the logarithm and decline as 1/√Ns, where Ns is the number of samples.
- A. Quantifying real information flows: Bias correction, Bayesian priors, density-estimation methods, and entropy bounds provide strategies for estimating information when data are limited.Bounds can use reliably measured averages or lower-dimensional projections rather than estimating the entire distribution.
- A. Quantifying real information flows: High-quality and increasingly abundant measurements have made quantitative information-theoretic analyses feasible across neural and cellular systems.Fluorescent antibody methods in fly embryos can measure concentrations to approximately 3% of the relevant dynamic range.
B. Quantifying correlations
Biological correlations are often nonlinear, so mutual information reveals dependencies that linear correlation misses. Applied to gene expression and genome composition, it supports clustering, interaction mapping, and evidence for functional modularity.
- B. Quantifying correlations: Linear correlations do not capture all relationships between variables in complex biological networks.
- B. Quantifying correlations: Gene-expression pairs with intermediate or minimal correlation can still share nearly half the mutual information of the most correlated pairs.
- B. Quantifying correlations: Mutual-information clustering groups genes by maximizing average pairwise information and remains invariant under invertible transformations of expression levels.
- B. Quantifying correlations: Representing genomes as binary vectors enables mutual-information analysis of gene presence and absence across organisms.A 1 denotes gene presence and a 0 denotes gene absence; identifying homologous genes requires evolutionary care.
- B. Quantifying correlations: Gene presence/absence information reveals gene clusters, higher-order interactions, and associations with phenotypes that point toward functional modularity.
C. Systems with many degrees of freedom
Maximum-entropy methods construct parsimonious models of high-dimensional biological distributions from measured statistics. Applications to neural activity, protein sequences, immune repertoires, and bird flocks show how constrained models can capture collective structure and local interactions.
- C. Systems with many degrees of freedom: Maximum entropy builds a model distribution that matches measured expectation values while remaining otherwise as unstructured as possible.The method is formulated as a variational problem and produces an exponential-form distribution.
- C. Systems with many degrees of freedom: These biological applications adapt statistical-physics ideas to systems whose measured quantities are not necessarily thermal energies.The Boltzmann-like form supplies intuition and tools without implying that the biological system is in thermal equilibrium.
- C. Systems with many degrees of freedom: The method creates a hierarchy of parsimonious models, adding only the structure required to reproduce additional measured expectations.
- C. Systems with many degrees of freedom: Retinal maximum-entropy models matching pairwise correlations corrected pattern-frequency errors of many orders of magnitude and predicted rare patterns in networks exceeding 100 neurons.
- C. Systems with many degrees of freedom: Protein-sequence models found effective amino-acid interactions to be spatially local despite correlations extending over long distances, suggesting sequence-based structural inference.The sufficiency of available data and the approximations used for this inverse problem remain open questions.
- C. Systems with many degrees of freedom: In antibody repertoires, pairwise maximum-entropy models accurately described the distribution as a whole, while flock models supported local interactions and explained long-range correlations.Couplings between birds and their kth neighbors declined exponentially with k; directional correlations were associated with Goldstone modes.
D. Information-theoretic parameter inference
Biological parameter inference is difficult when likelihood functions are unavailable because observations are downstream or measurement noise is unknown. Information-theoretic methods address this by identifying low-dimensional, maximally informative representations of high-dimensional inputs.
- D. Information-theoretic parameter inference: Maximum-likelihood inference can fail when biological likelihood functions cannot be written because intervening processes or measurement noise are insufficiently modeled.
- D. Information-theoretic parameter inference: In sensory systems, information-theoretic inference searches for a low-dimensional projection of a high-dimensional stimulus that preserves information relevant to output spikes.The transformation is represented as x → y → z, with x high-dimensional and y and z low-dimensional.
- D. Information-theoretic parameter inference: Maximally informative projections were used to infer transcription-factor binding-energy models from local DNA sequences.
- D. Information-theoretic parameter inference: Two independent experiments on the same transcription factor produced consistent TF–DNA binding models despite earlier contrary claims.
- D. Information-theoretic parameter inference: Information-theoretic inference is presented as part of a broader framework connecting parameter learning with information flow between data and compressed representations.
IV. OPTIMIZING INFORMATION FLOW
Information is essential for life, but organisms operate under physical costs that constrain information processing. This motivates asking whether biological systems optimize information gathering and representation within those constraints.
- Biological information processing may be driven toward extracting and representing the maximum possible information allowed by physical constraints.
A. The genetic code
The genetic code can be viewed as a communication strategy from DNA bases to amino acids, but its efficiency and fidelity involve structured redundancy, multiplexing, and energetic tradeoffs.
- The mapping from DNA bases to amino acids naturally frames the genetic code as a coding problem.
- The bacteriophage ΦX174 uses multiple DNA reading frames, an unexpected form of multiplexing that enhances efficiency.
- Structured codon redundancy can make single-base mutations silent or chemically similar, and the code may be nearly optimal in this respect.
- Kinetic proofreading increases molecular sorting precision beyond equilibrium binding selectivity, at the cost of additional energy.
- Cells trade fidelity against energetic expenditure by balancing the cost of errors against the cost of correcting them.
B. Efficient representations in sensory processing
Sensory systems use information-theoretic principles to match neural codes to natural input distributions and physical noise. Experiments across visual and other sensory systems reveal adaptive and structurally efficient representations.
- Efficient coding depends on the input distribution, so biological codes are expected to match signals encountered in natural environments.
- Natural stimulus distributions produce larger information rates and coding efficiencies, with some codes remaining efficient at sub-millisecond resolution.
- Retinal and sensory neurons adapt coding to changes in the variance of input signals across visual, auditory, somatosensory, and higher visual systems.
- In fly vision, adaptation can rescale the input/output relation, and the selected scaling factor was shown to optimize transmitted information.
- Retinal ganglion-cell lattices have geometries predicted by information optimization, while connectivity variations nearly compensate for information loss from spatial irregularities.
- Population codes can become highly inefficient when neurons independently represent narrowly tuned features, motivating direct measurements in natural conditions.
C. Biochemical and genetic networks
Biochemical and genetic networks transmit information through noisy, molecule-limited components. Information-based measurements and optimization models connect regulatory architecture, developmental patterning, and molecular constraints.
- Bacterial gene regulation is controlled by handfuls of transcription-factor molecules, making expression regulation inevitably noisy.
- In the early fly embryo, measurements of transcription-factor inputs and gene-expression outputs estimate the regulatory communication channel P(y|x).
- Multiple gap-gene expression levels can be analyzed for how much information they provide about cell position despite the complexity of developmental regulation.
- Position-estimation errors are nearly uniform, and redistributing cells could increase transmitted information by less than 2%, indicating optimization.
- When transcription-factor concentrations are severely limited, locally optimal networks use redundant targets to make multiple measurements of a weak signal.
- Optimization studies find that network topology can dominate parameter choice, while dynamic input/output optimization remains comparatively unexplored.
D. More ambitious goals
The review examines how information-theoretic ideas can guide theories of search, representation, language, and biological decision-making. It emphasizes both the usefulness of predictive information and the difficulty of identifying generally relevant information.
- Search and information: Reducing uncertainty is the only way to speed search beyond the entropy-limited mean search time.Search can approach but cannot exceed the speed allowed by the information in the probability distribution.
- Search and information: Infotaxis treats locating an odor source as information gathering, alternating upwind flight with cross-wind casting when signals arrive intermittently.The resulting trajectories lead to the source and resemble insect search behavior.
- Representation and language: Efficient representations can preserve task-relevant information while compressing sensory or linguistic descriptions.Examples include clustering nouns by their verb contexts and representing objects across multiple viewpoints.
- Representation and language: Pereira’s sixteen-cluster Markov model assigned Chomsky’s two example sentences probabilities differing by more than 10^5.The result illustrates that statistical models can distinguish grammatical from forbidden sentences despite limited model structure.
- Predictive information: A central unresolved question is how to identify biologically relevant information without listing it separately for each system.The review notes that qualitative knowledge of relevance may account for more predictive power than the optimization principle itself.
- Predictive information: Information is useful to organisms only when it has predictive power about future states relevant to action.Because acting takes time, evaluating information requires relating it to a future state of the world.
V. OUTLOOK
The outlook is that information-theoretic analysis will become more useful as quantitative biological data expand, while general optimization theories remain difficult to establish. Concrete near-optimal examples exist, but their generality across realistic biological systems is unknown.
- V. OUTLOOK: Information-theoretic approaches to biological data analysis are expected to grow in utility as quantitative experimental data increase.The review distinguishes data-driven analyses from the more ambitious search for general theories.
- V. OUTLOOK: Random spelling errors increase text entropy while decreasing information about events in the world.This contrast illustrates that entropy alone does not determine whether information is useful for a particular quantity of interest.
- V. OUTLOOK: Concrete biological systems operating near optimal information transmission or representation provide evidence for optimization, but their generality is unknown.The review asks whether these are isolated instances or examples of a general principle.
- V. OUTLOOK: Optimization theory is easier in simplified settings and harder with many interacting degrees of freedom and fully natural signals.The review also notes that reliably measuring basic biological information quantities required decades of work.