PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic information, autonomous agency, and nonequilibrium statistical physics

Artemy Kolchinsky, David H. Wolpert

arXiv:1806.08053v3cond-mat.stat-mech

TL;DR

The paper addresses the lack of a broadly applicable formal theory explaining which system–environment correlations have meaning for a physical system. It defines semantic information through counterfactual scrambling and viability maintenance, then develops intrinsic measures applicable to physical systems generally. The framework formalizes related concepts including value of information, semantic content, and agency, with thermodynamic interpretations from nonequilibrium statistical physics.

  • Problem

    Existing approaches did not provide a broadly applicable formal account of semantic information for arbitrary physical systems or a way to identify the meaning of particular system states.

  • Method

    The paper defines semantic information as system–environment syntactic information causally necessary for maintaining viability, measured by counterfactual interventions that scramble correlations.

  • Results

    The framework defines stored and observed semantic information and formalizes related concepts including value of information, semantic content, and agency for physical systems.

  • Takeaways & Limitations

    Semantic information can be grounded in a system’s intrinsic dynamics and self-maintenance rather than externally assigned goals or evolutionary history.

  • Takeaways & Limitations

    Connections between this framework and fitness-relevant information in evolutionary biology remain for future work.

Abstract

from arXiv · show

Shannon information theory provides various measures of so-called "syntactic information", which reflect the amount of statistical correlation between systems. In contrast, the concept of "semantic information" refers to those correlations which carry significance or "meaning" for a given system. Semantic information plays an important role in many fields, including biology, cognitive science, and philosophy, and there has been a long-standing interest in formulating a broadly applicable and formal theory of semantic information. In this paper we introduce such a theory. We define semantic information as the syntactic information that a physical system has about its environment which is causally necessary for the system to maintain its own existence. "Causal necessity" is defined in terms of counter-factual interventions which scramble correlations between the system and its environment, while "maintaining existence" is defined in terms of the system's ability to keep itself in a low entropy state. We also use recent results in nonequilibrium statistical physics to analyze semantic information from a thermodynamic point of view. Our framework is grounded in the intrinsic dynamics of a system coupled to an environment, and is applicable to any physical system, living or otherwise. It leads to formal definitions of several concepts that have been intuitively understood to be related to semantic information, including "value of information", "semantic content", and "agency".

I. INTRODUCTION

The paper develops a formal definition of semantic information as environmentally shared information that is causally necessary for a physical system to maintain its existence. The framework uses interventions that scramble system–environment correlations and connects semantic information to intrinsic dynamics, viability, and nonequilibrium statistical physics.

  • I. INTRODUCTION: Semantic information is defined as syntactic information between a physical system and its environment that causally contributes to maintaining the system’s viability.The approach aims to apply across living and nonliving physical systems without assigning goals externally.
  • I. INTRODUCTION: The framework addresses a gap in prior autonomous-agent work by formally quantifying semantic information and the semantic content of particular system states.Earlier proposals associated self-maintenance with meaning but did not provide these formal quantities.
  • B. Our contribution: Counterfactual interventions scramble selected system–environment correlations, and the resulting viability difference measures their causal contribution to continued existence.A positive viability difference indicates that at least some preserved syntactic information helps maintain the system; a negative difference indicates harmful use of information.
  • B. Our contribution: The optimal intervention preserves the smallest amount of syntactic information that maintains the actual viability, defining semantic information and separating meaningful from meaningless information.Semantic information is therefore bounded above by total syntactic information, while nonzero syntactic information alone is insufficient for nonzero semantic information.
  • B. Our contribution: The paper defines semantic content as the environment distribution conditioned on a system state under the optimal intervention, and distinguishes stored from dynamically acquired observed semantic information.Stored information derives from initial mutual information, whereas observed information derives from ongoing system–environment interactions.
  • B. Our contribution: The framework also gives thermodynamic interpretations to measures including value of information and semantic efficiency, while leaving viability, syntactic information, and intervention choices flexible for researchers.This flexibility makes the resulting measures relative to the selected viability function, information measure, and intervention procedure.

II. NONEQUILIBRIUM STATISTICAL PHYSICS

Nonequilibrium statistical physics connects information acquisition with the thermodynamic costs of maintaining low-entropy states. The paper uses this framework to relate the cost of environmental information to its viability benefit and semantic efficiency.

  • Thermodynamics of information: Maintaining a low-entropy state requires thermodynamic costs, while organisms also acquire and use environmental information for self-maintenance.The paper situates semantic information within nonequilibrium systems maintained by ongoing information exchange.
  • Information-powered states: Nonequilibrium statistical physics provides a rigorous framework for systems whose nonequilibrium states are maintained by ongoing information exchange between subsystems.Feedback-control processes exemplify this class of information-powered nonequilibrium states.
  • Information value: Environmental information often requires work to acquire but can provide arbitrarily large viability benefits, such as locating free energy or avoiding danger.The paper compares these costs and benefits through a thermodynamic multiplier.
  • Information value: The thermodynamic multiplier is the ratio of an information source’s viability value to its syntactic information, capturing its viability benefit per bit.Larger values indicate greater “bang-per-bit” and are associated with positive information value and high semantic efficiency.

III. PRELIMINARIES AND PHYSICAL SETUP

The paper models a finite coupled system and environment evolving over time, and defines existence through the system’s ability to remain in a concentrated, low-entropy state. Negative entropy is analytically useful but has important scope limitations as a viability function.

  • Physical setup: The framework considers two coupled systems, X and Y, evolving from t = 0 to t = τ, with finite discrete state spaces that may represent coarse-grained macrostates.The system/environment decomposition remains constant, and some analyses assume stochastic, discrete-time, first-order Markovian dynamics.
  • Viability function: The paper defines viability as the negative Shannon entropy of the system’s marginal distribution at time τ.This choice connects viability changes to thermodynamic quantities and bounds concentration of probability in small state subsets.
  • Viability function: Maintaining low entropy is necessary for remaining within a small viability set, because high entropy limits the probability of occupying that set.The argument uses entropy as an upper bound on probability concentration over the state space.
  • Viability function: Negative entropy can assign low viability despite low entropy and does not necessarily preserve system identity over time.Whether this is a drawback depends partly on how self-maintenance is conceptualized.
  • Scope and alternatives: Equilibrium-based viability measures may be undefined for open nonequilibrium systems, such as a Bénard cell with persistent probability fluxes.Alternative flux-based measures are possible, but their relationship to self-maintenance capacity is not necessarily clear.

V. SEMANTIC INFORMATION VIA INTERVENTIONS

The paper identifies semantic information by counterfactually scrambling system–environment correlations and retaining the smallest information sufficient to preserve viability. It develops interventions, information–viability curves, and measures of semantic information from stored correlations.

  • Intervention framework: Semantic information is quantified as syntactic information that contributes to the system’s ability to continue existing.Counterfactual intervened trajectory distributions selectively scramble correlations while preserving other aspects of the dynamics.
  • Stored and observed information: Stored semantic information scrambles initial system–environment mutual information while leaving the coupled dynamics unchanged.Observed semantic information instead changes dynamics to scramble environmental transfer entropy while preserving the initial distribution.
  • Optimal intervention: The optimal intervention destroys the most syntactic information while leaving viability unchanged, thereby separating meaningless from meaningful correlations.It is constructed from partial interventions induced by coarse-graining functions of the environment.
  • Information–viability curve: The information–viability curve gives the maximal viability achievable under an intervention preserving R bits of mutual information.The curve is defined only for information values attainable by the allowed coarse-grainings.
  • Stored semantic information: The optimal intervention preserves the actual viability with the smallest remaining syntactic information, and that remaining mutual information is defined as stored semantic information.Further scrambling would change viability, so the retained information is causally relevant to viability at time τ.
  • Stored semantic information: Semantic efficiency measures the portion of initial mutual information that causally contributes to system viability at time τ.A full-scrambling intervention removes all initial mutual information and provides a comparison for viability change.

2. Pointwise Measures

Pointwise semantic measures isolate which system–environment correlations affect viability under the optimal intervention. Semantic content records the environment distribution carrying those viability-relevant correlations.

  • Pointwise measures: The optimal intervention retains only semantic information that affects system viability at time τ.Pointwise semantic information is defined using pointwise mutual information under the optimal intervention.
  • Pointwise measures: Specific semantic information in system state x0 is the specific information about Y conditioned on x0 under the optimal intervention.It quantifies the viability-relevant environmental information associated with that system state.
  • Pointwise measures: Overall stored and specific semantic information are expectations of pointwise semantic information.The measures aggregate state-level viability-relevant correlations under the optimal intervention.
  • Semantic content: Semantic content of x0 is the conditional distribution of environmental states under the optimal intervention.It represents the precise correlations between x0 and the environment that causally affect viability at time τ.
  • Non-uniqueness: Multiple optimal interventions indicate redundant semantic sources, with each intervention yielding its own semantic content and information measures.Any one of these sources can be sufficient for preserving viability.

3. Thermodynamics

The paper interprets stored semantic information thermodynamically by comparing its viability benefit with the physical cost of system–environment mutual information. This yields a multiplier for evaluating how effectively information helps maintain viability, while acknowledging energetic and measurement-cost limitations.

  • Thermodynamic multiplier: The thermodynamic multiplier compares the viability benefit of mutual information with its physical acquisition cost.The benefit is quantified by the viability difference between actual and fully scrambled initial distributions.
  • Illustrative thermodynamic trade-off: In the food example, acquiring one bit costs kBT ln 2 of work, while failing to use it can forfeit 0.5×10^6 J of free energy.At typical temperatures, the lost free energy corresponds to far more than one bit of potential entropy reduction.
  • Thermodynamic multiplier: The multiplier measures the information’s “bang-per-bit” and exceeds one when its viability benefit outweighs its cost.It also provides a basis for comparing how different systems use information to maintain high viability.
  • Efficiency: When the value of information is positive, lower semantic efficiency corresponds to a lower thermodynamic multiplier.This connects attending to viability-relevant information with thermodynamic efficiency.
  • Limitations: The multiplier is a comparison framework, not a claim that the system itself spends kBT ln 2·I_p(X_0; Y_0) to acquire mutual information.The actual cost may be larger or paid by the environment or an external agent; direct measurement costs could instead be used when quantifiable.
  • Limitations: The operationalization ignores energetic consequences of interventions and interaction energies, leaving energy-aware definitions for future work.Scrambling correlations may involve large positive or negative changes in expected energy.

4. Example: Food-Seeking Agent

A food-seeking agent illustrates how semantic information depends on viability: only distinctions in food location that affect survival-related outcomes count as meaningful. The example separates total mutual information from the smaller amount causally necessary for maintaining viability.

  • Model: The model places food uniformly among five locations while the agent begins centrally, tracks the target, moves toward it, and risks a high-entropy death state.Failure to eat food makes escape from death extremely unlikely, on the order of ≈10^-34.
  • Results: At τ = 5, the agent stores log2 5 ≈2.32 bits of mutual information but only ≈1.37 bits of semantic information.The resulting stored semantic efficiency is ηstored ≈0.6.
  • Meaningless distinctions: Food locations {2, 3, 4} are semantically equivalent because the agent is close enough to eat immediately from any of them.The optimal intervention therefore coarse-grains these locations while preserving the viability-relevant distinctions.
  • Thermodynamic interpretation: The value of information is ΔV_stored^tot ≈22.1 bits, producing a thermodynamic multiplier of κstored ≈9.5.The example interprets the food’s value as about 9.5 times the possible information-acquisition cost.
  • Variation: The paper also describes a variation in which the agent moves away from food, yielding negative value of information.A Python implementation of the model is provided by the authors.

B. Observed semantic information

Observed semantic information extends the framework from initial mutual information to information dynamically transferred from environment to system. The method scrambles transfer through coarse-grained interventions and retains the smallest viability-preserving portion as meaningful.

  • Definition and intervention: Observed semantic information is defined by intervening on environmental-to-system information flow without changing the initial system–environment distribution.The framework uses transfer entropy because it is directed and captures dynamically acquired information.
  • Full scrambling: Fully scrambling transfer entropy makes the system’s next state conditionally independent of the environment given its current state, so transfer entropy vanishes at every timestep.The food-location example illustrates this intervention after the system initially lacks environmental information but dynamically acquires it.
  • Definition and intervention: Partial interventions replace the environment state with a coarse-grained version φ(Y_t), preserving only selected distinctions in the system’s conditional response.One-to-one coarse-graining leaves the trajectory distribution unchanged, whereas constant coarse-graining fully scrambles the environmental input.
  • Viability measures: The viability value of transfer entropy is the viability difference between actual and fully scrambled dynamics, while the information/viability curve gives maximal viability at each preserved transfer-entropy level.These quantities evaluate how dynamically transferred information affects the system at time τ.
  • Optimal intervention: The optimal intervention preserves the smallest amount of transfer entropy that achieves the actual viability, thereby retaining meaningful bits and scrambling meaningless bits.Observed semantic information is the transfer entropy remaining under this viability-optimal intervention.
  • Efficiency: Semantic efficiency is the portion of transfer entropy that determines viability and is non-negative and bounded above by one.The upper bound follows because the actual trajectory distribution is included among the possible interventions.
  • Semantic content: The semantic content of a transition is the optimal-intervention conditional distribution of the environment given the system’s current and next states.This distribution retains only correlations contributing to viability and supports pointwise observed-semantic-information measures.
  • Scope: A thermodynamic multiplier for observed semantic information is left for future treatment because transfer-entropy acquisition costs depend on how measurement is operationalized.The paper notes that this thermodynamic analysis is more involved than for stored semantic information.

C. Other kinds of semantic information

The framework can be extended beyond mutual information and transfer entropy to other syntactic information measures. One proposed extension concerns information transferred from the system to the environment.

  • Extensions: Future work could define semantic information relative to other syntactic measures, including system-to-environment transfer entropy.That direction would quantify information conveyed by the system to the environment, described as observations by the environment.

VI. AUTOMATIC IDENTIFICATION OF INITIAL DISTRIBUTIONS, TIMESCALES, AND DECOMPOSITIONS OF INTEREST

The framework can select initial distributions, timescales, and system–environment decompositions by maximizing semantic-information measures. These optimized choices identify systems and temporal scales that are especially informative for maintaining viability and support a quantitative notion of agency.

  • Choice of analysis factors: The measures depend on the system–environment decomposition, timescale τ, and initial joint-state distribution, which are generally scientist-selected factors.These choices specify which system, temporal scale, and initial conditions are being analyzed.
  • Optimizing initial distributions: For fixed decomposition and timescale, maximizing viability value over initial distributions identifies the distribution for which the system is best fit informationally.The maximizing distribution is the one under which the system benefits most from syntactic information about its environment.
  • Optimized semantic content: Semantic information measures can be defined relative to the maximizing initial distribution rather than an exogenously specified one.Semantic content then describes the environmental distribution that a system state is best fit to represent for maximizing viability value.
  • Agency and scale selection: Maximizing measures over timescales and decompositions can automatically detect subsystems and temporal scales with large amounts of semantic information.The approach also suggests a formal and quantitative definition of autonomous agency.

VII. CONCLUSION AND DISCUSSION

The paper defines semantic information intrinsically as system–environment information causally necessary for maintaining existence, and develops measures with asymmetric, viability-based properties. The framework applies broadly without requiring predefined sensors, effectors, or internal representations, while its connections to biology remain open for future work.

  • Definition and measures: Semantic information is syntactic information between a system and environment that is causally necessary for maintaining the system’s existence.Stored information uses mutual information at t = 0, whereas observed information uses transfer entropy over t ∈[0, τ].
  • Desirable properties: Negative viability values allow semantic information to be mistaken when using it harms the system’s ability to maintain its existence.The framework thereby distinguishes meaningful from harmful correlations through their effect on viability.
  • Desirable properties: The measures are asymmetric because they are defined by contributions to the system’s viability rather than the environment’s viability.This captures why a bacterium may have semantic information about its environment without the reverse relation holding.
  • Intrinsic applicability: The framework does not require separate sensor and effector degrees of freedom, which may be difficult or impossible to define for some systems.It also avoids assuming particular internal models or representations by grounding definitions in intrinsic dynamics.
  • Scope and implications: The theory applies to non-organismic physical systems, while connections to fitness-relevant information in evolutionary biology remain future work.The paper does not assume that the system of interest is an organism.

Appendix A: Relationship between entropy and probability of being in viability set

The appendix relates entropy to viability by bounding the probability that a system occupies a small desirable-state set. As entropy increases, concentration in that viability set decreases, providing the probabilistic basis for viability analysis.

  • Viability set: A viability set A is a small subset of desirable system states, with |A| ≪|X|.The analysis examines how entropy constrains the probability that X lies in A.
  • Entropy bound: The indicator function 1_A(x) distinguishes whether a system state belongs to the viability set.Entropy chain-rule arguments are applied to this binary indicator.
  • Entropy bound: Conditional entropy is bounded using the logarithms of the sizes of A and its complement, relying on the maximum entropy of distributions over finite sets.The derivation uses the fact that a distribution over n states has entropy at most log n.
  • Consequence: As entropy increases, the probability concentrated within any small viability set decreases.This is the appendix’s resulting relationship between uncertainty and occupancy of desirable states.

Appendix B: Model of food-seeking agent

The appendix models a food-seeking agent whose state includes location, satiation, and target belief, coupled to a food-location environment. The model illustrates both beneficial and harmful semantic information by comparing viability under accurate and intervened correlations.

  • Model definition: The environment tracks food location or absence, while the agent state combines location, satiation level, and target belief.The agent’s spatial location ranges over n positions and its satiation level ranges from dead to lmax.
  • Dynamics: The live agent moves toward its target, gains satiation when sufficiently close to food, and otherwise loses one satiation level per timestep.Food remains in place unless eaten or spontaneously degrades, and the agent never changes its target belief.
  • Initial conditions: The model initializes the agent at the center with maximum satiation, food uniformly distributed across locations, and perfect information about food location.The dynamics obey local detailed balance through assigned free-energy values.
  • Coarse-graining: The coarse-grained model assumes rapid local-equilibrium relaxation within each macrostate, with constant internal-entropy terms.For the dead macrostate, a large internal entropy represents many more ways of being dead than alive.
  • Food-seeking results: For the food-seeking model, mutual information is ≈2.32 bits, semantic information is ≈1.37 bits, and the viability value is ≈22.1 bits.With the stated parameters, semantic efficiency is κstored ≈0.6 and the thermodynamic multiplier is κstored ≈9.5.
  • Misleading information: When the agent moves away from its believed food location, the viability value of information becomes negative: ∆Vstored_tot ≈−13.7 bits.The same model parameters yield semantic information ≈1.37 bit, semantic efficiency κstored ≈0.6, and thermodynamic multiplier κstored ≈−5.9.
Loading 1806.08053v3…