PaperlayerView exact revision on arXiv

Source-linked AI summary

Partial Information Decomposition as a Unified Approach to the Specification of Neural Goal Functions

Michael Wibral, Viola Priesemann, Jim W. Kay, Joseph T. Lizier, William A. Phillips

arXiv:1510.00831v1q-bio.NCcs.ITmath.ITnlin.AOphysics.data-an

TL;DR

Neural circuits with similar structure can perform varied tasks, yet classical information theory focuses mainly on one-input, one-output transmission. The paper uses partial information decomposition (PID) to represent neural goals in a shared framework and derive new ones. It concludes that PID supports comparing many conceivable goals and designing goals from first principles, while empirical identification remains challenging.

  • Problem

    Neural circuits with similar structure serve different tasks, while classical information theory focuses mainly on one-input, one-output transmission rather than how multiple inputs combine to produce an output.

    Source: §1 · pp. 2–3

  • Method

    The paper applies PID to decompose information from two inputs about an output and use the resulting terms to represent neural goal functions.

    Source: §3 · p. 7

  • Results

    PID provides a common framework for comparing most, if not all, theoretically conceivable neural goal functions.

    Source: §7.2 · p. 20

  • Takeaways & Limitations

    PID can also support designing neural goal functions from first principles, as demonstrated with coding with synergy.

    Source: §7.2 · p. 21

    Empirically identifying a processor’s goal remains challenging because it requires recording at least two inputs and one output and estimating PID terms from data.

    Source: §7.1 · p. 20

Abstract

from arXiv · show

In many neural systems anatomical motifs are present repeatedly, but despite their structural similarity they can serve very different tasks. A prime example for such a motif is the canonical microcircuit of six-layered neo-cortex, which is repeated across cortical areas, and is involved in a number of different tasks (e.g.sensory, cognitive, or motor tasks). This observation has spawned interest in finding a common underlying principle, a 'goal function', of information processing implemented in this structure. By definition such a goal function, if universal, cannot be cast in processing-domain specific language (e.g. 'edge filtering', 'working memory'). Thus, to formulate such a principle, we have to use a domain-independent framework. Information theory offers such a framework. However, while the classical framework of information theory focuses on the relation between one input and one output (Shannon's mutual information), we argue that neural information processing crucially depends on the combination of \textit{multiple} inputs to create the output of a processor. To account for this, we use a very recent extension of Shannon Information theory, called partial information decomposition (PID). PID allows to quantify the information that several inputs provide individually (unique information), redundantly (shared information) or only jointly (synergistic information) about the output. First, we review the framework of PID. Then we apply it to reevaluate and analyze several earlier proposals of information theoretic neural goal functions (predictive coding, infomax, coherent infomax, efficient coding). We find that PID allows to compare these goal functions in a common framework, and also provides a versatile approach to design new goal functions from first principles. Building on this, we design and analyze a novel goal function, called 'coding with synergy'. [...]

1. Introduction

Repeated cortical motifs support diverse functions, motivating an abstract framework for specifying neural information-processing goals. The article proposes information theory, extended to multiple inputs through PID, as a way to compare existing goals and formulate new ones.

1. Introduction

  • Repeated anatomical and physiological motifs, including the cortical microcircuit, serve varied sensory, cognitive, and motor functions, raising the question of how one circuit supports different purposes.The authors seek a common abstract framework for specifying the relevant information-processing functions.Source: §1 · p. 2
  • A unifying framework could help compare existing proposals—including predictive coding, free energy, and coherent infomax—and encompass them as specific cases.The article presents information theory as a possible starting point for such a framework.Source: §1 · p. 2
  • A neural framework should avoid task-specific semantics because neurons receive action potentials rather than experimenter-imposed meanings.The authors argue that information theory is a natural abstract choice from this neurocentric perspective.Source: §1 · pp. 2–3
  • Classical Shannon information theory mainly describes one-input, one-output transmission, while neural processing can involve multiple inputs interacting to produce an output.The paper motivates an extension that can characterize information contributed uniquely, shared across inputs, or available only jointly.Source: §1 · pp. 3–4
  • The article introduces PID as a framework for specifying generic neural goal functions, representing existing goals, and defining new ones.It applies this approach to infomax, coherent infomax, and predictive coding, and uses it to develop goals from first principles.Source: §1 · p. 4

2. Partial Information decomposition

PID decomposes the information that two inputs jointly provide about an output into unique, shared, and synergistic contributions. The section presents the decomposition and its interpretation, while noting that accepted formulations are currently defined for two variables.

2. Partial Information decomposition

  • Unique information is output information available from one input but not the other; shared information is available from either input alone; synergy requires observing both jointly.The section also gives PID representations of conditional mutual information as unique plus synergistic information.Source: §2 · pp. 5–6
  • For two inputs X1 and X2 and output Y, PID partitions I(Y : X1, X2) into unique information from each input, shared information, and synergy.The section limits its treatment to the mutual information between one output variable and two input variables.Source: §2 · p. 5
  • The section treats two input variables because generally accepted PID formulations are currently defined only for two variables.Separate summation of contextual and driving inputs is also described as reducing the parameter space and making learning tractable.Source: §2 · p. 5
  • The information each input individually has about Y splits into that input's unique information and information shared by the inputs.The PI diagram depicts classical information terms alongside PID terms.Source: §2 · p. 6, Figure 2 · p. 6
  • Classical entropy and mutual-information relations do not determine all four PID terms, so at least one term requires a new axiomatic definition.The section notes that established axiom systems yield measures close to common-sense interpretations and satisfy the decomposition relations.Source: §2 · p. 6
  • Shared information can arise from input overlap or from the mechanism producing Y; for independent binary inputs, AND yields 0.311 bits shared and 0.5 bits synergistic information.The example has no unique information about either input in the output.Source: §2 · p. 7

3. A generic decomposition of the output information of a neural processor

The section decomposes a processor’s output entropy into PID information terms and unused bandwidth, then assigns independent weights to define a generic goal function. This parameterization is presented as a basis for comparing existing goals and introducing new ones.

3. A generic decomposition of the output information of a neural processor

  • The output entropy H(Y) is decomposed into information about Y held jointly by inputs X1 and X2 and residual entropy given both inputs.The section uses PID to decompose the information in the output of a general neural processor.Source: §3 · p. 7
  • The generic goal function G assigns independent coefficients to each non-overlapping PID term and to residual entropy, specifying how desirable each component is.This is presented as the most generic specification because the components do not overlap and their coefficients can be chosen independently.Source: §3 · p. 7
  • An alternative coefficient parameterization rewrites G using conditional mutual information terms, with the new coefficients related to the original weights.The transformation is given by γ_i = Γ_i − Γ_4 for i = 0...3, and γ_4 = Γ_4.Source: §3 · p. 7
  • The entropy decomposition is modeled on the coherent-infomax goal-function approach; the unused-bandwidth form is preferred because conditional entropy does not overlap with PI-diagram terms.A total-bandwidth form may be useful when goals explicitly concern increasing a processor’s total bandwidth.Source: §3 · p. 8
  • The paper uses this framework to analyze coherent infomax, reformulate infomax and predictive coding, and introduce coding with synergy to exploit synergy explicitly.Training changes both the goal-function value and the relative composition of output entropy.Source: §3 · pp. 7–8

4. The coherent infomax principle and its goal function as seen by PID

The section recasts coherent infomax in a partial-information-decomposition framework, separating shared, unique, and synergistic information in its objectives. This analysis shows both limits on which goals its generic function can express and trade-offs in how coherent infomax handles driving and contextual inputs.

4.1. The coherent infomax principle

  • CIP describes a processor receiving driving input X1 and contextual input X2, with their effects distinguished by their roles in local processing.The framework allows shared information transmission, some unique information from X1, reduced transmission of information only in X2, and effective use of output bandwidth.Source: §4.1 · pp. 8–9
  • Kay and Phillips’s generic function F reweights multi-information, two conditional mutual-information terms, and output entropy conditional on both inputs to express coherent-infomax goals.The terms were intended to represent shared information, unique information from each input, and unused bandwidth, respectively.Source: §4.1 · p. 9

4.2. F as seen by PID

  • The analysis rewrites F in PID terms and compares its coefficient range with the general PID goal function G.Because the mapping from F’s coefficients to G’s is not invertible, some PID coefficient choices have no counterpart in F.Source: §4.2 · p. 10
  • The generic function F cannot represent every PID goal: it lacks one degree of freedom, preventing joint maximization of shared and synergistic information while minimizing both unique contributions, or vice versa.The restriction follows because F is built from classical information-theoretic terms, whereas PID terms cannot be expressed using those terms alone.Source: §4.2 · p. 11

4.3. The coherent infomax principle as seen by PID

  • For the coherent infomax parameter choice, F_CIP maximizes shared information, promotes unique information from the driving input, and penalizes synergy between driving and contextual inputs.The objective does not explicitly include minimizing unused output bandwidth.Source: §4.3 · pp. 11–12
  • Increasing Φ1 promotes unique information from the driving input but simultaneously weakens pressure to minimize synergy, because its conditional mutual-information term also captures synergy.This is the trade-off in the parameterized coherent infomax objective.Source: §4.3 · p. 12
  • Because coherent infomax discourages synergy between driving and contextual inputs, tasks requiring substantial synergy across them, such as XOR, are not easily learned.Synergy among components within the driving input can still be learned under the conditions described in the paper.Source: §4.3 · p. 12
  • When inputs are independent, maximizing shared information and minimizing synergy compete, giving greater effective weight to unique information from the driving input.Any limited shared information in this case is mechanistic rather than source-shared.Source: §4.3 · p. 13

5. Partial information decomposition as a unified framework to generate neural goal functions

The section applies PID to infomax and formulates an information-theoretic goal for predictive coding. It shows how PID distinguishes infomax’s information components and derives an error-minimization objective for a predictive-coding circuit.

5. Partial information decomposition as a unified framework to generate neural goal functions

  • PID is used to investigate infomax and formulate an information-theoretic goal for predictive coding.Source: §5 · p. 14

5.1. Infomax

  • Infomax maximizes I(Y : X1), but neither favors unique over shared information nor uses X2 to gain synergistic information.The goal is implicitly suited to limited output bandwidth.Source: §5.1 · p. 14

5.2. Predictive coding

  • Prediction uses past inputs, potentially from arbitrarily far back, but unconstrained mutual-information maximization is trivial under Markov sampling; equal-bandwidth predictions make the problem nontrivial.Source: §5.2 · pp. 14–15
  • The minimal circuit separates prediction from error detection: an XOR compares past prediction and input, and errors trigger updates to the prediction model.Source: §5.2 · p. 15
  • The circuit’s goal is to minimize H(Y), since a stable constant output indicates no prediction errors; the objective presupposes the prediction machinery.Source: §5.2 · p. 16
  • If the XOR unit must first learn its function, it can maximize synergy from two uniform random binary inputs before being frozen for prediction learning.Source: §5.2 · p. 16

6. Coding with synergy

Coding with synergy (CWS) uses PID to value information that is available only by combining incoming input with prior or contextual information. The section derives its goal, distinguishes it from simpler objectives, and discusses its implications for bandwidth and measurement.

6. Coding with synergy

  • CWS assumes prior information X2 is already available and seeks to pass information from X1 alongside information created synergistically by X1 and X2.Source: §6 · p. 17
  • The CWS goal maximizes unique information about X1, shared information, and synergy, while minimizing unique information about X2 and unused output bandwidth.Source: §6 · p. 17
  • Unlike maximizing I(Y : X1|X2) or I(Y : X1) alone, CWS retains shared and synergistic information and has no equivalent classical-information goal function.Source: §6 · p. 17
  • Under bandwidth limits, prioritizing shared information leads to coherent infomax, while prioritizing synergy favors information recoverable only by combining X1 and X2.Source: §6 · pp. 17–18
  • Cryptographic decoding illustrates CWS: ciphertext and key can each carry no information about plaintext alone, while their combination carries the full information.Source: §6 · p. 18
  • Individual input-output recordings can miss synergistic information; detecting it requires recording two inputs and one output together.The authors suggest it may be detected more often as dense, highly parallel recordings become available.Source: §6 · p. 18
  • CWS generalizes Barlow’s efficient-coding principle by dropping what the input and prior are about, and synergy is also presented as useful for describing information modification.Source: §6 · p. 18

7. Discussion

PID describes processors by decomposing output information across inputs and assigning coefficients to those components in a maximized goal function. The discussion presents PID as a way to compare neural goals, design coding with synergy, and frame empirical analyses, while noting assumptions and measurement challenges.

7.1. Biological neural processors and PID

  • PID separates output information into unique, shared, and synergistic contributions, which coefficients Γ weight in the processor's maximized goal function G(Γ).Source: §7.1 · p. 19
  • The framework can compare existing neural goals and design coding with synergy, but does not yet provide a circuit implementation for a desired goal.The authors contrast this with coherent infomax, for which a working implementation is known.Source: §7.1 · p. 19
  • The framework assumes two inputs reasonably approximate a neuron or microcircuit; pyramidal cells offer an example of segregated inputs whose apical signals can amplify basal responses.The two-input approximation is useful when inputs fall into two groups and are measured before they fuse at the soma.Source: §7.1 · pp. 19–20
  • Identifying which goal function a neural processor performs requires separate recordings of an output and two inputs, plus estimating PID terms from data.For coherent infomax, the two inputs must also be of different types; such estimation remains an ongoing research area.Source: §7.1 · p. 20
  • Earlier psychophysical analyses found attentional influences to be modulatory and distinguished modulatory interactions from integration of multiple driving streams.The authors describe these findings as a promising beginning for larger-scale neuronal analyses using information-theoretic tools such as PID.Source: §7.1 · p. 20
  • Relating local active information storage in inputs to transfer entropy at outputs is proposed as a way to investigate predictive coding in neurons.The paper refers readers to for further discussion.Source: §7.1 · p. 20

7.2. Conclusion

  • PID supplies measures for comparing most theoretically conceivable neural goal functions and supports first-principles design, including coding with synergy.The paper introduces coding with synergy for decoding input based on prior knowledge.Source: §7.2 · pp. 20–21

Appendix A.1. Probability distributions

The appendix distinguishes notation for a probability distribution as an object from notation for its values at particular realizations.

Appendix A.1. Probability distributions

  • P(X1, X2, Y) denotes the joint probability distribution as an object in the space of distributions.Source: Appendix A.1 · p. 24
  • P(X1 = x1, X2 = x2, Y = y) denotes the distribution's value at specified realizations.Source: Appendix A.1 · p. 24
  • The shorthand p(x1, x2, y) is also used for the probability of those specified realizations.Source: Appendix A.1 · p. 24

Appendix A.2. Notation of PID terms

The appendix uses set notation to clarify which input variables or input sets share information about an output.

Appendix A.2. Notation of PID terms

  • I(Y : A, B, C, D) asks for information that all four variables share about Y.Source: Appendix A.2 · pp. 24–25
  • Ishd(Y : {A, B}; {C, D}) denotes information about Y shared between the two input sets {A, B} and {C, D}.Source: Appendix A.2 · p. 25
  • For two sets, the shared information may come from any variable or combination within each set, including jointly held information shared across the sets.Source: Appendix A.2 · p. 25

Appendix B.1. Partial information decomposition based on unique information

The appendix motivates unique information through a betting interpretation and defines it by considering distributions that preserve pairwise input-output marginals. A roulette example illustrates how unique, redundant, and synergistic information can coexist.

Appendix B.1. Partial information decomposition based on unique information

  • The formalism assumes discrete random variables X1, X2, and Y with finite alphabets and a joint probability distribution.Source: Appendix B.1 · p. 25
  • Unique information is motivated by whether access to X1 can provide an advantage in a suitably constructed game about Y against someone lacking X1.The game-theoretic intuition is framed through Alice and Bob, who have access to different inputs but not directly to Y.Source: Appendix B.1 · p. 25
  • Because the betting outcome depends on each player's input-output marginal, the unique-information claim should remain constant across joint distributions with those same pairwise marginals.Source: Appendix B.1 · p. 26
  • Across distributions with fixed pairwise marginals, the shared information remains constant while synergy can vary; the proposed unique-information measure minimizes conditional mutual information.The full joint distribution is needed to determine synergy, which varies over that set of distributions.Source: Appendix B.1 · pp. 26, 28
  • The measures require constrained minima and maxima of conditional mutual information; for two inputs these optimization problems are convex, though imposing the constraints may be non-trivial.Source: Appendix B.1 · p. 28

Appendix B.2. PID by example: Of casinos and spies

  • In the roulette example, Alice and Bob each have unique information about different outcomes, while both have redundant information about zero.Their spies tell the truth on even versus uneven outcomes, respectively.Source: Appendix B.2 · pp. 28–29
  • Combining the spies' reports provides some synergy, but not enough to determine the roulette outcome with certainty.Source: Appendix B.2 · p. 29

Appendix B.3. Estimating Synergy and PID for jointly Gaussian variables

The appendix reviews PID for jointly Gaussian variables and then presents learning-rule details for maximizing F and coherent infomax.

Appendix B.3. Estimating Synergy and PID for jointly Gaussian variables

  • Jointly Gaussian variables permit direct computation of classical information-theoretic terms from the covariance matrix, while general continuous-variable PID remains difficult to extend.The passage notes the analytic relevance of Gaussian models and their empirical use in neuroscience.Source: Appendix B.3 · p. 29
  • Barrett demonstrated nonzero synergy and shared information for jointly Gaussian variables using net synergy, whose positive and negative values respectively indicate sufficient conditions for each.The passage notes that this challenged the intuition that linear Gaussian relationships could not support synergy.Source: Appendix B.3 · p. 29
  • The MMI PID satisfies the original axioms but is limited to a univariate output; it assigns shared information the minimum marginal mutual information and zero unique information to the weaker input.Synergy is the additional information supplied by the weaker input after considering the stronger input.Source: Appendix B.3 · pp. 29–30

Appendix C. Learning rules for maximizing F and for learning the coherent infomax goal function FCIP

  • For the basic learning-rule case, a binary-output processor receives driving and contextual input groups whose weighted sums enter goal function F.The appendix restricts this presentation to a single neural processor.Source: Appendix C · p. 30
  • The activation function determines the output probability, and under fixed input distributions, learning the information-processing goal means learning the input weights.The probability is specified as a logistic function of activation A(x1, x2).Source: Appendix C · p. 30
  • Online learning requires recomputing input-distribution- and weight-dependent expectations after weight changes, using a suitable window of past inputs.The appendix introduces a nonlinear floating average of these expectations for concise learning rules.Source: Appendix C · p. 30
  • The appendix gives gradient updates for the input weights and bias coefficients.The updates are presented in concise notation for weights w and v and biases w0 and v0.Source: Appendix C · pp. 30–31
  • The chosen coherent-infomax activation preserves the driving-input sign, amplifies matching contextual input, and reverses that modulation for opposite signs.Its listed properties also require zero output when summed driving input is zero and equality with the driving input when contextual input is zero.Source: Appendix C · p. 31
Loading 1510.00831v1…