Redundancy and synergy in multivariate Gaussians via the Blackwell order

1 INTRODUCTION

• For multivariate Gaussians, Blackwell ordering yields optimal Gaussian channels for redundancy and union, an ellipsoid interpretation, and efficient computation at large scale.The computation uses convex optimization and scales to a thousand dimensions or sources.Source: §1 · p. 2, Figure 1 · p. 2

2 BACKGROUND: THE BLACKWELL PID

• The Blackwell PID treats channels as information-bearing objects: redundancy maximizes information in a channel less informative than every source, while union minimizes information in one more informative than every source.The Blackwell order compares channels by whether one can be generated from another through stochastic processing; it also has a decision-theoretic characterization.Source: §2 · pp. 2–3

3 OTHER PRIOR WORK

• Blackwell and BROJA union information and synergy are numerically equal, but their redundancy measures differ; scalar-target MMI results do not generalize to multivariate targets.Different sources may inform different target directions, which MMI cannot distinguish.Source: §3 · p. 3

4 WHY BLACKWELL REDUNDANCY?

• Blackwell redundancy is the only evaluated multi-source Gaussian measure satisfying both monotonicity under added sources and zero redundancy for independent target features observed separately.These are the monotonicity and Independent Identity conditions.Source: §4 · p. 4
• In the COPY and COPY+ examples, only Blackwell redundancy gives the expected zero in both cases; competing measures violate Independent Identity or monotonicity.COPY assigns separate independent target features to two sources, while COPY+ adds a third source.Source: §4 · p. 4

5 BLACKWELL PID FOR GAUSSIANS

• The Blackwell order for channels with Gaussian posteriors of constant covariance is characterized by a matrix inequality between their conditional covariance matrices.The corresponding geometric view represents the order as inclusion between covariance ellipsoids.Source: §5 · pp. 4–5

5.1 REDUNDANCY

• Blackwell redundancy can be optimized over Gaussian channels, with output dimension bounded by the target dimension and the smallest source dimension.This Gaussian restriction applies even though the original definition optimizes over all channels.Source: §5.1 · p. 5
• Redundancy is the largest ellipsoid containing the no-information ellipsoid while remaining inside every source ellipsoid.For two sources, the inclusion–exclusion formula holds precisely when the no-information constraint is non-binding.Source: §5.1 · pp. 5–6

5.2 UNION INFORMATION AND SYNERGY

• When each source–target pair is jointly Gaussian and has positive-definite conditional covariance, union information can likewise be optimized over Gaussian channels.The union problem is the geometric dual of redundancy: its ellipsoid is the smallest one enclosing all source ellipsoids.Source: §5.2 · p. 6
• Synergy is joint-source information beyond union information; for a jointly Gaussian system, it is the log-volume difference between the joint-source and union ellipsoids.For multidimensional targets, union information and synergy need not reduce to simple mutual-information expressions.Source: §5.2 · pp. 6–7
• Blackwell and BROJA union information and synergy coincide, and two-source union information has a closed-form solution that also gives closed-form BROJA union and synergy.The two-source closed form applies when both normalized conditional covariance matrices are positive definite.Source: §5.2 · pp. 6–7

6 NUMERICAL OPTIMIZATION

• Redundancy and union are convex log-determinant problems in one target-sized matrix, solved with Newton–PCG, log-barrier methods, and JAX compilation.The implementation returns certified lower and upper bounds and stops when their gap is at most the specified tolerance.Source: §6 · p. 7
• When sources observe lower-dimensional target subspaces, the algorithms reparameterize redundancy or union using smaller matrices and discard unobserved directions.The reduced dimension is bounded by the target dimension and the relevant source dimensions.Source: §6 · p. 7
• For two sources, union uses a generalized-eigenvalue closed form, while redundancy tests a fast-path candidate and falls back to Newton–PCG when needed.The redundancy shortcut applies when its candidate satisfies the no-information constraint.Source: §6 · pp. 7–8
• Figure 2 compares runtimes across five matched trials, sweeping dimensions and source count for redundancy and union methods.Its panels vary source and target dimensions together, individual or paired source dimensions, target dimension, or source count.Source: Figure 2 · p. 7, §6 · p. 8
• The optimizer certifies a solution within 0.01 nats of global optimality; competing methods are run to its certified upper bound without requiring their own certificate.This comparison gives the competing methods an advantage, according to the authors.Source: §6 · p. 8

7 EXAMPLE: OPTIMAL CONTROL

• The controller combines a noisy position measurement with memory, its Kalman prediction from earlier measurements, to compute the applied control.The example uses a linear–quadratic–Gaussian controller for a two-dimensional particle.Source: §7 · p. 8
• Synergy peaks at 0.31 nats near ν ≃2.6 as sensor noise increases.At high sensor noise, the sensor is nearly ignored and its information is redundant with memory; the remaining union information is unique to memory.Source: §7 · pp. 8–9, Figure 3 · p. 8
• At low sensor noise, memory predicts the current measurement well but adds little information beyond the sensor, so redundancy is high and synergy negligible.As noise grows, the sources carry different information and part of the information about control is available only jointly.Source: §7 · p. 8

8 BEYOND GAUSSIANS

• The paper focuses on multivariate Gaussians; extending the framework to non-Gaussian systems is a natural next direction, although some results already apply there.Source: §8 · p. 9
• The Blackwell–matrix-order result does not require Gaussian sources or targets if each target can be predicted from its source with additive homoscedastic Gaussian noise.The prediction function may be nonlinear; the stated formulas additionally assume finite positive-definite target and conditional target covariances.Source: §8 · p. 9
• For a Gaussian target, redundancy and union information are exact even when source marginals are non-Gaussian.For a general target, Blackwell redundancy is bounded above by the union information for a Gaussian target with matching mean and covariance.Source: §8 · p. 9
• Invertible transformations to a Gaussian latent space preserve mutual information and Blackwell comparisons, making the programs exact when transformed source–target pairs are Gaussian and approximate otherwise.The paper also distinguishes PIDReg, which applies Gaussian PID to learned representations without requiring invertible encoders.Source: §8 · p. 9

9 DISCUSSION

• Blackwell PID has an exact, tractable characterization for arbitrarily many Gaussian sources and target dimensions, with a geometric interpretation and efficient numerical methods.The authors frame it as a principled way to characterize information across distributed representations.Source: §9 · p. 9
• Optimal channels extract redundant and union information and could help identify shared modes in high-dimensional neuroimaging data.The authors describe these data-driven applications as possibilities they have not explored here.Source: §9 · p. 9
• Tracking redundancy and synergy through deep linear network training could reveal whether units represent the same target features or specialize in different ones.Source: §9 · p. 9
• PID-based objectives could favor complementary or robust representations: penalizing redundancy favors different target features, while rewarding it favors robustness to unit loss.Penalizing synergy would favor representations whose information can be read from individual components; the approach could extend PID-based local neuron objectives to continuous multivariate representations.Source: §9 · p. 9

REPRODUCIBILITY STATEMENT

• Proofs, the optimal-control model, and optimization-algorithm details are provided in the supplementary material.Source: REPRODUCIBILITY STATEMENT · p. 10
• The solver and current timing-benchmark code are available in the Gaussian-PID repository.Source: REPRODUCIBILITY STATEMENT · p. 10
• Code reproducing the remaining figures and tables will be released with the paper.Source: REPRODUCIBILITY STATEMENT · p. 10

• AI tools assisted with mathematical derivations, numerical solvers, figure generation, and manuscript editing.Source: AI USE STATEMENT · p. 10
• The author verified the final manuscript’s mathematical proofs and retained responsibility for its scientific claims, results, and text.Source: AI USE STATEMENT · p. 10

S1.1 GAUSSIAN POSTERIORS AND BLACKWELL ORDER

• With constant-covariance Gaussian posteriors, one source Blackwell-dominates another exactly when its posterior covariance is no larger in the positive-semidefinite order.The proof constructs a Gaussian garbling between the posterior-mean channels when the covariance difference is positive semidefinite.Source: S1.1 GAUSSIAN POSTERIORS AND BLACKWELL… · p. 13

S1.2 GAUSSIAN OPTIMALITY OF REDUNDANCY

• Gaussian channels suffice to compute Blackwell redundancy for jointly Gaussian source–target pairs; for Gaussian targets, the result also holds for non-jointly-Gaussian sources under the stated posterior condition.For jointly Gaussian pairs, an optimal channel can use output dimension no greater than min(dY, mini di).Source: S1.2 GAUSSIAN OPTIMALITY OF REDUNDANCY · p. 15

S1.3 GAUSSIAN OPTIMALITY OF UNION INFORMATION

• For jointly Gaussian source–target pairs with positive-definite conditional covariances, Gaussian channels suffice to compute Blackwell union information.The result follows by optimizing over Gaussian channels subject to each source being Blackwell-dominated by the candidate channel.Source: S1.3 GAUSSIAN OPTIMALITY OF UNION… · p. 16

S1.4 TWO-SOURCE CLOSED FORM

• For two sources, the union-information optimizer is diagonal in generalized eigenvalue coordinates, with entries min(1, λj) in each direction.The generalized eigenvalues solve C2V = C1VΛ with V⊤C1V = I.Source: S1.4 TWO-SOURCE CLOSED FORM · p. 17

S1.5 ON THE INCLUSION–EXCLUSION PRINCIPLE FOR TWO SOURCES

• For two sources, the relaxed redundancy and union information satisfy inclusion–exclusion; the original redundancy does when the relaxed optimizer also meets the no-information constraint.When that constraint holds, the relaxed optimizer solves the redundancy problem and justifies the redundancy fast path.Source: S1.5 ON THE INCLUSION–EXCLUSION… · p. 18
• Blackwell redundancy never exceeds BROJA redundancy for two sources, and equality holds exactly when R+ ⪯I.The BROJA redundancy is defined by inclusion–exclusion; the equality condition is the stated Blackwell-order relation.Source: S1.5 ON THE INCLUSION–EXCLUSION… · p. 18

S1.6 CALCULATIONS FOR THE REDUNDANCY COMPARISON

• For COPY and COPY+, Blackwell redundancy is 0, whereas MMI is 1/2 ln 2 ≈ 0.35 for both examples.Pointwise redundancy and shared-exclusion redundancy also equal 1/2 ln 2 for COPY; their COPY+ expectations are evaluated numerically.Source: S1.6 CALCULATIONS FOR THE REDUNDANCY… · pp. 18–19
• For COPY, common change in surprisal redundancy is 0, while pointwise redundancy is 1/2 ln 2; COPY+ values are evaluated numerically.For COPY, the sources are independent and conditionally independent given Y, making local co-information zero pointwise.Source: S1.6 CALCULATIONS FOR THE REDUNDANCY… · p. 19
• Gács–Körner redundancy vanishes for COPY and COPY+ and is trivial for any nondegenerate joint Gaussian, even when two noisy copies contain entirely redundant information.The same limitation holds more generally for a joint density that is positive everywhere.Source: S1.6 CALCULATIONS FOR THE REDUNDANCY… · p. 19

S2 OPTIMAL CONTROL MODEL

• The system is a particle with two-dimensional position, linear restoring force, rotational drift, control input, and Brownian noise.Source: S2 OPTIMAL CONTROL MODEL · p. 20
• At each step, the controller combines the current measurement with Kalman-predicted memory, applies LQR feedback, and propagates the estimate through known dynamics.The steady-state Kalman and LQR gains follow from discrete algebraic Riccati equations.Source: S2 OPTIMAL CONTROL MODEL · p. 20
• With sampling interval Δt fixed at 0.15, the analysis sweeps sensor-noise strength from 10^-3 to 10^3 at 73 logarithmically spaced values and recomputes controller gains and information measures.Source: S2 OPTIMAL CONTROL MODEL · p. 21

S3 NEWTON–PCG OPTIMIZATION ALGORITHM

• Both measures use matrix-free Newton steps computed by preconditioned conjugate gradients, log-barrier constraints, and primal–dual stopping bounds to certify accuracy.Source: S3 NEWTON–PCG OPTIMIZATION ALGORITHM · p. 21

S3.1 COVARIANCE PREPROCESSING

• Covariance preprocessing uses only source–target marginals, removes zero-variance source directions, and can whiten the target and source noise.Redundancy and union information are invariant under invertible affine transformations of the target and each source separately.Source: S3.1 COVARIANCE PREPROCESSING · p. 21

S3.2 REDUNDANCY

• For redundancy, the solver restricts the problem to target directions observed by every source; singular cases are handled by smoothing and certified bounds.Source: S3.2 REDUNDANCY · pp. 21–22

S3.3 UNION INFORMATION

• For union information, the solver drops target directions unobserved by all sources; singular posterior covariances make the union information infinite.Source: S3.3 UNION INFORMATION · p. 23

S3.4 NEWTON–PCG ITERATIONS

• The shared iterations solve reduced matrix problems; two-source union can instead use its closed form, and two-source redundancy uses a fast path when its candidate satisfies the stated constraint.Source: S3.4 NEWTON–PCG ITERATIONS · p. 24

1. Update the bounds. Set

• The loop updates upper and lower bounds, then returns the interval midpoint when its width meets the requested absolute accuracy.For union and redundancy, the midpoint uses opposite signs, as specified in the procedure.Source: §1 · p. 24
• Conjugate gradients approximately solve the Newton equation; barrier continuation and a feasibility-checked line search guide updates to the optimization variable.The full loop is JIT-compiled in JAX, and a non-descent direction is replaced with a preconditioned negative gradient.Source: §1 · p. 25

S3.5 BENCHMARK INSTANCES AND COMPARISON METHODS

• The random-channel benchmarks use five matched instances per setting and vary dimensions and source count; each call has a 300-second wall-clock limit.Timings exclude compilation, input generation, and independent final accuracy checks; crosses indicate at least one timeout.Source: S3.5 BENCHMARK INSTANCES AND… · p. 25, Figure S1 · p. 26
• The formulation reports success only with a certified optimality gap, while competing BROJA implementations stop on objective stagnation and are run to reach the certified upper bound or the time limit.The comparison protocol favors competitors because their runtimes exclude certification.Source: S3.5 BENCHMARK INSTANCES AND… · p. 25
• The target-space union solver optimizes a single dY × dY matrix, whereas source-space BROJA optimizes source-noise covariances; both compute the same union quantity.The Newton comparison uses a log-barrier path and a compiled Newton–PCG driver; Fig. S1 also includes the Thin-PID optimizer.Source: S3.5 BENCHMARK INSTANCES AND… · pp. 25–26