PaperlayerView exact revision on arXiv

Source-linked AI summary

Emergent Compositional Communication for Latent World Properties

Tomek Kaszyński

arXiv:2604.03266v1cs.MAcs.LG

TL;DR

The paper asks whether communication pressure can extract discrete, compositional representations of invisible physical properties from frozen video features. It combines multi-agent discrete communication with factored messages and iterated learning, finding reliable compositional protocols when perception and task structure support them, while identifying real-video and task-structure boundaries.

  • Problem

    Prior emergent communication focused on visible attributes, while video-based physical understanding used continuous representations; the paper asks whether agents can communicate about invisible physical properties compositionally.

  • Method

    Agents observe separate physics videos and communicate through factored Gumbel-Softmax messages, trained with population-based iterated learning and evaluated through property comparisons, interventions, and controlled backbone comparisons.

  • Results

    100% of 80 four-agent seeds reached near-perfect compositionality, with PosDis=0.999 and 98.3% holdout accuracy; backbone advantages reversed between ramp and collision physics.

  • Takeaways & Limitations

    Communication pressure can induce inspectable compositional representations of continuous latent physical properties, but what becomes communicable depends on perceptual priors and task structure.

  • Takeaways & Limitations

    The experiments use controlled laboratory-style scenes, and zero-shot transfer to unconstrained real-world video remains limited despite successful validation on controlled Physics 101 footage.

Abstract

from arXiv · show

Can multi-agent communication pressure extract discrete, compositional representations of invisible physical properties from frozen video features? We show that agents communicating through a Gumbel-Softmax bottleneck with iterated learning develop positionally disentangled protocols for latent properties (elasticity, friction, mass ratio) without property labels or supervision on message structure. With 4 agents, 100% of 80 seeds converge to near-perfect compositionality (PosDis=0.999, holdout 98.3%). Controls confirm multi-agent structure -- not bandwidth or temporal coverage -- drives this effect. Causal intervention shows surgical property disruption (~15% drop on targeted property, <3% on others). A controlled backbone comparison reveals that the perceptual prior determines what is communicable: DINOv2 dominates on spatially-visible ramp physics (98.3% vs 95.1%), while V-JEPA 2 dominates on dynamics-only collision physics (87.4% vs 77.7%, d=2.74). Scale-matched (d=3.37) and frame-matched (d=6.53) controls attribute this gap entirely to video-native pretraining. The frozen protocol supports action-conditioned planning (91.5%) with counterfactual velocity reasoning (r=0.780). Validation on Physics 101 real camera footage confirms 85.6% mass-comparison accuracy on unseen objects, temporal dynamics contributing +11.2% beyond static appearance, agent-scaling compositionality replicating at 90% for 4 agents, and causal intervention extending to real video (d=1.87, p=0.022).

1 Introduction

The paper asks whether communication can make agents extract and compositionalize invisible physical properties from video. It finds that factored messages, suitable perception, iterated learning, and multi-agent structure produce robust compositional protocols, with domain-specific limits and real-video validation.

  • Physical properties such as elasticity and friction are latent in photographs and become observable through temporal dynamics like bouncing, sliding, and deceleration.
  • The task uses agents observing separate ball–ramp videos and communicating discrete symbols to compare elasticity and friction without property labels.Properties are drawn from a 5×5 grid, and the bottleneck compresses physical information into categorical messages.
  • 100% of 80 four-agent seeds reached near-perfect compositionality, with PosDis=0.999 and 98.3% holdout accuracy.With two agents, 54% of 80 runs became compositional and generalized significantly better; causal intervention selectively disrupted encoded properties.
  • The perceptual backbone determines communicability: V-JEPA 2 leads on collision dynamics, whereas DINOv2 leads on ramp physics.Collision accuracy was 87.4% versus 77.7%, while ramp accuracy was 98.3% versus 95.1%; scale matching attributes the collision gap to video pretraining rather than capacity.
  • Specialization follows domain information structure, with ratios from 0.20 to 0.95 and bandwidth allocation correlated with property extractability at r=0.964.This specialization is consistent with rate-distortion principles.
  • Compositionality succeeds for continuous property variation but fails for categorical visual recognition, where agents memorize holistic class codes.Real-video validation achieved 85.6% mass-comparison accuracy on unseen objects, with temporal dynamics adding +11.2% beyond static appearance.

2 Related Work

Related work spans emergent communication, physical understanding from video, and vision foundation models. This paper combines these areas by requiring discrete communication about latent physics rather than prediction or communication about directly visible attributes.

  • Emergent-communication research studies whether learned protocols exhibit compositional structure, with iterated learning, population training, and factored messages encouraging it.These mechanisms promote learnability, universal decodability, and positional specialization.
  • Prior physical-understanding work learns physical properties through prediction-centric approaches using continuous latent representations.The present work instead requires agents to communicate about physics.
  • DINOv2 supplies image-based visual features, whereas V-JEPA 2 uses video pretraining to capture temporal dynamics and motion understanding.The study freezes features from both backbones and compares their consequences for communicating physical properties.

3 Method

The method pairs controlled physics videos with frozen perceptual features, discrete factored messages, cooperative comparison, and population-based iterated learning. It evaluates compositionality through held-out property combinations and information-based metrics while controlling bandwidth and structure.

  • 3.1 Environment and Task: Ramp scenes vary elasticity and friction independently on a 5×5 grid, while collision scenes use visually identical spheres whose hidden mass ratio and restitution require temporal reasoning.The ramp dataset contains 300 rendered scenes; the collision setup isolates dynamics from static appearance.
  • 3.1 Environment and Task: Each sender observes one input and emits a discrete message, while the receiver compares the two inputs separately for every property.Senders receive gradients only through the receiver’s comparison loss and never observe property values.
  • 3.1 Environment and Task: A Latin-square holdout removes five property combinations while retaining every individual value, testing compositional generalization to novel combinations.The split contains 240 training and 60 test scenes.
  • 3.2 Architecture: Frozen DINOv2 and V-JEPA 2 features are temporally encoded into 128-dimensional scene representations before message generation.DINOv2 provides per-frame features, while V-JEPA 2 processes the video jointly and produces spatiotemporal tokens.
  • 3.2 Architecture: Each sender uses K independent Gumbel-Softmax heads to form a factored message, with the main configuration using K=2 and V=5.The positional structure provides capacity for specialization, but training determines whether agents use it compositionally.
  • 3.2 Architecture: Controls compare multi-agent structure against matched bandwidth, holistic messaging, and oracle pretraining.The matched-bandwidth single sender uses four positions and removes independent partial observation; oracle encoders are pretrained before sender initialization.
  • 3.3 Training: Population-based iterated learning resets three receivers every 40 epochs across nine generations, favoring protocols learnable by new receivers.Receivers learn faster than senders, making the sender protocol the stable element under cultural-transmission pressure.
  • 3.4 Metrics: PosDis measures whether each message position predominantly encodes one attribute, while TopSim and BosDis assess distance structure and position-independent specialization.Entropy regularization prevents per-head vocabulary collapse, and gradient clipping addresses Gumbel-Softmax instability.

4 Experiments

Across the experiments, compositional protocols were associated with better generalization, emerged more reliably with more agents, and depended on both the perceptual encoder and training procedure. Causal and downstream tests showed that message positions can serve as reusable, property-specific interfaces.

  • Compositionality and generalization: 54% of 80 seeds developed compositional protocols, with compositional seeds generalizing better than holistic seeds on unseen property combinations.Compositional seeds achieved 79.3% versus 75.9% holdout accuracy (p=0.011, Cohen’s d=0.59).
  • Compositionality and generalization: Position 0 encoded elasticity and position 1 encoded friction, with minimal cross-encoding in the mutual-information matrix.The reported mutual information values were 1.08 for elasticity at position 0 and 1.32 for friction at position 1.
  • Training ingredients and scaling: Without iterated learning, only 20% of runs became compositional and the train-holdout gap reached 15.9 percentage points; LazImpa produced 0% compositionality.The no-iterated-learning condition achieved 93.6% training versus 77.7% holdout accuracy.
  • Training ingredients and scaling: 100% of 80 four-agent seeds achieved near-perfect compositionality, with PosDis = 0.999 and holdout accuracy rising to 98.3%.The ablation results contrast this with 54% compositionality in the two-agent characterization.
  • Perceptual bottleneck: A standard CNN achieved 60.6% on elasticity but chance on friction, while end-to-end fine-tuning reduced holdout accuracy to 67.8% versus 78.0% for frozen features.The results indicate that communication structures information available from perception and that fine-tuning can improve probe fit while degrading communication generalization.
  • Causal evidence and reusability: Zeroing relevant message positions caused large task-specific drops, while irrelevant positions had negligible effects, supporting surgical property extraction.The receiver extracted elasticity from position 0 of message A and friction from position 1 of message B.
  • Causal evidence and reusability: Frozen messages supported new receivers on same-property comparison at 81.8% holdout and cross-property comparison at 89.8%, but single-message regression reached only 23%.The failure of single-message regression indicates that the protocol encodes relational structure rather than independently recoverable property values.

4.3 Generalization Across Domains

The unchanged communication architecture produces compositional protocols across domains, while specialization and bandwidth allocation reorganize according to each domain’s information structure. Scaling independent agents reliably improves compositionality, whereas categorical identity tasks encourage holistic codes.

  • Cross-domain replication: Compositional communication emerges across domains with the same communication module, despite domain-specific encoders.The encoders differ by domain, but the sender, receiver, and training recipe remain identical.
  • Information-driven specialization: 0.992 specialization appears in spring-mass, while ramp agents specialize by friction and elasticity across sliding and bounce phases.Spring-mass damping is directly encoded from the decay rate γ = b/(2m); ramp observations distribute information across temporal phases.
  • Information-driven specialization: Damping dominates all message positions in three-property physics, with MI 0.53–0.66 because it is more extractable than elasticity or friction.Agents redundantly encode the highest-SNR signal rather than assigning one position per property.
  • Information-driven specialization: r = 0.964 correlation links total mutual information per property to oracle accuracy across six visual properties.Loss reweighting preserves the pattern, supporting bandwidth allocation toward higher-SNR signals.
  • Categorical boundary: ∼27% novel-class accuracy shows categorical referential games produce near-chance generalization, unlike continuous property variation.Nearest-neighbor raw features reach 50.2%, while the communication conditions favor class-specific, non-compositional codes.
  • Agent scaling and controls: 100% compositionality across 80 four-agent seeds reaches PosDis = 0.999 and 98.3% holdout accuracy.Random frame assignment preserves 100% compositionality, while matched-bandwidth single senders achieve only 35% compositionality and 87.1% holdout.

4.7 Perception Determines What Is Communicable

The perceptual backbone determines which physical properties can be communicated when sufficient multi-agent bandwidth exposes representational differences. Video-native features dominate dynamics-only collision physics, and their advantage transfers through the discrete interface to downstream counterfactual prediction.

  • Perceptual priors: 88.0% oracle performance for V-JEPA 2 versus 78.7% for DINOv2 on collision shows a backbone advantage for dynamics-only physics.The collision spheres are visually identical, so mass ratio and elasticity must be inferred from post-collision velocity changes.
  • Perceptual priors: At four agents, V-JEPA 2 reaches 87.4% versus DINOv2’s 77.7% on collision, while DINOv2 reaches 98.3% versus V-JEPA 2’s 95.1% on ramp physics.The ramp properties are partially readable from individual frames, whereas collision mass ratio requires temporal velocity information.
  • Controls: 87.4% versus 74.6% at matched scale preserves V-JEPA 2’s collision advantage, with d=3.37.DINOv2 ViT-L performs worse than the smaller DINOv2 ViT-S, ruling out parameter count as the explanation within this comparison.
  • Controls: d=6.53 remains after frame-count matching, so additional image frames do not compensate for DINOv2’s collision deficit.The frame-matched DINOv2 condition reaches 71.6% ± 3.6%, below its 24-frame result of 77.7%.
  • Downstream utility: 10.2pp separates V-JEPA 2 and DINOv2 messages in downstream outcome prediction from frozen representations.The messages transfer to a novel task without retraining the sender, while discrete communication retains interpretability and property alignment.
  • Downstream utility: 91.5% accuracy on held-out velocity × property combinations accompanies counterfactual correlation r=0.780.Predictions change monotonically in the physically correct direction when velocity varies while the scene and frozen message remain fixed.

4.9 Why Discrete? Discrete vs. Continuous Communication

Discrete messages provide sharper positional selectivity than continuous messages while preserving a structured interface for causal manipulation. On real camera footage, the same framework communicates mass from temporal dynamics and transfers causal alignment beyond synthetic environments.

  • Discrete versus continuous: 0.807 discrete-message selectivity exceeds 0.722 for continuous messages under block-zeroing.The continuous and discrete systems have the same 40 total dimensions and comparable comparison accuracy, but discretization sharpens positional property disruption.
  • Discrete versus continuous: Discrete bottlenecks provide addressable compositional structure and greater training stability beyond raw task performance.The positions can be selectively queried or causally manipulated as physical-property factors.
  • Real-video validation: 0.905 V-JEPA 2 temporal accuracy exceeds 0.819 DINOv2 static accuracy on spring-scenario mass comparison.Both exceed the 0.761 volume-only baseline, where mass is dynamically visible through spring extension.
  • Real-video validation: 85.6% ± 5.8% holdout accuracy on unseen objects is achieved by two-agent communication on real spring footage.The result includes TopSim 0.49–0.79 and mass–symbol Spearman |ρ| = 0.71–0.90.
  • Real-video validation: +11.2% beyond appearance shows temporal dynamics contribute to full communication accuracy on Physics 101.The matched static-feature baseline reaches 72.9%.
  • Real-video causal alignment: 7.8pp disruption from zeroing the mass-relevant agent versus 2.1pp for the other yields a selectivity gap of 5.7pp.The causal intervention extends to real footage with p=0.022 and Cohen’s d=1.87.
  • Real-video scaling: 90% compositional emergence is reached with four agents on real video, while PosDis increases from 0.294 to 0.483.The absolute compositionality is lower than synthetic results because restitution labels and real footage are noisier.

5 Discussion

The paper argues that communicative pressure can organize latent physical information into reusable, causally addressable structure, but that both compositionality and accessible content depend on task structure, perception, and domain scope.

  • Communication pressure: Multi-agent pressure yields 100% compositionality across tested 3- and 4-agent seeds without property labels or message-structure supervision.Randomized temporal assignments preserve the effect, whereas matched-bandwidth single senders do not.
  • Information structure: Specialization ratios range from 0.20 in six-property vision to 0.95 in spring-mass as information structure changes.Cleanly separated observations produce tighter specialization, while uniformly distributed information produces distributed coding.
  • Boundary conditions: Categorical identity tasks produce holistic codes, establishing a boundary for compositionality under continuous property variation.The paper links compositionality to repeated property values across inputs with different values of other properties.
  • World-model interfaces: The emergent protocol compresses spatiotemporal features into discrete symbols that are position-addressable, causally aligned, and reusable across tasks.The backbone comparisons indicate that perceptual pretraining determines which physical aspects become accessible through the interface.
  • Evaluation methodology: The communication framework complements probing by testing whether information can be structured into compositional, reusable form.The paper proposes it as a scalable diagnostic for which latent properties become communicable in frozen world-model representations.
  • Planning interface: 91.5% novel-velocity outcome accuracy with r=0.780 counterfactual correlation connects the interface to planning-relevant reasoning.The proposed downstream use is a discrete perception-to-planning interface with selectively queryable physical descriptors.
  • Limitations: The main scope boundary is unconstrained real-world video: current validation uses controlled laboratory footage, and zero-shot cross-domain transfer largely collapses messages.Direct training on Physics 101 succeeds at 85.6% on unseen objects, indicating domain-specific visual mapping.

6 Conclusion

The conclusion argues that communication pressure can induce compositional representations of invisible world properties when perception and message structure are adequate. It further identifies multi-agent structure and perceptual priors as important constraints on what becomes communicable.

  • Communication pressure induces compositional representations of invisible world properties when agents have adequate perception and factored message structure.
  • Multi-agent structure, rather than temporal coverage or raw channel capacity, is identified as the critical ingredient supporting reliable emergence.
  • The perceptual prior bounds communication: image features suffice for spatially visible physics, whereas video-native features are needed for collision dynamics.
  • Compositionality emerges reliably for continuous properties but not categorical identity, indicating a task-structure limitation.

A Hyperparameters

This section identifies the hyperparameter table used across the experiments.

  • Table 13 lists the hyperparameters used across all experiments unless otherwise noted.

B Domain Details

The domain details cover rendered physical scenes, spring-mass dynamics, abstract geometric scenes, and visual-attribute and categorical controls. These domains vary the physical properties, observations, feature encoders, and held-out evaluation settings.

  • Ramp Physics: Ramp Physics uses 300 Kubric/PyBullet scenes with a 5×5 elasticity–friction grid and a Latin-square holdout of 5 cells.
  • Collision Dynamics: Collision Dynamics uses visually identical spheres with randomized approach velocity and a 5×5 grid of mass-ratio and restitution values.
  • Spring-Mass Oscillation: Spring-Mass Oscillation models x(t) = A · e−γt · cos(ωt), varying stiffness and damping across five bins each.
  • Abstract Geometric Scenes: Abstract Geometric Scenes vary numerosity and mean size across randomly placed shapes viewed through four spatial quadrants.
  • CIFAR-100 Visual Attributes: CIFAR-100 Visual Attributes evaluate brightness, saturation, and additional color and spatial properties using quintile-binned features.
  • CIFAR-100 Categorical: The CIFAR-100 categorical control is a referential game with 80 training classes, 20 held-out classes, and same-superclass distractors.

C Additional Experiments

Additional experiments test transfer, protocol adaptation, capacity use, loss weighting, encoder robustness, and end-to-end perception. Together, they show that compositional protocols can transfer and adapt, while frozen perception remains important for reliable discrete communication.

  • Cross-Physics Transfer: 42.3% holdout accuracy transfers from ramp scenes to flat-drop scenes, exceeding the random-sender baseline of 32.7%.
  • Protocol Adaptation: 92.4% train accuracy is reached within 11 epochs after adding an interaction property, while original elasticity and friction accuracy is retained.
  • Emergent Vocabulary Structure: 82.4% accuracy and TopSim = 0.730 occur with 4×5 overcomplete messages, which use all positions redundantly rather than discovering minimal factorization.
  • Inverse Loss Weighting: Inverse loss weighting leaves overall accuracy essentially unchanged at 40.0% versus 40.5%, with mutual information remaining distributed.
  • Encoder Ablation: Deterministic feature tiling lowers spring-mass both-correct accuracy from 93.8% to 89.4%, while damping specialization persists across encoders.
  • End-to-End Perception Ablation: End-to-end DINOv2 yields better elasticity features but worse communication, with holdout 67.8% ± 9.3% versus 78.0% ± 5.1% for frozen encoding.

D NeurIPS Paper Checklist

The paper reports empirical contributions, limitations, experimental practices, resources, data provenance, human-subject status, and broader-impact considerations.

  • Claims: All claims are supported by experiments with multiple seeds and reported standard deviations.
  • Limitations: The study is limited by synthetic environments, frozen features, a 54% two-agent emergence rate, and a small sample for bandwidth correlation.
  • Theory: The work presents empirical contributions rather than formal theorems.
  • Experiments: Experiments use 20 seeds per condition, 80 seeds for the main two-agent ramp characterization, reported standard deviations, and approximately 120 GPU-hours equivalent.
  • Code and Data: The authors state that code, datasets, and trained models will be released with the arXiv preprint, and that the study uses open or publicly available data and model weights.
  • Human Subjects and Broader Impact: The study involves no human subjects and frames its broader impact around representation learning, interpretable AI, and embodied-AI world-model design.
Loading 2604.03266v1…