PaperlayerView exact revision on arXiv

Source-linked AI summary

One Future, Every Robot: Label-Efficient Collective-State Prediction with Decentralized JEPA

Alan-Barsag Gazzaev, Alexey Gavrilov, Sergey Muravyov

arXiv:2607.28443v3cs.RO

TL;DR

Decentralized robots need compatible predictions of their collective future from different local evidence, without centralized estimation or pooled outputs. CS-JEPA trains each robot to predict a shared latent future, and finds that predictions become more compatible while accuracy improves, including under topology and swarm-size shifts.

  • Problem

    Decentralized robots must infer a compatible collective future from different local histories without a central estimate or pooled outputs.

  • Method

    CS-JEPA trains each robot’s local history and bounded neighbor messages to predict the same fixed-width latent future without agreement loss.

  • Results

    Accuracy AUC falls 28.4% in-distribution and 64.4–75.6% under topology and size shift, while independently formed predictions become more compatible.

  • Takeaways & Limitations

    A shared latent future can align fragmented local evidence while preserving information useful for low-label probes and independently executed decisions.

  • Takeaways & Limitations

    The approach still assumes shared orientation and scale, despite removing absolute origin and task displacement.

Abstract

from arXiv · show

Decentralized robots often need a common view of what their team is becoming, even though each robot sees different evidence and cannot rely on a central estimate or output-level consensus. We ask whether compatible collective-state predictions can emerge under this constraint. Collective-State JEPA (CS-JEPA) trains every robot to predict the same fixed-width latent future from its own history and bounded neighbor messages, with no agreement loss; predictions and plans are never pooled at deployment. In a fresh independent replication, agreement improves for every seed and every evaluated split. Accuracy improves at the same time, ruling out the uninformative solution in which all robots merely collapse to one prediction: relative to capacity-matched raw-future reconstruction, collective-state error falls by 28.4 percent in distribution and by 64.4 to 75.6 percent under topology and swarm-size shift. Translation-free and crossed-pretraining controls preserve this joint result, while action-conditioned and rigid-body evaluations show that the receiver-local representation supports independent decisions. A shared latent future can therefore align decentralized predictions without consensus training while preserving useful, label-efficient information.

I. INTRODUCTION · II. RELATED WORK · A. Global information from local interactions

CS-JEPA tests whether robots can independently predict a compatible shared future from local evidence and bounded communication, without output pooling or agreement loss. The section frames success as joint compatibility and physical accuracy, then motivates transfer and decentralized control evaluations.

  • I. INTRODUCTION: The decentralized challenge is that robots observe partial evidence of global regimes, so independently plausible predictions can describe incompatible futures without a central estimate.
  • II. RELATED WORK: Prior work uses permutation-invariant encoders, graph policies, global-state predictors, multiagent representation learning, world models, and JEPAs to connect local exchange with prediction and control.
  • I. INTRODUCTION: CS-JEPA trains different local views toward the same latent future to align their meanings without training robots to match one another’s outputs.
  • I. INTRODUCTION: Agreement alone is insufficient because constant predictions can agree perfectly; the decisive criterion is improved compatibility alongside preserved or improved physical accuracy.
  • I. INTRODUCTION: The proposed system uses recurrent local evidence and fixed-width outputs, with communication and output width remaining invariant as swarm size grows.
  • I. INTRODUCTION: Fresh-cohort results report more compatible receiver-local predictions without agreement loss while accuracy improves across label budgets, topology shifts, and swarm-size shifts.
  • I. INTRODUCTION: Action-conditioned value estimation and fully decentralized rigid-body control test whether each robot can use the learned common future to select and execute its own plan.
  • A. Global information from local interactions: Existing approaches establish that local communication can recover global context, but the open question is whether every receiver can independently predict one compatible future collective target with few labels.

B. Predictive representations for multiple agents · C. JEPA beyond single observations · III. PROBLEM FORMULATION

The paper positions JEPA as a label-efficient alternative for decentralized multi-agent prediction, then formulates the challenge of constructing a common future from receiver-specific local evidence and bounded messages. Collective state is defined as a learned global-plus-spatial latent target with an audited physical decoding task, not a concatenated robot-state vector.

  • B. Predictive representations for multiple agents: Multi-agent representation learning studies whether prediction can improve team behavior, with prior systems using compact message aggregates, joint transition models, or centralized aggregation.MASIA reconstructs and predicts future information, while TIMAR uses a joint transition model to inform local representations.
  • B. Predictive representations for multiple agents: Recent multi-agent world models separate local and joint latent states or learn communication and coordination directly, but are usually judged by return or imagined outcomes.
  • C. JEPA beyond single observations: JEPA predicts task-relevant content in a learned target space rather than reconstructing every observation detail, motivating its use for decentralized collective-state prediction.Related extensions include population-level JEPA, V2X-JEPA, and TrajJEPA for sets, cooperative perception, and multiagent trajectory forecasting.
  • III. PROBLEM FORMULATION: The central problem is making one future predictable from many local information sets without copying a centralized estimate back to the robots.A simulator or centralized training system can define a global future directly; the difficulty is decentralized predictability.
  • III. PROBLEM FORMULATION: At time t, the active swarm is represented as a dynamic graph G_t = (V_t, E_t), with robot i receiving messages only from its neighbor set N_t(i).
  • III. PROBLEM FORMULATION: Each robot’s local input contains normalized position, velocity, a two-dimensional task vector, and an active bit, while neighbors provide only previous 64-float recurrent memories.Raw neighbor features are unavailable; the task vector depends on flocking, formation, or coverage.
  • III. PROBLEM FORMULATION: The target is common but evidence and prediction error are receiver-specific, and each robot must construct the shared future without averaging robot predictions.The final output is h = 4 collective_prediction.
  • III. PROBLEM FORMULATION: Collective state is a learned global-plus-spatial latent target decoded into ten future collective quantities, not the concatenated state vector of all robots.The audited quantities include polarization, target alignment, cohesion, dispersion, angular momentum, connectivity, normalized cluster count, collision fraction, mean speed, and task score.

A. Decentralization and scaling semantics · IV. COLLECTIVE-STATE JEPA

CS-JEPA remains decentralized by producing one output per robot from local hidden states and bounded communication, rather than copying a central estimate. Fixed-width aggregation and tokenization make model parameters independent of swarm size, while agreement remains meaningful when accuracy also improves.

  • A. Decentralization and scaling semantics: Equation (2) defines |V_t| outputs, so each robot retains its own prediction rather than receiving one estimate copied from a central node.Robots share parameters and target semantics, but robot i never reads robot j’s prediction.
  • A. Decentralization and scaling semantics: Each robot retains its own hidden state and does not read another robot’s prediction.The decentralized design shares parameters and target semantics without pooling predictions.
  • A. Decentralization and scaling semantics: Information travels beyond one hop only through successive recurrent updates, so a 16-frame context is not instantaneous global access.The communication mechanism therefore preserves temporal and local-information constraints.
  • IV. COLLECTIVE-STATE JEPA: Agreement is substantive when accuracy improves simultaneously, rather than being attributable to indistinguishable centralized estimates.The passage explicitly links the meaning of agreement to concurrent accuracy improvement.
  • A. Decentralization and scaling semantics: 256|E_t| bytes are prescribed as total traffic per step, while robot i receives 256|N_t(i)| bytes.These quantities specify communication for a directed communication graph.
  • A. Decentralization and scaling semantics: Mean aggregation fixes the GRU input width as degree changes, and the tokenizer fixes output width as N changes.Both mechanisms keep tensor widths fixed despite changing graph degree or swarm size.
  • A. Decentralization and scaling semantics: These fixed-width choices make model parameters independent of swarm size and keep per-robot traffic bounded on degree-four training graphs.They do not make total network traffic independent of N.

A. Receiver-local recurrent transport · B. One variable-size future target

CS-JEPA transports evidence through receiver-local recurrent states and predicts a shared fixed-width future target for variable-size swarms. The design excludes prediction sharing and agreement training while preserving fixed communication and target dimensions.

  • A. Receiver-local recurrent transport: The communication module carries evidence rather than a centralized answer, using a frozen local encoder for each robot’s self observation.
  • A. Receiver-local recurrent transport: Each active robot broadcasts only its previous recurrent state, keeping communication receiver-local.
  • A. Receiver-local recurrent transport: Mean aggregation keeps input width fixed as degree changes, while recurrence transports evidence across multiple steps without exposing another robot’s prediction.
  • A. Receiver-local recurrent transport: One message round occurs per environment step, with 64 float32 values (256 bytes) sent per directed edge.With 16 frames, measured warm-up is 15 message steps.
  • B. One variable-size future target: The target combines one global token with a fixed 4 × 4 spatial field, avoiding both a layout-blind single vector and a swarm-size-dependent per-robot tokenization.
  • B. One variable-size future target: Seventeen 65-D tokens form a fixed 1105-D Z_t+h regardless of swarm size, using the active-set mean embedding and a presence mass.
  • B. One variable-size future target: A role-conditioned predictor recursively forecasts t+2 then t+4, stopping the gradient through its first prediction.
  • B. One variable-size future target: Every receiver trains toward the same target, not toward other receivers, with agreement loss weight zero.

C. Receiver anchor and matched reconstruction

The receiver anchor preserves receiver-local dynamics by predicting its frozen future embedding during training only. Future-Recon provides a capacity-matched comparison that changes only the self-supervised target from the common latent target to raw-field reconstruction, without deployment pooling of the anchor head.

  • Receiver anchor: A training-only MLP predicts each receiver’s frozen future embedding to retain the receiver’s own dynamics.The anchor head is absent from predict_collective and deployment counts, while both methods use λ = 2.
  • Matched reconstruction: Future-Recon preserves the frozen encoder, messages, role-conditioned predictor, 1105-D bottleneck, schedule, probe, and deployment parameters, changing only the self-supervised target.Its training-only decoder reconstructs a global-plus-spatial field of normalized future position, velocity, and task vector.
  • Matched reconstruction: 139,137 online trainable pretraining parameters are used by CS-JEPA versus 148,744 by Future-Recon, whose raw decoder adds 9,607 parameters.Both methods deploy 123,713 representation parameters, or 134,773 with the probe, so the comparison does not favor CS-JEPA through extra capacity.

D. Training-to-deployment separation · V. EXPERIMENTAL SETUP · A. Data and training

CS-JEPA confines privileged future information to training: every receiver learns the same target while deployment uses only local recurrent prediction. The evaluation separates topology and swarm-size transfer under matched training, architecture, and checkpoint-selection conditions.

  • D. Training-to-deployment separation: Privileged future state supervises a stopped-gradient target shared by every receiver, but never enters the GRU, messages, or predictor.The online branch is unrolled independently from receiver-local inputs during pretraining.
  • D. Training-to-deployment separation: The receiver-anchor head contributes only a training loss and is excluded from collective_prediction at inference.The privileged path specifies the prediction target without computing the prediction at runtime.
  • D. Training-to-deployment separation: After checkpoint selection, deployment discards privileged modules, freezes the local encoder and recurrent predictor, and fits a ridge probe from receiver-local h = 4 predictions.Robots run the same function with separate recurrent states and errors; shared semantics do not imply synchronized latents or a server estimate.
  • A. Data and training: The label-efficiency evaluation tests flocking, formation, and coverage in 200step episodes with process noise 0.02 and no observation noise.Training and in-distribution data use bounded-small-world graphs with N ∈{10, 18} and maximum degree four.
  • A. Data and training: Topology shift is isolated with ring and mutual-kNN graphs, while the size split uses unseen N ∈ {36, 72, 108}, reaching 6× the largest training swarm.These splits separate communication-graph changes from increases in swarm size.
  • A. Data and training: Both methods train for 50 epochs with batch size 256, Adam learning rate 5×10−4, hidden width 128, and latent width 64.Within anchor-matched cohorts, methods share the same frozen stage-0 encoder; initial and fresh cohorts use five and twelve independently trained stage-1 outer seeds, respectively.
  • A. Data and training: Stage 0 remains frozen, while a crossed control retrains it under both latent-mean and raw-future objectives; checkpoint selection uses only the model-selection split.The stage-0 encoder is a separately trained mean-target CS-JEPA encoder on unlabeled swarm episodes, not an external pretrained model.
  • V. EXPERIMENTAL SETUP: Table I defines the fixed representation and deployment contract, while Table II specifies the label-efficiency data family.Together, the tables document the deployment boundary and evaluation data organization.

B. Matched comparison and leakage controls · C. Label efficiency and endpoints

The study uses a capacity-matched comparison and leakage controls to isolate the self-supervised prediction target. Label efficiency is evaluated across multiple budgets with normalized error, agreement, and a learning-curve AUC endpoint.

  • B. Matched comparison and leakage controls: CS-JEPA and Future-Recon share data, histories, architecture, training schedule, checkpoint selection, label subsets, probes, test episodes, representation width, receiver anchor, and parameter count.They differ in the primary self-supervised prediction problem and raw-reconstruction decoder.
  • B. Matched comparison and leakage controls: Downstream labels remain hidden during representation learning, while episode-level subset selection is stratified and performed without reading label values.Model-selection episodes select checkpoints but never fit downstream probes.
  • C. Label efficiency and endpoints: Ridge probes use α = 10−3 and 6, 12, or 24 globally labeled episodes, with five task-and-size-stratified draws nested within each budget.Subset selection does not inspect labels.
  • C. Label efficiency and endpoints: Accuracy is fixed-physical-scale normalized MSE over ten decoded quantities, whereas agreement is unnormalized within-episode MSE between decoded robot predictions; lower is better for both.The two metrics have different scales and are interpreted through paired within-seed comparisons.
  • C. Label efficiency and endpoints: For each time window, decoded robot predictions byi ∈R10 are compared with a common target y across N active robots and fixed physical scales sd.The supplied formulation defines window-level contributions for the endpoint metrics.
  • C. Label efficiency and endpoints: The Eagr centroid is used only to score post-inference dispersion, is not fed to robots, and is not the primary prediction.Window values are reduced to episode values and task-balanced so larger swarms and longer episodes do not receive more inferential weight.
  • C. Label efficiency and endpoints: The primary label-efficiency endpoint is a log-budget-span-normalized trapezoidal AUC, AUC = 0.25E6 + 0.50E12 + 0.25E24.It summarizes the full learning curve rather than relying on one label budget.
  • C. Label efficiency and endpoints: The reported paired effect is ∆= AUCF uture−Recon −AUCCS−JEP A, so positive values favor CS-JEPA; episodes are resampling units and training seeds are outer units.The design uses n = 5 initially and n = 12 per extension.

D. Fresh-cohort replication and objective-isolation controls · E. Action-conditioned counterfactual value estimation

The paper tests CS-JEPA’s robustness through a fresh-cohort replication and crossed objective controls, then evaluates whether receiver-local, action-conditioned representations contain planning-relevant information. The counterfactual evaluation uses exhaustive branch supervision with paired value-error testing rather than averaging robots.

  • D. Fresh-cohort replication and objective-isolation controls: The fresh replication introduces a new cohort and twelve new stage-1 seeds while retaining the original checkpoint, splits, budgets, draws, architecture, and endpoints.The replication is analyzed separately from the initial cohort.
  • D. Fresh-cohort replication and objective-isolation controls: A position-control experiment retrains both stages with self-relative receiver positions, a centroid-centered teacher field, and zeroed task-vector chan…
  • D. Fresh-cohort replication and objective-isolation controls: The registered 2×2 control crosses stage-0 latent-mean JEPA prediction or matched raw-future reconstruction with stage-1 CS-JEPA or Future-Recon.Eight independently trained stage-0 seeds are paired across all four cells while initialization, data, architectures, endpoints, and selection rules remain unchanged.
  • D. Fresh-cohort replication and objective-isolation controls: The crossed-objective experiment uses equal-weighted four-split accuracy AUC contrast as its primary endpoint.
  • E. Action-conditioned counterfactual value estimation: Action-conditioned variants use eight outer seeds, with each four-step plan conditioning collective_prediction before a frozen predictor and scalar ridge readout estimate its outcome.Both methods share the stage-0 encoder, receiver-local architecture, communication, anchor, plans, and counterfactual data.
  • E. Action-conditioned counterfactual value estimation: The counterfactual readout trains on 1,800 branches from 12 base episodes and tests on 1,800 disjoint branches from 12 new base episodes across 72 shared contexts and N ∈{8, 16, 32}.Robots are not averaged, and known costs are retained while the predicted score replaces u(a) by bu(a).
  • E. Action-conditioned counterfactual value estimation: The exhaustive branch-supervision experiment separately probes planning-relevant information, with paired value MSE primary and Pearson correlation averaged within context–receiver groups over 25 plans.Training seed is the outer unit (n = 8), using an exact paired sign-flip test and a 100,000-resample bootstrap interval.

F. Fully decentralized dual-axis rigid-body control · VI. RESULTS · A. Compatible predictions emerge without consensus training

CS-JEPA produces compatible and informative decentralized predictions without agreement loss, output pooling, or consensus targets. In rigid-body control, receiver-local plans are independently selected and evaluated under communication faults, while fresh replication improves both agreement and accuracy.

  • F. Fully decentralized dual-axis rigid-body control: Matched action-conditioned predictors share the frozen encoder, recurrent backbone, anchor, communication, deployment capacity, and receiver-local planning procedure.The nominal comparator uses fixed flocking gains, while Future-Recon matches CS-JEPA’s candidate grid and value-head capacity.
  • F. Fully decentralized dual-axis rigid-body control: Each robot independently scores a 5 × 5 cohesion–cruise grid over a 0.8-s horizon, executes four steps, and replans without pooling predictions, scores, or plans.Communication remains a 256-byte recurrent message, with no centralized selector.
  • F. Fully decentralized dual-axis rigid-body control: The rigid-body evaluation tests N ∈{8, 16} under clean communication, packet loss, delay, jitter, and combined asynchronous faults in Crazyflie cf2x PyBullet dynamics.Each outer seed yields 32 paired episodes per swarm size and condition, using shared episode seeds and faults across methods.
  • VI. RESULTS: Every row and metric is positive in 12/12 seeds (p = 0.000488), favoring CS-JEPA over reconstruction-minus-JEPA error AUC.Table III reports reconstruction-minus-JEPA error AUC with 95% outer-seed confidence intervals.
  • A. Compatible predictions emerge without consensus training: CS-JEPA reduces decoded receiver-to-receiver disagreement relative to Future-Recon on every evaluated split and in all 12 independently trained stage-1 seeds without agreement loss or consensus targets.Compatibility emerges from shared target semantics rather than output pooling or an explicit consensus penalty.
  • A. Compatible predictions emerge without consensus training: 28.4% accuracy AUC falls on ID, while 64.4–75.6% falls under topology and size shift relative to Future-Recon.Accuracy improves on the same seeds and splits, ruling out agreement produced solely by collapse.
  • A. Compatible predictions emerge without consensus training: 18.1% equal-weighted accuracy AUC and 52.4% agreement AUC reductions persist without a shared absolute origin or task displacement in 12/12 seeds (p = 0.000488).Accuracy AUC falls from 0.008852 to 0.007248, and agreement AUC from 0.005735 to 0.002729.
  • A. Compatible predictions emerge without consensus training: The crossed stage-0 objective control remains positive across all four split estimates and all eight paired stage-0 seeds, with a −0.00789 [−0.01241, −0.00326] interaction (p = 0.03125).The interaction means the CS-JEPA advantage is larger, not smaller, after reconstruction pretraining; agreement improves in 8/8 seeds.

B. Action-conditioned value estimation · C. Latent-target prediction improves fully decentralized closed-loop utility

CS-JEPA preserves action-relevant information and improves decentralized decision quality without consensus training. Across closed-loop evaluations, it raises utility, task outcomes, connectivity, and receiver-local action agreement relative to matched reconstruction and nominal control.

  • B. Action-conditioned value estimation: CS-JEPA improves value MSE and within-context score correlation at both training sizes and unseen N = 32.The result supports action-relevant information beyond decoded collective variables.
  • C. Latent-target prediction improves fully decentralized closed-loop utility: 5.3% utility improvement over matched reconstruction occurs across clean and communication-fault episodes, favorable in 12/16 seeds.Task score and connectivity move in the same direction, with no collision or nearcollision in evaluated episodes.
  • C. Latent-target prediction improves fully decentralized closed-loop utility: Every seed improves utility, task score, and connectivity against nominal control when each robot acts from its own prediction.This links the training target to realized fully decentralized behavior.
  • C. Latent-target prediction improves fully decentralized closed-loop utility: 0.14472 [0.10282, 0.18618] higher first-action agreement and 0.05888 [0.04048, 0.07701] lower pairwise disagreement occur relative to Future-Recon.Both effects are favorable in 15/16 seeds, without a consistency loss.
  • C. Latent-target prediction improves fully decentralized closed-loop utility: 12/12 seeds favor CS-JEPA for compatibility and corresponding accuracy evidence against collapse in the fresh twelve-seed replication.Both effects are positive, with exact p = 0.000488.
  • C. Latent-target prediction improves fully decentralized closed-loop utility: At 6/12/24 globally labeled episodes, fresh-cohort accuracy is summarized across four families, while every paired family-wise AUC effect favors CS-JEPA in 12/12 seeds.The preregistered primary inference remains the paired AUC contrast in Table III.
  • C. Latent-target prediction improves fully decentralized closed-loop utility: Table VI evaluates fully decentralized dual-axis rigid-body control with n = 16 and defines ∆ as CS-JEPA minus comparator.Higher values are better, with confidence intervals and exact tests using training seed as the outer unit.

VII. DISCUSSION AND LIMITATIONS · VIII. CONCLUSION

CS-JEPA yields compatible receiver-local latent-future predictions without consensus training, while improving physical accuracy and preserving action-relevant information. The evidence is limited to simulator settings with shared orientation and scale, leaving hardware transfer and independent local frames open.

  • VII. DISCUSSION AND LIMITATIONS: Compatible receiver-local predictions emerge without consensus training, while physical decoding accuracy improves rather than converging to a constant.Each robot uses different evidence and recurrent state without reading another robot’s prediction.
  • VII. DISCUSSION AND LIMITATIONS: CS-JEPA isolates latent-target prediction from raw-future reconstruction by matching message width, deployment capacity, receiver anchor, and the absence of output pooling.Matched 64-D h_t probes place the compatibility effect before the final readout.
  • VII. DISCUSSION AND LIMITATIONS: Translation-free controls retain the advantage, while crossed reconstruction pretraining preserves and enlarges the stage 1 advantage.These controls indicate the gain comes from latent-target prediction rather than simply inheriting a JEPA-aligned representation.
  • VII. DISCUSSION AND LIMITATIONS: Improved counterfactual value estimates and rigid-body results show that the latent future retains action-relevant structure usable through a receiver-local interface.The rigid-body study covers clean and faulty communication.
  • VII. DISCUSSION AND LIMITATIONS: The evidence is simulator-only, retains shared orientation and scale, and covers only the reported graph and size families.Hardware transfer, independent local frames, and specialized controllers remain open.
  • VII. DISCUSSION AND LIMITATIONS: Improved value estimation does not reduce selected-plan regret, and privileged future state plus exhaustive branch labels are used only during training.These boundaries restrict the claim to compatible receiver-local prediction and demonstrated label-efficient and downstream utility.
  • VIII. CONCLUSION: Across fresh seeds, topology shifts, and swarm-size shifts, independent predictions become more compatible without agreement loss while accuracy improves and low-label probes preserve accessible shared state.Translation-free and crossed-objective controls weaken coordinate and objective-inheritance explanations.
Loading 2607.28443v3…