PaperlayerView exact revision on arXiv

Source-linked AI summary

Inside-JEPA: From Prediction to Relational Grounding in Latent World Models

Hossein Resani, Javen Qinfeng Shi

arXiv:2610.31194v1cs.LGcs.CVcs.RO

TL;DR

Latent world models can support planning, but their vector states are opaque and do not commit to relational meanings. Inside-JEPA adds relational channels to the state and trains them to explain actions through inverse dynamics without relation labels. Across video, robot, and tracked-object evaluations, this grounds channels when the inverse head reads them and supports relational forecasting and one-step action selection.

  • Problem

    Latent world models support planning but have opaque states, and relation probes do not commit the state to the relations used to express goals.

    Source: §1 · p. 2

  • Method

    Inside-JEPA places relational channels for ordered entity pairs inside the JEPA state and adds an inverse-dynamics loss that recovers actions from channel changes without relation labels.

    Source: §1 · p. 2

  • Results

    Grounding improves when inverse dynamics reads the channels: matched relational AUROC rises by .065–.103 across ten annotated-box settings, .081 on held-out-task robot episodes, and .050–.061 with tracked objects on five relations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Grounded channels forecast action-driven relational changes and support one-step relational-goal action selection, though their code is distributed and grounding is selective.

    Source: §1 · p. 2, §7 · p. 9

    The study compares against one prediction-only baseline with fixed architecture, uses one human-video and one robot dataset, and evaluates planning only for one-step choices within clips.

    Source: §6 · p. 9

Abstract

from arXiv · show

Joint-embedding predictive architectures (JEPAs) learn world models in latent space, enabling efficient planning but leaving their states opaque. We introduce Inside-JEPA, which places relational channels for ordered entity pairs inside the JEPA latent state and shows that an inverse-dynamics loss can ground these channels without relation labels when its head reads them, either alone or as part of the full state. On Something-Something V2, latent prediction alone performs no better than a random relation head, while inverse-dynamics supervision that recovers actions from channel changes improves relational AUROC by .065-.103 across ten settings with annotated boxes, by .081 on RH20T robot episodes with held-out tasks, and by .050-.061 across five relations using off-the-shelf detection and tracking. An inverse head on entity latents decodes actions more accurately but barely improves relational grounding; shuffled labels, two label-free objectives, and action labels on only 5% of the pairs also fail to ground the channels. Given an action, predicted relational changes approach the quality of observed future relations (AUROC .599 vs .621). For one-step relational goals, a supervised readout of the 16 channels ranks the correct action among the top five of 65 in 43% of cases, compared with 24% for prediction-only latents and 17% for a frequency prior.

1. Introduction

Latent world models support planning but their states remain opaque, and relation probes do not commit the state to those relations. Inside-JEPA puts relational channels in the latent state and tests whether inverse dynamics can ground them; across video, robot, and tracked-object settings, grounding improves when the inverse head reads the channels.

1. Introduction

  • Inside-JEPA places soft relational channels for every ordered entity pair inside the state, where the predictor consumes and forecasts them without relation labels.Its inverse-dynamics head recovers actions from the channels before and after an action.Source: §1 · p. 2
  • Inverse dynamics improves matched relational AUROC by .065–.103 across ten annotated-box settings, .081 on held-out-task robot episodes, and .050–.061 with tracked objects on five relations.Latent prediction alone remains at or near a random relation head; gains are positive in all 18 reported settings.Source: §1 · p. 2
  • Inverse-dynamics grounding requires correct action labels and a head that reads the relational channels; entity-only action decoding, shuffled labels, label-free objectives, and labels on 5% of pairs do not ground them.The entity-latent inverse head decodes actions more accurately but barely improves relational grounding.Source: §1 · p. 2
  • Given an action, forecasts of changing relations approach observed-future quality, while a readout of the 16 channels ranks goal-reaching actions in the top five better than prediction-only latents.The reported top-five rates are .425 versus .244, with a frequency prior at .174.Source: §1 · p. 2

2. Related work

Related work spans latent-predictive world models, object- and relation-structured dynamics, symbolic planning, inverse-dynamics supervision, and representation probes. Inside-JEPA differs by making pairwise relational channels part of JEPA dynamics and testing whether action supervision gives them meaning without relation labels.

Latent-predictive world models

  • JEPA and other latent world models plan with unstructured vector states, whereas Inside-JEPA asks what relational channels inside that state come to mean.Prior latent-predictive approaches include V-JEPA 2 and feature-space planning models.Source: §2 · pp. 2–3

Objects and relations in dynamics models

  • Object-centric dynamics and scene-graph methods model entities and relations, but scene-graph predicates are supervised; Inside-JEPA uses pairwise channels trained without relation labels as part of dynamics.Its learned-object experiments also try slots, then a detector and tracker.Source: §2 · p. 3

Symbols for planning

  • A related Something-Something method grounds predicates from action labels using hand-written PDDL conditions [33], while Inside-JEPA uses an inverse loss without those conditions or a separate classifier.The comparison varies training signals under a fixed architecture rather than running that method.Source: §2 · p. 3

Inverse dynamics as a learning signal

  • Prior inverse-dynamics work shapes latent features for control; Inside-JEPA instead uses the signal to give selected relational channels meaning and measures whether it succeeds.INV-Z also tests whole-latent inverse dynamics in the same framework.Source: §2 · p. 3

Concepts, probes and identifiability

  • Linear probes assess decodable information rather than training the representation, while label-free temporal objectives provide controls and relation-supervised channels serve as a ceiling.The paper also uses probe readouts for planning.Source: §2 · p. 3

3. Inside-JEPA

Inside-JEPA augments a JEPA world model with relational channels for ordered entity pairs and an inverse-dynamics objective that asks those channels to explain actions. Its controlled arms vary what the inverse head reads or whether its targets are shuffled, alongside label-free and supervised controls.

3. Inside-JEPA

  • Inside-JEPA combines a JEPA prediction loss with inverse dynamics on relational channels, without using relation labels.The state contains entity latents and one relational-channel vector per ordered entity pair.Source: §3 · p. 3

3.1. A relational latent state

  • Each training example pairs two clip frames, boxes for up to four entities present in both, and the intervening action; entities are encoded from cropped images and box coordinates.A relation head maps each ordered pair of distinct entities to soft channels; the predictor consumes and forecasts the resulting state.Source: §3.1 · p. 3

3.2. Latent prediction

  • The JEPA predictor forecasts the target encoder’s later latent from the current state and action, but prediction loss alone allows static or scene-unrelated channel codes.The target encoder is an exponential moving average of the online encoder.Source: §3.2 · p. 3

3.3. Inverse dynamics on the relational channels

  • The inverse head recovers the action from relational channels at both frames, sending its gradient through the relation and entity encoders.The design hypothesis is that changing relations such as contact and containment can help identify manipulation actions, but the loss does not assign individual relations to channels.Source: §3.3 · pp. 3–4

3.4. Arms and controls

  • Controlled arms keep architecture, data, budget, and loss weight fixed while varying whether the inverse head reads channels, the full latent, or entity latents, or uses shuffled targets.Arrow-of-time and InfoNCE are label-free controls; supervised channels provide a ceiling.Source: §3.4 · p. 4, Figure 2 · p. 4

4. Evaluation protocol

The evaluation measures whether individual relational channels align with box-computed relations, whether forecasts capture changing relations, and whether action rankings reach relational goals. It uses matched AUROC, controlled reference comparisons, and datasets spanning annotated boxes, robot episodes, tracked objects, and a simulated world.

Relations and matched relational AUROC

  • Matched relational AUROC assigns each relation a distinct channel using Hungarian matching, then evaluates channel scores against binary relations computed from entity boxes.The headline metric uses no sign flip and is strict about one channel per relation.Source: §4 · p. 4

Grounding metric

  • Grounding is assessed against both prediction-only PRED and a randomly initialized relation head, with the reported gain and trained-minus-random margin separating alignment from geometry available to every state.The matching is fitted on validation pairs; results are generally averaged over three seeds.Source: §4 · p. 4

Forecasting and planning

  • Forecasting scores predicted channels against future box relations only where a relation changes, comparing true-action forecasts with copying current channels, observed future channels, and permuted-action forecasts.The true-action minus permuted-action AUROC difference is the action effect.Source: Figure 3 · p. 5, §4 · p. 5
  • Planning ranks candidate actions by forecast-goal likelihood using relation readouts fitted on labelled training pairs, and reports whether the true action ranks in the top five against chance and a frequency prior.The main setup uses Something-Else clips about 3–4 seconds apart; RH20T holds out whole tasks, and tracked objects replace annotated boxes in that evaluation.Source: §4 · p. 5

5. Experiments

Across four experimental questions, the paper tests whether inverse dynamics grounds relational channels, what conditions support grounding, how grounded channels can be used, and where grounding remains limited. The experiments find that action supervision must reach the channels, that grounding varies by relation and setting, and that grounded channels support relational forecasting and planning.

5.1. Q1: Prediction does not ground the channels; inverse dynamics does

  • PRED’s channels score like a random relation head, while inverse heads that read the channels ground them; routing the inverse head only through entity latents barely helps.INV-Z and INV-S score .670 and .658, respectively, while INV-R scores .608 against PRED’s .593 on the main setup.Source: §5.1 · pp. 5–6, §5.2 · p. 6
  • + .065 to +.103 matched relational AUROC gains appear across ten annotated-box settings; gains are +.081 on RH20T and +.050 to +.061 with tracked objects.The matched comparison is between PRED and INV-S; the reported gains are positive across the evaluated settings.Source: Figure 4 · p. 6, §5.1 · p. 6

5.2. Q2: What grounds the channels

  • Grounding is selective: contact, proximity, and vertical relations gain, but left_of, much_larger, and hand relations do not; the pattern varies across datasets and object sources.Overlap and near gain across datasets and object sources, while left_of does not; other relations vary.Source: Figure 5 · p. 7, §5.2 · p. 7
  • With action labels on 25% of pairs (3,490; about 54 per template), INV-S’s grounding gain over PRED is +.047 (2 SE .009), versus +.066 with all labels.Source: §5.2 · p. 7

5.3. Q3: What grounded channels are good for

  • +.599 AUROC for action-conditioned forecasts of changing relations approaches .621 for observed future relations; a probe on the 16 channels ranks relational goals in the top five at .425.For relational planning, the channel probe exceeds PRED’s whole-latent probe (.244); on RH20T, the authors limit their relational-planning claim because positional goals and little headroom affect results.Source: §5.3 · pp. 7–8, Figure 6 · p. 8

5.4. Q4: Where grounding stops

  • left_of never grounds in real-video settings despite being readable from the latents, and tested explanations for the gap were unsupported.The one-channel-per-relation readout plans at .334 versus .425 for a probe on all 16 channels, indicating that the code is distributed.Source: §5.4 · p. 8
  • Inverse dynamics recovers only part of the supervised grounding ceiling, yet inverse-trained channels plan as well as supervised channels through a probe.With a frozen V-JEPA 2 encoder, the gain is smaller, especially with video context; the paper reports no single cause for the smaller still-frame gain.Source: §5.4 · p. 8

6. Discussion and limitations

The discussion finds that inverse dynamics can align relational channels with box-defined relations when the inverse head reads them and action labels are available. The evidence is bounded by the tested baselines, data, horizons, and planning setting.

6. Discussion and limitations

  • Grounding aligns channels with box-defined relations across caption-labelled video, robot episodes, and tracked objects, more weakly in the tracked-object setting.Grounding requires the inverse head to read the channels and correct action labels on thousands of pairs.Source: §6 · p. 9
  • The measured grounding does not establish physical relations beyond box-defined relations, and the experiments do not establish why particular relations ground.The design rationale fits Something-Else but not RH20T; two tested label-free objectives failed, without ruling out all such objectives.Source: §6 · p. 9
  • The study compares against PRED with fixed architecture, uses one human-video and one robot dataset, and evaluates horizons within clips of about four seconds.On RH20T, boxes were correct in 34 of 40 hand-checked frames; the Something-Something action is a caption.Source: §6 · p. 9
  • Effects are modest and weaker with tracked objects and V-JEPA 2; the planning test covers one-step action selection within clips on Something-Else only.The channels are a distributed code rather than clean predicates, and the planning readouts are fitted on relation-labelled pairs.Source: §6 · p. 9

7. Conclusion

The conclusion argues that relational channels do not gain meaning from latent prediction alone. Requiring them to explain actions aligns them with relations and supports relational forecasting and goal-directed action selection within the tested scope.

7. Conclusion

  • Inverse-dynamics supervision aligns relational channels with relations across caption-labelled video, robot episodes, and automatically found objects, given readable channels and enough action labels.The channels forecast action-driven relational changes and carry a signal for selecting actions toward relational goals, selectively and as a distributed code.Source: §7 · p. 9

in Latent World Models

The appendix outlines implementation and evaluation details, then specifies the model, training setup, and frame horizons used in the experiments.

in Latent World Models

  • The appendix covers implementation and evaluation, additional grounding results, the controlled 2-D world, detector-tracker objects, robot episodes, label controls, planning, V-JEPA 2, and the supervised ceiling.It also lists decision rules C1–C6.Source: in Latent World Models · p. 12

A. Implementation details

  • The model encodes cropped entities and box geometry into 32-dimensional entity latents, then produces 8 or 16 sigmoid channels for each ordered pair.The entity encoder includes a frozen ImageNet ResNet-18, box-coordinate features, and entity type; padded-entity and self-pair entries are held constant.Source: A. Implementation details · p. 12
  • Training uses AdamW with learning rate 2×10^-4, weight decay 10^-4, batch size 64, and λ_inv = 1, with fixed budgets across experimental arms.Experiments use three to five seeds, and reported intervals reflect seed variance only.Source: A. Implementation details · p. 12
  • Something-Something frames span about 3–4 seconds, while RH20T uses frames 30 steps apart, corresponding to three seconds at 10 Hz.Something-Something frames are extracted at the native 12 fps.Source: A. Implementation details · p. 12

B. Evaluation details

Evaluation relations are computed from normalized entity boxes, with matching and forecasting metrics defined for ordered pairs. The appendix also specifies how planner candidates are ranked and calibrated.

B. Evaluation details

  • Matched relational AUROC uses validation pairs and Hungarian channel-to-relation matching; a random-head reference uses a freshly initialized relation head on the checkpoint’s entity embeddings.The random head scores .52–.60 per relation, and the probe is a linear classifier on arm-specific pair features.Source: B. Evaluation details · pp. 12–13
  • Relations are computed for each distinct ordered entity pair from normalized box geometry, with five- and thirteen-relation sets and explicit rules for distance and hand relations.Figure 7 illustrates the directed pairs and the geometric features used by the relation rules.Source: B. Evaluation details · p. 12, Figure 7 · p. 13
  • Forecasting metrics compare predicted future channels with future relations, including entries whose relation changes; the action effect contrasts true and permuted actions.References include copying current channels, observed future channels, and a forecast with validation actions permuted.Source: B. Evaluation details · p. 13
  • Planners rank candidate templates by forecast-channel distance, calibrated goal likelihood, or an amortized inverse head, with readouts fitted on labelled training pairs.Raw distance scoring underperformed the prior in some settings; calibrated Chan and Inv improved scores in the 40-template, thirteen-relation analysis.Source: B. Evaluation details · p. 13

C. Additional grounding results

Additional grounding results test the method across template counts, relation groups, and a controlled 2-D world. Inverse supervision's effect varies by relation and data scale, and action recovery from entity latents alone does not suffice to ground channels.

C. Additional grounding results

  • In the first real-video test, C1 was +.097 and C2's margin gain was +.092 with the seeded random head.C2's margin gain was +.116 with the original unseeded random head.Source: C. Additional grounding results · p. 13, Table 4 · p. 14
  • An inverse head on entity latents decoded actions best on the main setup (top-1 .424), yet its channels scored .608 versus .593 for PRED.By comparison, INV-S reached .316 top-1 from channels; the reported result distinguishes accurate action decoding from relational grounding.Source: C. Additional grounding results · p. 14
  • On 65 templates, the eleven non-hand relations gained +.100 ± .008 together, while the two hand relations scored −.128 ± .088.The reported gains are over PRED; the hand relations account for a distinct, weaker pattern.Source: C. Additional grounding results · p. 14
  • Across six settings, overlap and near gained in every setting, far wherever scored, left_of in none, and inside in five of six.On 174 templates, inside was the exception; it gained −.046, while left_of gained −.013.Source: C. Additional grounding results · pp. 14–15, Figure 8 · p. 14
  • The 13-relation grounding gain shrank by .010 per doubling across five training-set sizes, as PRED learned the hand relations.The reported non-hand slope was +.011 per doubling, versus −.105 for hand relations.Source: C. Additional grounding results · p. 14, Table 7 · p. 15
  • On all 174 templates, INV-S reached .691, while INV-R matched PRED at .622 vs. .623; the eleven non-hand relations gained +.088 and the two hand relations −.045.The one-channel readout Chan fell to the frequency prior on this setting (.081 vs. .077).Source: C. Additional grounding results · p. 15

D. Blocks2D

  • In Blocks2D, inverse dynamics raised relational AUROC to .758 IID, .738 on held-out object counts, and .775 on tall stacks, versus .522 for prediction-only IID.Adding vertical lifts left IID grounding essentially unchanged: .757 ± .010 with horizontal-only actions and .755 ± .016 with lifts.Source: D. Blocks2D · p. 15, Table 8 · p. 15

E. Detector-and-tracker objects

This section evaluates grounding with detector-and-tracker objects and examines where their gap to annotated boxes comes from. Results also test how tracking quality, object ordering, and prompting affect this setting.

E. Detector-and-tracker objects

  • On fixed validation pairs, missed objects accounted for +.044 ± .013 of the +.075 annotated-box gap, distractor tracks +.020 ± .013, and loose boxes +.011 ± .004.This five-seed breakdown was post hoc; missed objects, distractors, and loose boxes account for 59%, 27%, and 14% of the gap, respectively.Source: E. Detector-and-tracker objects · p. 16, Figure 10 · p. 16
  • Keeping a fifth track recovered objects but increased distractors from 42% to 52% of kept tracks, while leaving the gain essentially unchanged at +.083 versus +.080.Of the entities not kept in the four tracks, 1,209 were never detected and 1,208 were lost in tracking.Source: E. Detector-and-tracker objects · p. 16
  • With tracked objects, the shuffled-label control behaves like PRED, while true actions lift channel scores .045–.059 above it.The comparison is presented as cleaner than PRED alone because PRED's channels score below its random head in the five-relation runs.Source: E. Detector-and-tracker objects · p. 16
  • Randomizing object order with the hand last cost .027, and moving the hand first cost about .025; stable role order mattered more for INV-S than INV-Z.INV-S moved from .600 to .651 across the three tested orders, while INV-Z ranged from .627 to .640.Source: E. Detector-and-tracker objects · p. 17
  • On all clips, tracked-object grounding gained +.050; the tracker-handled half gained +.055, versus +.033 for the other half.Annotating 10% or 25% of training videos added nothing measurable in either format.Source: E. Detector-and-tracker objects · p. 17
  • With the prompt “object.” alone, the detector found 28% of annotated hands, probes fell to about .72, and the gain on found pairs was +.054.The run was considered uninformative because the probe fell below the L4 bar.Source: E. Detector-and-tracker objects · p. 17

F. RH20T robot episodes

On RH20T robot episodes, inverse-dynamics supervision grounds relational channels, but the reported planning evidence supports relational goals only on Something-Else. Label-free objectives and sparse action labels fall short, while relational information is distributed across channels rather than aligned one-to-one with relations.

F. RH20T robot episodes

  • +.081 grounding gain on RH20T robot episodes; the result rests on supplied boxes, whose dependence remains untested.With one camera per setup, the gain is +.046; left_of still does not ground.Source: F. RH20T robot episodes · p. 17
  • RH20T planning from relational goals reaches .371 top-5 against PRED’s .234, but positional goals and entity latents limit the relational-planning claim to Something-Else.The relational channels hold .64 of INV-S’s planning information on RH20T, versus .17 for PRED.Source: F. RH20T robot episodes · pp. 17–18

G. Label-free objectives and label fractions

  • Arrow-of-time and InfoNCE objectives gain −.007 and +.010 over PRED, respectively, and both fall short of INV-S.On contact and proximity as a group, they gain +.035 and +.029, against +.138 for inverse dynamics.Source: G. Label-free objectives and label… · p. 18
  • Grounding gain stays flat through 5% action labels and grows by .011 per doubling; at 25%, planning rules hold but forecasting narrowly misses its bar.Contact and proximity recover most of their full-label gain by 25%, while above, inside, and strictly_above need the full label set.Source: G. Label-free objectives and label… · p. 18, Table 13 · p. 18

H. Planning and the distributed code

  • The 16-channel probe plans at .425 versus .334 for the one-channel readout, showing relational information is spread across channels.Three label-free attempts to align each relation with its own channel failed.Source: H. Planning and the distributed code · p. 18
  • Decorrelation reduces mean absolute off-diagonal correlation from .24 to .014 but lowers one-channel readout AUROC by .037 and grounding gain to +.028.The code remains available to the full-channel probe; sparsity, by contrast, destroys grounding and planning.Source: H. Planning and the distributed code · p. 19
  • Across rollouts, the action effect grows from +.021 after one step to +.057 after three, but shorter-step training falls below the grounding bar.Planning from relational goals with shorter-step training does not beat the prior.Source: H. Planning and the distributed code · p. 19

I. Frozen V-JEPA 2 encoder

With a frozen V-JEPA 2 encoder, the inverse-dynamics grounding gain persists on unseen videos, though video context shrinks it. The supervised ceiling shows inverse dynamics recovers only part of label-trained grounding, and seeded evaluation stabilizes margin comparisons.

I. Frozen V-JEPA 2 encoder

  • INV-S’s margin gain over PRED is +.042 ± .010 on unseen videos and +.045 ± .016 on matched seen samples, so the gain is unchanged.The seen and unseen sets also come from different SSv2 splits, combining exposure and split effects; the V-JEPA 2 ceiling was not re-scored and may be optimistic.Source: §I · p. 19
  • +.042 ± .015 matched-AUROC gain at five seeds with still frames; video context reduces the gain to +.016.With video context, INV-S remains at .640–.641 while PRED rises to .619–.625.Source: §I · pp. 19–20

J. Supervised ceiling

  • Inverse dynamics recovers .160 ± .032 of the supervised ceiling with one-channel matching and .371 ± .011 with a channel probe on ResNet-18.On V-JEPA 2, the corresponding shares are .123 ± .067 and .252 ± .067.Source: J. Supervised ceiling · p. 20, Figure 13 · p. 20
  • Without box coordinates, label-trained V-JEPA 2 channels still read relations missed by inverse dynamics at .988–.998, indicating the miss is in the objective, not the encoder.The passage cautions that this measures position available through positional embeddings, not relational understanding.Source: J. Supervised ceiling · p. 20

K. Pre-registration record

  • Seeding the random head reproduced matched AUROC and changed no overall verdict, while roughly halving the 2 SE of margin-gain comparisons.The audit rescored 812 real-video checkpoints; two tracker evaluations could not be rescored because track alignments were not stored.Source: K. Pre-registration record · p. 21

Related papers

Loading 2610.31194v1…