PaperlayerView exact revision on arXiv

Source-linked AI summary

D-JEPA: A Decision-Aligned Latent World Model

Shuaijun Liu, Chengyu Wu, Qifu Wen, Feiyang You, Chenglong Zhang, Shuyang Hao, Xi Lin, Ningxin Su

arXiv:2609.24749v1cs.ROcs.LG

TL;DR

Latent world models can misrank the few candidate futures that determine an action, even when their predictions are informative. D-JEPA learns decision-relevant relations from executed outcomes, and evaluations report improved action selection across control, manipulation, pretrained models, robots, and driving.

  • Problem

    Among the few futures competing for execution, smaller predicted goal distance can favor a failing action over a successful alternative.

    Source: §1 · p. 2

  • Method

    D-JEPA learns a bounded relational correction over candidate futures, jointly processing goal-relative predictive features and ordinal evidence from executed outcomes.

    Source: §1 · p. 2

  • Results

    Across evaluations, D-JEPA improves action selection; across four RoboTwin tasks, average native VLA success rises from 61.72% to 76.76%.

    Source: §6.1 · p. 9, §7 · p. 10

  • Takeaways & Limitations

    The findings connect predictive world models to effective control by aligning latent geometry with realized action consequences.

    Source: §7 · p. 10

Abstract

from arXiv · show

Latent world models predict the consequences of actions, but accurate prediction does not guarantee that latent distance reflects which candidate will execute successfully. We identify a decision-local prediction gap: among the few futures competing for execution, a candidate predicted closer to the goal can produce a worse realized outcome than an available alternative. We introduce D-JEPA, a decision-aligned latent world model that learns decision-relevant relations among candidate futures from executed outcomes. A bounded, permutation-equivariant operator jointly reasons over goal-relative predictive features and ordinal evidence, refining pretrained predictive geometry where action choices are most consequential. Restricted predictor adaptation and a shared ordinal interface extend this alignment across complementary predictive geometries. D-JEPA further realizes the learned decision structure in JEPA-compatible future representations, enabling deployment through native latent-distance planning. Evaluations across latent control, manipulation, pretrained action-producing models, physical robots and autonomous driving demonstrate improved action selection, including 87.89% success on PushT, a 15.04-point average gain on RoboTwin, and a 17-point gain on physical robot tasks. These results establish decision-relevant relational structure as a direct bridge between predictive world modeling and effective control.

1 INTRODUCTION

D-JEPA addresses the decision-local gap between predicted goal distance and executed outcomes by learning relations among competing futures and realizing the learned decision structure in JEPA-compatible future geometry.

1 INTRODUCTION

  • On matched PushT decisions, within-start Spearman correlation falls from 0.90 to 0.11 for LeWM and 0.80 to 0.13 for TD-JEPA as the candidate set narrows to the preferred four (Figure 2).Executed candidate outcomes supervise relations among the few futures competing to determine the next action.Source: §1 · p. 2
  • D-JEPA uses a bounded, permutation-equivariant operator to jointly process goal-relative predictive features and ordinal evidence across the complete candidate set.The operator sharpens distinctions near the decision boundary.Source: §1 · p. 2
  • Restricted predictor adaptation complements the relational operator, while a common ordinal interface integrates evidence across heterogeneous predictive geometries.Source: §1 · p. 2
  • D-JEPA realizes learned decision structure in JEPA-compatible future representations through bounded temporal transport and exact ordinal realization in terminal goal-relative geometry.The latter makes aligned action choices available through native latent-distance planning.Source: §1 · p. 2
  • The framework is evaluated across latent control, robotic manipulation, pretrained action-producing models, physical robots, autonomous driving, and geometric and visual shifts.Matched ablations isolate the core relation operator and complementary predictive evidence.Source: §1 · p. 2

2 RELATED WORK

Prior work develops predictive futures and makes predictive geometry relevant to control; D-JEPA instead learns which distinctions among competing futures matter for execution. Its shared alignment interface preserves predictive geometries while transferring decision-relevant ordinal evidence across them.

2 RELATED WORK

  • D-JEPA learns decision-relevant relations among predictive futures that compete for execution, beyond the latent prediction and planning capabilities of PlaNet, Dreamer, TD-MPC, joint-embedding methods, V-JEPA 2, DINO-WM, and LeWM.PlaNet, Dreamer, and TD-MPC support latent planning or behavior learning (Hafner et al., 2019; 2025; Hansen et al., 2024); joint-embedding methods predict target embeddings (Assran et al., 2023), while V-JEPA 2, DINO-WM, and LeWM extend predictive world modeling (Assran et al., 2025; Zhou et al., 2025; Maes et al., 2026).Source: §2 · p. 2
  • Unlike prior approaches that shape predictive geometry or connect plan quality to cost fidelity, D-JEPA trains a complete-set operator from executed consequences and exposes its learned order through a latent readout.TD-JEPA mines directed temporal progress and studies task-dependent deployment of temporal and Euclidean costs (Bai & Xiong, 2026); A Control Theory of Predictability connects plan quality to cost fidelity on planner-reachable candidates (You et al., 2026).Source: §2 · p. 3
  • D-JEPA aligns complementary predictive sources by keeping dense descriptors within each geometry and sharing ordinal coordinates across heterogeneous geometries, preserving source-geometry semantics while improving action selection.The comparison includes JEPA-WM’s study of architectures, objectives, and planning choices (Terver et al., 2026) and Fast LeWorldModel’s action-prefix prediction operator (Gao & Xu, 2026).Source: §2 · p. 3
  • Complementing methods that learn action distributions from demonstrations, D-JEPA selects among available predictive futures for execution across latent planners, pretrained VLA action chunks, Drive-JEPA trajectories, and V-JEPA 2-AC candidates.Diffusion Policy learns action distributions directly from demonstrations (Chi et al., 2023).Source: §2 · p. 3

3 THE DECISION-LOCAL PREDICTION GAP

Predictive latent geometry can remain informative overall yet fail to distinguish execution-relevant candidates. The section motivates learning decision relations from counterfactual outcomes while retaining predictive context.

3 THE DECISION-LOCAL PREDICTION GAP

  • Across 96 PushT starts, each model preferred a failing candidate on eight despite an available successful alternative; mixed-outcome top-four inversion rates reached 49.0% for LeWM and 38.2% for TD-JEPA.A recorded pair also shows LeWM assigning a smaller predicted goal RMS distance to a failing candidate than to a successful alternative.Source: §3 · p. 3
  • Within-start Spearman correlation between predicted and realized costs fell below 0.13 for both models among their four lowest-cost candidates.Predictive geometry remained informative across the full candidate set but weakly distinguished actions closest to execution.Source: §3 · p. 3
  • Training-time counterfactual execution identifies successful and unsuccessful alternatives, teaching their decision relations while fixed candidate pools hold action availability constant.At deployment, context-derived predictive evidence supports action selection, and executed outcomes are evaluated independently; the relational formulation generalizes to candidate sets of size K.Source: §3 · p. 3
  • Global predictive agreement alone cannot guarantee correct action selection, so the approach retains complete-set predictive context while focusing supervision on decision-relevant candidates and bounding representational or score changes.The section states this limitation as Proposition 1.Source: §3 · p. 3

4 D-JEPA

D-JEPA aligns pretrained predictive geometry with executed outcomes by reasoning over relations among candidate futures, combining complementary predictive evidence, and realizing the aligned structure in future latents. This enables decision-aligned choices through native latent-distance planning.

4.1 PREDICTIVE EVIDENCE AND BOUNDED RELATIONAL ALIGNMENT

  • A permutation-equivariant set operator combines goal-relative descriptors and scale-free ordinal evidence, then applies a bounded correction to predictive scores.The correction preserves preferences across base-score gaps above 2ϵ and selects within 2ϵ of the base minimum; objectives sharpen local success/failure ordering while regularizing unnecessary global reordering.Source: §4.1 · pp. 4–5

4.2 COMPLEMENTARY ADAPTATION AND PREDICTIVE GEOMETRIES

  • Predictive plasticity adapts only the final TD-JEPA predictor block and projection, while a shared ordinal interface extends relational alignment across additional predictive geometries.A calibrated margin gate accepts the plastic native-distance proposal over the relational default when its score advantage exceeds a threshold; the four-model instance appends JEPA-WM and DINO-WM ranks, and task-specific evidence also supports alignment over action-producing foundations.Source: §4.2 · pp. 5–6

4.3 REPRESENTATION LIFTING

  • D-JEPA realizes the learned decision structure in future latents, making aligned actions recoverable through native goal distance and planning.Its ordinal branch sets terminal goal-distance radii from strict ranks while leaving earlier trajectory steps unchanged; a successful future can be identified even when both original predictive criteria prefer failing actions.Source: Figure 5 · p. 5, §4.3 · p. 8

5 EXPERIMENTS

The experiments test decision alignment across physical dynamics, pretrained action-producing models, robots, driving, distribution shifts, and native future-latent planning. Shared evaluation protocols compare methods fairly across tasks.

5 EXPERIMENTS

  • Evaluations span PushT, DMC-Reacher, Granular goal shaping, RoboTwin, PiPER manipulation, driving scenes, unseen PushObj shapes, and fresh PushT starts.The study also evaluates a separate PushT mechanism population and tests native future-latent geometry.Source: §5 · p. 8
  • Methods share starts, goals, candidate actions, and execution horizons, with candidate executions restoring the same initial state.Disjoint fitting, calibration, and evaluation identities separate learning, model selection, and testing.Source: §5 · pp. 8–9
  • Controls include pretrained models, calibrated fusion, matched optimization, uncertainty baselines, and task-specific preservation variants.Physical-robot and driving comparisons retain the reference planner and search budget; success is measured over starts or paired physical trials, with an equal-weight scene mean for driving.Source: §5 · p. 9

6 RESULTS

D-JEPA improves action selection across latent control, manipulation, physical robots, and driving, including transfer to unseen shapes and visual conditions. Ablations and future-latent realizations connect these gains to relational alignment and predictive evidence.

6.1 ACTION SELECTION ACROSS PHYSICAL TASKS

  • Across four RoboTwin tasks, average native VLA success rises from 61.72% to 76.76%, while two physical PiPER tasks rise from 64.0% to 81.0%.The physical-task comparison uses the same V-JEPA 2-AC planner.Source: §6.1 · p. 9
  • D-JEPA reaches 87.89% success on PushT and 93.75% on Reacher under identical candidate availability, outperforming pretrained predictive models and matched adaptation controls.PushT benefits from calibrated composition, whereas Reacher improves through relational selection; Granular manipulation shows its largest gain at the strict goal threshold.Source: §6.1 · p. 9
  • In seven focused driving scenes, mean PDMS rises from 57.36 to 95.34 under shared candidates, with aligned trajectories maintaining more clearance as traffic evolves.Figure 8 illustrates the clearance behavior.Source: §6.1 · p. 9

6.2 ALIGNMENT UNDER SHAPE AND VISUAL CHANGES

  • D-JEPA improves every held-out PushObj geometry and evaluated appearance condition, with object-colour change yielding the largest appearance gain.These evaluations test unseen shapes and fresh starts within appearance families used for task-local fitting.Source: §6.2 · p. 9

6.3 ABLATIONS OF ALIGNMENT AND PREDICTIVE EVIDENCE

  • Relational alignment is the strongest individual matched-ablation intervention, while predictive adaptation improves PushT confirmation success through calibrated composition.Relational alignment outperforms fixed correction and scalar confidence gating, and a preservation gate adds further benefit.Source: §6.3 · p. 9, §6.1 · p. 9
  • On the 128-start mechanism population, dual LeWM and TD-JEPA evidence reaches 93.75%, versus 84.38% and 83.59% from either source alone.The same relational operator incorporates four predictive geometries and achieves the success rate of calibrated four-source fusion; performance improves with greater candidate availability.Source: §6.3 · p. 9

6.4 REALIZING DECISIONS IN FUTURE GEOMETRY

  • Exact ordinal realization encodes learned relational order in future geometry, so native goal distance reproduces aligned decisions at comparable latency.On Reacher, temporal transport learns signed updates across five future steps within the prescribed bound.Source: §6.4 · p. 9

7 CONCLUSION … A.4 EXPANDED POPULATIONS AND TIMING

D-JEPA addresses misleading goal distances among competing candidate futures by learning bounded, set-wise relations from executed outcomes, then realizes the learned structure in JEPA-compatible representations for native latent-distance planning. The appendices specify evaluation populations and candidate construction, relational and predictive learning, task-specific alignment and transport, and timing protocols.

7 CONCLUSION

  • D-JEPA learns decision-relevant relations from executed outcomes to refine predictive geometry where candidate choices matter, enabling aligned action selection through native latent distance.The conclusion reports evaluations spanning latent control, manipulation, pretrained action models, physical robots, and driving scenes, with further gains under geometric and visual changes.Source: §7 · pp. 9–10

A.1 EVALUATION UNITS AND CANDIDATE CONSTRUCTION

  • The PushT confirmation cohort comprises 256 starts disjoint from fitting, calibration, and prior development, selected by a preestablished rule for nontrivial candidate boundaries.Methods share starts, goals, candidate availability, and execution duration; task populations are reported separately, and statistical units follow each task protocol.Source: §A.1 · p. 12

A.2 RELATIONAL AND PREDICTIVE LEARNING

  • The relational model uses a bounded rank-eight correction initialized to reproduce the base score, trained on a 16-candidate decision subset with fitting, calibration, and confirmation kept separate.Fusion is fitted over a 101-point grid; relational weights use the fitting subset, while checkpoint and gate selection use 64 calibration starts before final evaluation on the confirmation cohort.Source: §A.2 · p. 12
  • Predictor adaptation updates only the final predictor block and projection while preserving teacher trajectories and score relationships outside the targeted decision subset.The loss combines latent MSE, success/failure boundary ordering, non-tail distributional KL, and score-difference consistency; the adapted predictor’s standalone readout minimizes native goal cost.Source: §A.2 · p. 12

A.3 TASK-SPECIFIC ALIGNMENT AND TEMPORAL TRANSPORT

  • Task-specific alignment combines visual, proprioceptive, and native-joint ranks, while temporal transport uses source ranks, inferred relational rank, gate state, and time to refine candidate ordering.Physical labels define training targets, but deployment uses those rank and gate inputs; physical success is evaluated on a separate 128-start test population using the relational score for action selection.Source: §A.3 · pp. 12–13

A.4 EXPANDED POPULATIONS AND TIMING

  • Core populations span PushT, Reacher, Granular, RoboTwin, PiPER, and seven driving scenes, with shared candidate availability for core model comparisons.The protocol uses 256 independent PushT starts, 128 Reacher starts, 64 Granular starts, 128 starts per RoboTwin task, 50 paired PiPER trials per task, and 32 trajectories per driving scene.Source: §A.4 · p. 13
  • Expanded evaluations include PushObj shape confirmation and appearance-change tests, with fitting and calibration separated from new evaluation starts.PushObj uses shapes T, L, Z, and + for fitting and calibration; confirmation combines two 150-start sets and includes 100 starts each for I, small T, and square. Appearance checkpoints are evaluated on 50 new base starts under each of seven known conditions.Source: §A.4 · p. 13
  • Timing compares end-to-end inference per start on the same 256 label-free PushT inputs, including configured predictive and relational paths.Table 6 uses batch size 16, two warm-up repeats, and ten timed repeats; the reported checkpoint-file counts are three, four, and one, and logical package counts are four, five, and one.Source: §A.4 · p. 13

A.5 METRIC DEFINITIONS AND STATISTICAL UNITS · A.6 BASELINE MECHANISMS AND COMPARISON IDENTITIES · A.7 NOTATION AND INFERENCE CORRESPONDENCE

These appendices define how outcomes, metrics, and statistical units are computed; specify the mechanisms and identities of comparison baselines; and map the paper’s notation to decision and inference quantities.

A.5 METRIC DEFINITIONS AND STATISTICAL UNITS

  • Success is the selected candidate’s executed binary outcome rate, while paired improvement is 100(G − L)/n percentage points with shared outcomes retained in the denominator.Shape groups use distinct starts, appearance conditions reuse 50 base identities, PiPER averages two task success rates equally, RoboTwin pools four equally sized task populations, and driving averages seven scene-level PDMS values equally on a 0–100 scale.Source: §A.5 · p. 13
  • Shortlist latent-cost alignment is measured by within-start or pooled Spearman coefficients, neither of which correlates latent costs with binary success labels.Each model retains its k lowest predicted-cost candidates per start; the within-start statistic averages 96 coefficients, while the pooled statistic concatenates shortlisted predicted/realized cost pairs before computing one coefficient.Source: §A.5 · p. 13
  • Attainment thresholds are 0.18 for main and 0.12 for strict, and physical stacking requires support after arm withdrawal rather than grasping or lifting alone.The thresholds apply to the same quantity; single-method intervals are Wilson 95% intervals, paired differences use 10,000 bootstrap resamples of whole paired starts, candidate-budget bands span extrema over 16 shared subset seeds, and latency is median end-to-end inference time per start excluding simulator execution.Source: §A.5 · p. 13

A.6 BASELINE MECHANISMS AND COMPARISON IDENTITIES

  • Table 1 separates native-model planning criteria scored in each model’s predictive space from optimization or stochastic-planning controls on the shared JEPA decision problem.Fusion and task-specific controls include four-geometry fusion without learned set-wise correction, visual/proprioceptive rank fusion, RoboTwin mechanism ablations, and native action-producing controls with shared candidate/search budgets; Candidate Oracles quantify headroom using executed outcomes, unlike pre-execution learned selectors.Source: §A.6 · p. 14
  • The PEGrad-inspired control projects conflicting auxiliary ranking gradients and caps their norm, while Var-JEPA-inspired adaptation trains an action-conditioned trajectory code and residual decoder.The variational control uses a target-conditioned training posterior and prior at inference; conflict-safe variational adaptation adds conflict-only projection for reconstruction/KL gradients and is a combined control, not an additional pretrained backbone.Source: §A.6 · p. 14
  • Stochastic controls sample 16 prior futures per candidate and choose by either mean terminal cost or the mean of the two largest costs.The latter is empirical upper-tail CVaR at 0.9; posterior information is training-only, persistent candidate IDs resolve deployment-score ties, and the controls instantiate sampling-based planning (Enwerem et al., 2026), with CVaR related to Ni et al. (2024).Source: §A.6 · p. 14

A.7 NOTATION AND INFERENCE CORRESPONDENCE

  • Section 4 notation uses K for candidate count, H for horizon, D for latent dimension, and n for evaluation starts, with superscripts identifying the four predictive geometries.Normalized score i ∈ [0, 1] prefers smaller values, and final strict rank π_i includes the calibrated gate; rank recovery, selected-action agreement, and success respectively measure ordering, decision, and executed-goal agreement.Source: §A.7 · p. 14

B DECISION-ALIGNMENT OBJECTIVES AND TASK INTERFACES … C COMPLETE NUMERICAL DETAILS

The section details how D-JEPA learns decision-relevant relations among candidate futures and integrates them into predictive models and task interfaces. It also describes realizing aligned decisions through native latent distances and reports improved RoboTwin success.

B.1 COMPLETE-SET EVIDENCE AND LEARNING OBJECTIVES

  • The dual-model token comprises two separately normalized 192-dimensional descriptors and two within-set ranks (386 inputs); two candidate-order-position-free Transformer layers yield permutation-equivariant outputs, with the final correction projection initialized to zero.A shared encoder maps inputs to 64 dimensions; the correction head is 64→8→1.Source: §B.1 · p. 14
  • The full-list objective rewards any successful alternative, while refreshed low-score subsets focus pairwise supervision; inference still processes every candidate and applies a calibrated relational-winner gate.If a subset contains only one outcome class, pairs are drawn from the complete set; the base action is retained unless the relational winner clears the calibrated margin threshold.Source: §B.1 · pp. 14–15
  • Predictive plasticity adapts only the final predictor block and projection, using boundary ordering, rank preservation, and latent consistency to limit unnecessary trajectory changes.The adapted future’s native goal distance supplies its standalone score; in composition, the relational choice is replaced by the predictor choice only when the calibrated score margin exceeds its threshold.Source: §B.1 · p. 15

B.2 PREDICTIVE GEOMETRIES AND ACTION-PRODUCING MODELS

  • Across predictive geometries, ordinal coordinates provide a shared comparison interface while dense directional features remain within each space; shape and appearance tasks use visual, proprioceptive, and native-joint ranks.The four-geometry PushT operator refines the JEPA-WM base score; shape and appearance use a 3→64 encoder, the same two-layer set operator, and task-local calibrated fusion.Source: §B.2 · p. 15
  • RoboTwin compares a native VLA future with 16 spatial/temporal variants and writes the selected relation back to its JEPA-compatible future/action interface; Drive-JEPA and PiPER re-rank candidate trajectories while preserving native models and controls.RoboTwin uses a learned preservation gate; Drive-JEPA retains the native trajectory unless corrected margins support replacement, and PiPER leaves the predictor, search budget, and low-level controller unchanged.Source: §B.2 · p. 15

B.3 FUTURE-LATENT REALIZATION

  • Reacher uses a zero-initialized 5→32→1 coefficient network with ρ = 0.1 to update corresponding action-time indices, while PushT encodes gated candidate order in terminal latent radius and preserves rank identity at deployed precision.The self-contained PushT checkpoint embeds both predictive models and relational computation, exposing the aligned decision through native latent distance.Source: §B.3 · p. 15
  • 15.04 points: D-JEPA improves RoboTwin overall success over the no-preservation variant, whose overall success is 71.48%.The evaluation uses each task’s full completion criterion and 128 independent test starts per task; Oracle is the paired candidate-outcome ceiling.Source: §B.3 · p. 16, Table 7 · p. 16

C.1 DECISION-LOCAL RANKING DIAGNOSTICS … E ADDITIONAL PHYSICAL VISUALIZATIONS

The supplementary analyses characterize decision-local ranking errors, test performance across geometries and candidate budgets, and describe how learned preferences are realized in future representations. They also provide visual context for matched physical executions and formal task outcomes.

C.1 DECISION-LOCAL RANKING DIAGNOSTICS

  • All 96 audit starts contain successful and failing candidates, and each model has eight starts where the lowest-distance failing candidate outranks every successful alternative.A positive normalized gap indicates this decision-local prediction error; the recorded LeWM illustration has a failing candidate at 0.2027 and a successful alternative at 0.2167.Source: §C.1 · p. 16
  • At K = 4, mean inversion percentages are 49.02% for LeWM and 38.22% for TD-JEPA, measuring pair ordering rather than planner failure rate.Each model’s shortlist retains its lowest predicted-cost candidates; the percentages average within-start inversion fractions across eligible mixed-outcome starts.Source: §C.1 · p. 16

C.2 GRANULAR AND GENERALIZATION DETAILS

  • Granular’s main relational instance improves attainment over native selection at the registered 0.18 and 0.12 thresholds, alongside continuous-cost evaluation.Table 1 reports mean costs, and Figure 14 shows the complete paired cost distribution.Source: §C.2 · p. 17, Figure 14 · p. 19
  • D-JEPA improves over native selection and calibrated fusion on every held-out geometry, with its largest gain over fusion on the I shape.The three equally sized shape populations contribute equally to the aggregate; paired outcomes separate recovered successes from regressions.Source: §C.2 · p. 17, Table 13 · p. 17

D.2 REPRESENTATION REALIZATION

  • Equation 4 realizes each final strict rank through native mean-squared distance (π_i/(K + 1))^2, enabling native goal distance to recover the aligned choice.Persistent IDs resolve score ties, and a deterministic unit-RMS direction handles zero terminal displacement; the realization preserves the first four predicted steps of the five-step PushT trajectory.Source: §D.2 · pp. 17–18, Figure 12 · p. 18
  • Equation 3 bounds each temporal L2 update by ρ; the observed maximum is 0.0017227774 on the 128-start calibration representation diagnostic.Time-conditioned coefficients can have either sign, allowing different parts of one candidate trajectory to shift in different directions along the prescribed cross-model displacement.Source: §D.2 · p. 18

E ADDITIONAL PHYSICAL VISUALIZATIONS

  • Matched-start visualizations pair recorded executions and show selected sequences with measured success or failure under geometric and visual change.Additional figures provide task execution context, appearance cases, longer sequences, and synchronized videos.Source: E ADDITIONAL PHYSICAL VISUALIZATIONS · p. 18, Figure 15 · p. 19

F TASK COVERAGE AND ADDITIONAL DIAGNOSTICS … H THEORETICAL PROPERTIES OF DECISION ALIGNMENT

The supplied sections document task coverage and application-specific interfaces for manipulation, driving, and physical robots, and frame theoretical properties linking decision-local prediction gaps to bounded alignment and native-distance realization.

F TASK COVERAGE AND ADDITIONAL DIAGNOSTICS

  • Controlled adaptation exposed a mismatch between global prediction-error gains and action selection, motivating decision-boundary supervision that preserves predictive structure elsewhere.Appearance evaluation covers blur, noise, illumination, and colour, using task-local fitting followed by fresh-start evaluation. In Cube, LeWM and TD-JEPA both succeed from the same start through different selected action sequences.Source: F TASK COVERAGE AND ADDITIONAL… · pp. 19–20, Figure 17 · p. 20

F.1 ROBOTIC MANIPULATION

  • D-JEPA reaches the grasping criterion in two configurations where native execution remains unsuccessful.The comparisons use saved action sequences from an identical restored state, with oblique and overhead views showing the approach, contact geometry, and terminal outcomes.Source: §F.1 · p. 20, Figure 18 · p. 21, Figure 19 · p. 21

F.2 AUTONOMOUS DRIVING

  • Only D-JEPA passes the turning comfort criterion, while it travels farther and scores higher progress in a second scene.Both methods pass collision and TTC criteria in the turning comparison; both pass collision, TTC, and comfort criteria in the progress comparison.Source: Figure 20 · p. 22, §F.2 · p. 22, Figure 21 · p. 23

G APPLICATION-SPECIFIC INPUTS AND EXECUTION INTERFACES

  • Decision alignment is instantiated where predictive or action-producing models expose alternative futures, using task-specific evidence to guide execution.Physical interaction uses goal-relative predictive evidence, manipulation adds gripper, object, and temporal relations, and driving uses trajectory queries and predicted scores.Source: G APPLICATION-SPECIFIC INPUTS AND… · p. 22

G.1 ROBOTIC MANIPULATION: OBSERVATIONS, PROPOSALS AND EXECUTION

  • RoboTwin manipulation aligns scene-conditioned grasp proposals while retaining the native future and applying calibrated gates at execution.The interface includes four RGB views, robot and end-effector state/action vectors, and 16-dimensional bimanual commands. Sixteen offset variants preserve predicted relative grasp geometry; a timing or distance gate selects a corrected future or preserves the native one.Source: §G.1 · pp. 22–23

G.2 AUTONOMOUS DRIVING: TRAJECTORY EVIDENCE AND CALIBRATED READOUTS

  • Drive-JEPA supplies 32 trajectories over four seconds, and calibrated set-based correction and risk predictions inform candidate selection.The score-correction model compares candidates with a set encoder; a separate relative head predicts candidate benefit and collision/TTC risk, with leave-one-source-log-out calibration retaining native fallback.Source: §G.2 · pp. 23–24

G.3 PHYSICAL ROBOT: PREDICTED FUTURES TO ACTION SEQUENCES

  • For physical robots, a bounded relational readout combines goal-relative terminal descriptors with within-set native-cost ranks to select an action.With cross-entropy search, ranks are applied only after forming the shared decision pool; the selected candidate ID passes unchanged to the device controller for execution of its original sequence.Source: §G.3 · p. 24

H THEORETICAL PROPERTIES OF DECISION ALIGNMENT

  • The theoretical section frames properties connecting the decision-local prediction gap to bounded alignment, scale-free predictive evidence, and native-distance realization.It considers a fixed candidate set with K ≥2 and retains the score conventions established earlier in the paper.Source: H THEORETICAL PROPERTIES OF DECISION… · p. 25

H.1 GLOBAL PREDICTION QUALITY AND EXECUTION DECISIONS … H.4 EXACT REALIZATION THROUGH NATIVE LATENT DISTANCE

These sections show why global prediction quality can miss execution errors, then characterize how bounded relational alignment and scale-free ordinal evidence affect candidate selection. They also show that final decision ranks can be realized through native latent distance for planning.

H.1 GLOBAL PREDICTION QUALITY AND EXECUTION DECISIONS

  • Vanishing average prediction error and near-perfect global rank agreement can coexist with fixed execution regret when prediction swaps the best two candidates.The construction has regret ∆; for K = 63, its correlation is approximately 0.999952, and a threshold 0 < θ < ∆ makes the predicted choice fail despite an available successful candidate.Source: §H.1 · p. 25

H.2 DECISION MARGINS AND BOUNDED ALIGNMENT

  • If every candidate cost error is at most η, execution regret is at most 2η, and a realized-cost margin greater than 2η guarantees unique selection of the optimum.Source: §H.2 · p. 25
  • A bounded relational correction preserves base-score gaps greater than 2ϵ, and the selector can promote only candidates within 2ϵ of the base minimum.The relational/base gate returns either the corrected selector or the base winner, so it obeys the same selection bound.Source: §H.2 · p. 25

H.3 SCALE-FREE EVIDENCE ACROSS PREDICTIVE GEOMETRIES

  • Strictly increasing transformations of each source’s costs preserve ordinal evidence, while persistent candidate IDs make tie resolution independent of presentation order.With descriptors and parameters fixed, the dual-model token, ordinal base score, and relational readout remain unchanged; dense descriptors retain model-specific geometry.Source: §H.3 · p. 26

H.4 EXACT REALIZATION THROUGH NATIVE LATENT DISTANCE

  • The final strict candidate ranks can be realized in goal-relative latent representations whose native distances reproduce the full order and select the same action.Source: §H.4 · p. 26
  • The realization uses the original terminal direction when displacement is nonzero and a fixed unit-RMS direction otherwise, leaves earlier future steps unchanged, and is numerically verified in the deployed implementation.Table 6 reports the numerical recovery; the passage supplies no specific values.Source: §H.4 · p. 26

Related papers

Loading 2609.24749v1…