PaperlayerView exact revision on arXiv

Source-linked AI summary

SaiVLA-JEPA: Evaluating Future Latent Prediction in JEPA-Conditioned VLA Control

Rufeng Xiao, Ruihao Xiao, Wei Chen, Xinhai Sun

arXiv:2610.38508v1cs.ROcs.AI

TL;DR

The paper asks whether future-latent prediction improves closed-loop control when a VLA policy already uses V-JEPA features. It adds future-prediction heads to a JEPA-conditioned flow policy and evaluates paired controls. Current JEPA conditioning improves success over BaseFlow, but future supervision shows no detectable aggregate gain at the evaluated training budget.

  • Problem

    The action loss alone does not explicitly require a V-JEPA-conditioned policy trunk to model how visual representations change over time.

    Source: §1 · p. 1

  • Method

    SaiVLA-JEPA adds frozen V-JEPA features to a VLM-conditioned flow policy and trains auxiliary future-latent heads; the paper separately evaluates context and selected-parameter adaptation.

    Source: §1 · pp. 1, 3

  • Results

    82.45% versus 71.63%: CurrentJEPA versus BaseFlow; FutureJEPA reaches 81.99% versus CurrentJEPA’s 82.45%, with no detectable aggregate future-objective gain.

    Source: §7 · p. 15

  • Takeaways & Limitations

    The results support assessing future-latent objectives through paired closed-loop rollouts rather than latent loss alone.

    Source: §1 · p. 3

    The future-head comparison is from a single training run, so replication with independent training runs remains necessary.

    Source: §6 · p. 14

Abstract

from arXiv · show

Joint-embedding predictive architectures (JEPAs) learn to predict representations rather than pixels, offering a compact way to model how visual scenes evolve. We ask whether this predictive objective improves closed-loop control when a vision-language-action (VLA) policy already uses V-JEPA features. We call our JEPA-conditioned flow policy family SaiVLA-JEPA. It combines a vision-language model with a diffusion transformer action expert and receives features from a frozen V-JEPA encoder. Two lightweight auxiliary heads predict future representations during training and are omitted at inference, leaving the action sampler unchanged. We compare policies without JEPA features, with current JEPA conditioning, and with additional future supervision on 10,030 paired LIBERO-Plus episodes. Current JEPA conditioning reaches about 82.5% success versus 71.6% without JEPA, while future supervision provides no detectable aggregate gain at the evaluated training budget. We separately evaluate SaiVLA-JEPA-TCA, which adapts latent context, and SaiVLA-JEPA-TPA, which adapts selected parameters. The context-retry gain is a system-level result under a larger attempt budget; parameter adaptation shows no consistent advantage over the frozen policy and target controls. These results motivate assessing latent-prediction objectives through paired closed-loop rollouts rather than latent loss alone.

1 Introduction

The paper tests whether future-latent supervision improves a V-JEPA-conditioned flow policy’s closed-loop control. It finds a gain from current JEPA conditioning, but no detectable aggregate gain from future supervision in the paired comparison.

1 Introduction

  • The action loss alone does not explicitly require the policy trunk to model how visual representations change over time; the paper evaluates that additional supervision with episode-paired controls.The paired design contrasts current-feature and no-JEPA controls across the complete LIBERO-Plus benchmark.Source: §1 · pp. 1, 3
  • SaiVLA-JEPA adds frozen V-JEPA features to a VLM-conditioned flow policy, and FutureJEPA adds training-only residual heads for future representations without changing the inference sampler.The family also includes TCA, which infers persistent and transient contexts with weights frozen, and TPA, which updates selected weights.Source: §1 · pp. 1, 3
  • Under final mixed-cap evaluation, FutureJEPA reached 81.99% overall and 73.84% on LIBERO-10, versus BaseFlow’s 71.63% and 53.12%; this joint contrast does not isolate future supervision.The completed paired comparison found no detectable aggregate success-rate benefit from future supervision at this training budget.Source: §1 · p. 1
  • +0.03 percentage points: FutureJEPA versus CurrentJEPA under the original all-300-step protocol, with p = 0.958 and a 95% interval of approximately [−0.73, +0.77].The study interprets this as no detectable aggregate future-head gain in a single training run, not proof that the true effect is zero.Source: §1 · p. 2
  • The paper positions future-latent supervision as distinct from a planner: its predictor is evaluated as supervision for a frozen-encoder policy, not as a planning model.JEPA-style methods predict representations rather than pixels [5], while V-JEPA motivates use of a reusable visual state abstraction [1].Source: §1 · p. 2

2 Method

The method preserves the native flow action path while adding JEPA tokens and training-only future-latent prediction heads. The auxiliary branch is absent at inference, so the action sampler remains the same across the policy arms.

2.1 Shared action path

  • Projected JEPA tokens join the native prefix and noisy action tokens in the DiT self-attention sequence, while VLM features enter through per-block cross-attention.The flow policy retains its native velocity loss; robot state is encoded separately.Source: §2.1 · p. 3

2.2 Future heads

  • FutureJEPA predicts representations at h = 4 and h = 16 with independent residual heads and adds 0.1Lfuture to the flow loss.Tokenwise mean-squared error uses detached targets from the frozen JEPA encoder, updating the trainable projection, shared trunk, and prediction heads.Source: §2.2 · p. 4

2.3 Control definition

  • CurrentJEPA retains JEPA conditioning but sets Lfuture = 0, whereas BaseFlow removes JEPA conditioning; only CurrentJEPA versus FutureJEPA isolates the future objective.Both JEPA-conditioned policies retain the native learned prefix, and future queries are masked from noisy action tokens.Source: §2.3 · p. 4, §2.2 · p. 4

2.4 Inference

  • At inference, the auxiliary heads are absent and all three arms use the same four-step Euler sampler and replan interval.The 16-action chunk is executed open-loop for one replan interval; the future heads add no action-sampling forward passes, though wall-clock latency was not independently benchmarked.Source: §2.4 · p. 4

3 Test-Time Adaptation

SaiVLA-JEPA-TCA adapts persistent and transient latent contexts around a frozen policy, while SaiVLA-JEPA-TPA updates selected network parameters. Both are test-time adaptation routes evaluated around the same base policy.

3 Test-Time Adaptation

  • TCA and TPA are alternative test-time update scopes: TCA adapts latent context with weights frozen, while TPA updates a selected parameter subset.The shared policy family uses a VLM-conditioned DiT action expert and frozen V-JEPA features.Source: §3 · p. 5, §3.1 · p. 5

3.1 Context-conditioned policy

  • Persistent context m is intended to retain environment information across local transitions and retries, while transient context u captures short-lived prediction discrepancies and resets between retries.The two 16-dimensional contexts are concatenated and supplied to the policy; neither is assumed to represent an interpretable physical parameter.Source: §3.1 · p. 5, §3.2 · p. 6

3.2 Online context updates and retries

  • On the first attempt, actions use zero context throughout; inferred terminal context can initialize a retry but cannot affect that first rollout.Later within-attempt context updates follow executed four-action windows and affect the next 16-action chunk, with network weights frozen.Source: §3.2 · p. 6

3.3 Parameter adaptation

  • TPA proposes a selected-parameter update from recent transitions, but acceptance checks compare target quality and action changes, with rejected updates rolled back.Parameters are restored at each episode start; online loss reduction or an accepted update alone does not establish a control benefit.Source: §3.3 · p. 7

4 Data and Training

Training and evaluation use paired LIBERO-Plus episodes, with future JEPA targets included when available and masked when absent. The study trains a future-prediction policy alongside current-feature and no-JEPA controls, and separately trains an observer and context adapter.

4.1 Data and target construction

  • Evaluation covers 10,030 LIBERO-Plus episodes across four suites, with matched initial states and action-noise streams for each comparison.Final success-rate tables use 300-step caps on Spatial/Object and 600-step caps on Goal/LIBERO-10; primary paired tests use 300 steps across all suites.Source: §4.1 · p. 8

4.2 Policy training

  • FutureJEPA adds residual future-latent MSE at offsets 4 and 16 to the flow loss, while CurrentJEPA and BaseFlow use flow loss only.All three policies use 25,000 optimizer steps; the VLM and visual encoder remain frozen, while the flow action expert and JEPA projection are trained.Source: §4.2 · p. 8

4.3 Training the transition observer

  • The transition observer infers persistent context from support windows across same-task episodes and transient context from a separate query-episode window before predicting a held-out four-action query.Observer training uses independent randomized training and validation partitions; these are not additional benchmark evaluation runs.Source: §4.3 · p. 8

4.4 Training the context adapter

  • The context adapter is trained with a frozen FutureJEPA policy, then used with the 25,000-step CurrentJEPA policy in the complete retry evaluation.The adapter is not retrained on evaluation episodes, and this transfer between policy checkpoints is retained as a limitation.Source: §4.4 · pp. 8–9

4.5 Evaluation statistics

  • Primary paired comparisons report success counts, candidate-minus-control percentage points, bootstrap intervals, and exact McNemar tests; mixed-cap rates are descriptive.A lower latent-prediction loss alone is not treated as evidence of better closed-loop control.Source: §4.5 · p. 9

5 Experiments and Evidence

Current JEPA conditioning substantially outperforms BaseFlow in the final mixed-cap rates, while adding future supervision yields no detectable aggregate success gain over CurrentJEPA. Among episodes completed by both JEPA-conditioned policies, CurrentJEPA finishes sooner.

5.1 Main paired comparison

  • +0.03 points separates FutureJEPA from CurrentJEPA (p = 0.958; 95% interval [−0.73, +0.77]), indicating no detectable aggregate future-supervision benefit in this run.The all-300-step FutureJEPA-versus-BaseFlow difference is +12.92 percentage points, but combines JEPA conditioning with future supervision.Source: §5.1 · p. 9
  • 82.45% for CurrentJEPA and 81.99% for FutureJEPA compare with 71.63% for BaseFlow in the final mixed-cap evaluation.The paired statistics use the original all-300-step protocol; the mixed-cap rates are descriptive.Source: §5.1 · p. 9

5.2 Completion-time efficiency

  • CurrentJEPA finishes 4.34 steps earlier on average than FutureJEPA across the 7,191 episodes both complete, and is faster on 65.7% of paired episodes.The paired advantage is consistent across all four suites; the analysis includes successful episodes only.Source: §5.2 · pp. 9–10
  • Against BaseFlow, CurrentJEPA has a 7.01-step paired mean advantage; the BaseFlow-versus-FutureJEPA mean favors FutureJEPA by 3.19 steps, but its signed-rank test is not significant.For that latter contrast, 55.5% of paired episodes favor BaseFlow, while a minority of near-cap stalls carries the mean.Source: §5.2 · p. 10

5.3 Zero-shot JEPA conditioning and projection transfer

The zero-shot projection study finds JEPA-token gains that depend on the projection and suite, while training-side diagnostics show near-parity in action loss and limited evidence that latent prediction improves control.

5.3 Zero-shot JEPA conditioning and projection transfer

  • +1.85 points over BaseFlow came from random-projection JEPA injection, and trained-projection transfer added +2.75 points in this zero-shot construction.The gains were suite-dependent; the protocol measures projection transfer into an untrained input path, not future supervision.Source: §5.3 · p. 10

5.4 Training-side diagnostics

  • At h=16, prediction MSE is 0.10295 versus 0.11177 for repetition; at h=4, it is 0.04152 versus 0.04134.This separate adapter-only diagnostic does not establish that temporal conditioning improves action prediction.Source: §5.4 · p. 11
  • Final Flow losses were 0.03017 for BaseFlow, 0.03185 for CurrentJEPA, and 0.02873 for FutureJEPA, with trailing means within 1.3%.The passage notes that open-loop loss parity is compatible with rollout gaps because closed-loop execution is sequential and policy-generated states compound differences.Source: §5.4 · p. 11
  • Across 323 shared trainable tensors, checkpoint cosine similarity was 0.9686–0.9733, while the JEPA projection similarity fell to 0.805–0.825.The future-latent heads in BaseFlow and CurrentJEPA remained near random initialization; the passage flags the similarity record as provenance-limited.Source: §5.4 · p. 11

5.5 Effect of the future-loss weight

Increasing future-loss weight improved latent prediction but produced at most about one point of aggregate rollout change. Retry-time context reuse increased cumulative successes under a larger attempt budget, with higher evaluation cost and limits on attributing the gains to inferred memory.

5.5 Effect of the future-loss weight

  • λ increases of 3× and 10× improved h=16 validation MSE by 18% and 33%, but changed aggregate rollout success by only +0.71 and +0.81 points.The raised-weight arms did not differ detectably from each other or from CurrentJEPA overall; the observed increases were concentrated on LIBERO-Object.Source: §5.5 · p. 11, Table 4 · p. 12

5.6 Retry-time context adaptation

  • In the equal-budget comparison, fixed decay and evidence gating did not differ detectably (exact McNemar p = 0.4863); most added recoveries occurred on attempt two.There was no five-attempt no-context curve for comparison.Source: §5.6 · p. 12
  • The context methods used about 2.17 and 2.21 times the baseline environment steps, excluding context-optimization and offline-training costs.A complete equal-budget no-context arm and shuffled-context arm are needed before attributing gains specifically to inferred memory.Source: §5.6 · pp. 12–13, Table 6 · p. 13
  • Although first attempts matched initial state, observation, noise, and action, full trajectories matched in only 8,709 and 8,704 cases.The passage attributes first-attempt outcome differences to evaluation variability rather than context effects.Source: §5.6 · p. 12
  • With up to five attempts, fixed-decay context reuse reached 8,914/10,030 successes versus 8,124/10,030 for the one-attempt baseline.Evidence-gated reuse reached 8,901; these are bounded-retry results with unequal attempt budgets. The retry baseline and CurrentJEPA results come from separate runs.Source: §5.6 · pp. 12–13, Figure 3 · p. 13

5.7 Additional retry comparisons

Additional retry comparisons report selected-case results and a partial varying-noise comparison, not a pooled complete-benchmark evaluation. The context comparisons do not isolate memory from retry diversity or equalize every attempt budget.

Persistent context versus reset

  • Carrying persistent context while resetting transient context succeeded on 14/16 records, but zero-context rollouts already succeeded on 14/16 and fresh action noise yielded 16/16 for both carry and reset.The selected cases support an effect of retained context but do not isolate it from retry diversity.Source: §5.7 · p. 13

Selected-case studies

  • A fixed schedule improved 54/80 baseline successes to 69/80, while a 500-case Spatial study recorded 396, 423, and 420 successes for baseline, fixed decay, and evidence gating.The separate 26-case persistent-context pilot recorded 12 memory-only and one no-context-only success over 338 attempts and 100,206 steps; these selected-case studies were not pooled with the complete four-suite evaluation.Source: §5.7 · p. 13

Partial varying-noise comparison

  • 4,329/4,315/4,297 successes were recorded for baseline, fixed decay, and evidence gating across 4,790 cases; after excluding four incomplete budgets, counts were 4,327/4,314/4,297.These partial results do not show an advantage for context reuse and cannot substitute for a complete equal-budget evaluation.Source: §5.7 · pp. 13–14

5.8 Parameter-level adaptation

Parameter-level adaptation tests whether transition-prediction updates improve closed-loop control. Across tested update locations and controls, no method demonstrates a reliable aggregate gain.

5.8 Parameter-level adaptation

  • Table 7 compares online updates with a no-update policy and target controls, but task subsets and sample sizes differ across rows, so results are not pooled.Initial states and action-noise streams are paired within each row.Source: §5.8 · p. 14

5.8.1 Update location and outcome

  • No tested parameter-update method demonstrates a reliable aggregate closed-loop gain over no-update or target controls.Visual or projection updates sometimes improve prediction losses without consistently improving success; matched-target adaptation also does not separate from controls in larger replay and interval studies.Source: §5.8.1 · p. 14
  • The 40-episode DiT-block update reaches 27.5% success versus 32.5% without updates, while joint visual–DiT adaptation matches no-update on 58/80 episodes.Shadow evaluation executes frozen-policy actions, so equal success rates do not test the utility or safety of candidate actions.Source: §5.8.1 · p. 14

6 Limitations, Safety, and Reproducibility

The evidence is limited to paired comparisons at the evaluated training budget and simulation-based LIBERO-Plus results. Context retry, deployment safety, and reproducibility each have additional scope constraints.

Experimental scope

  • The paired CurrentJEPA–FutureJEPA comparison finds no detectable aggregate future-head gain after 25,000 steps, but this does not establish an exactly zero effect.Independent training-run replication remains necessary; the BaseFlow–FutureJEPA contrast changes both JEPA conditioning and future supervision, and exact equality of all initial parameters and stochastic training draws was not established.Source: §6 · p. 14

Representation and generalization

  • Evidence is simulation-based and concentrated on LIBERO-Plus; transfer to new embodiments, real cameras, or long-horizon household tasks is not established.Future latent MSE can improve without improving action selection, making closed-loop evaluation important alongside prediction metrics.Source: §6 · p. 14

Context adaptation

  • The complete context-retry study gives the no-context baseline one attempt but allows context reuse up to five, combining memory, repeated execution, and extra compute.A complete five-attempt no-context comparison and shuffled-context control are still needed; the adapter was trained with an earlier policy, and its final action residual has no hard norm bound.Source: §6 · p. 15

Safety

  • Real-robot use requires workspace and velocity limits, collision monitoring, an emergency stop, human supervision, and camera-calibration and timing validation.Simulation success and accepted latent-gradient updates are not evidence of deployment safety.Source: §6 · p. 15

Reproducibility

  • Independent reproduction additionally requires benchmark data, visual and language-model weights, trained policies, and the simulator environment, which are not bundled with the paper.Available checkpoint and source records do not establish complete historical run-time identity.Source: §6 · p. 15

7 Conclusion and Future Work

The paper adds training-only future-latent prediction heads to a VLM-conditioned DiT flow policy without changing its inference-time action route. Current JEPA conditioning provides the main success-rate gain, while future supervision shows no detectable aggregate gain at the evaluated budget.

Conclusion

  • 82.45% versus 71.63% overall success: CurrentJEPA exceeds BaseFlow under the final mixed-cap protocol, while FutureJEPA reaches 81.99% versus 82.45% for CurrentJEPA with no detectable aggregate rollout gain.The method adds residual prediction heads at offsets 4 and 16 while leaving the inference-time action route unchanged.Source: §7 · p. 15
  • Bounded retry with inferred context improves the retry system’s success rate, but uses a larger attempt budget and is an operational recovery result rather than a direct estimate of memory’s causal effect.Parameter-level online adaptation has not shown a reliable aggregate control gain.Source: §7 · p. 15

Future work

  • The next steps include independent training-run replication, matched five-attempt no-context and shuffled-context controls, and tests of transfer and real-robot safety.Other proposed directions include action-conditioned latent dynamics, uncertainty-aware replanning, task-disjoint and cross-embodiment transfer, and multimodal grounding.Source: §7 · p. 15

A Evaluation Protocol and Additional Retry Studies

The main evaluation covers all 10,030 LIBERO-Plus episodes under paired conditions, while additional retry studies use distinct scopes and budgets. The authors also note limits on reconstructing the exact original runs from historical records.

A Evaluation Protocol and Additional Retry Studies

  • The main comparison covers all 10,030 LIBERO-Plus episodes, with policies sharing initial states, action sampler, and paired action-noise streams.Final success-rate tables use 300-step caps on Spatial/Object and 600-step caps on Goal/LIBERO-10; future-prediction heads are absent at inference.Source: A Evaluation Protocol and Additional… · p. 18, Table 8 · p. 18
  • Strict primary paired statistics instead apply the original 300-step cap to every suite.Source: A Evaluation Protocol and Additional… · p. 18
  • Additional retry studies use selected cases, different checkpoints, or different attempt budgets, and their counts are not pooled with one another or the main benchmark.Differences in policy, case selection, and retry budget preclude pooling these results.Source: A Evaluation Protocol and Additional… · p. 18, Table 9 · p. 18
  • Historical evaluation records do not preserve every run-time checkpoint identity, so exact reproduction of the original runs cannot be established from those records alone.The authors retain episode-level records and configuration files, and some original observations were repaired and re-evaluated.Source: A Evaluation Protocol and Additional… · p. 18

Related papers

Loading 2610.38508v1…