PaperlayerView exact revision on arXiv

Source-linked AI summary

Juno: Taming Predictive Latents for Vision-Language-Action Models

Yuchen Zhu, Chenyi Xu, Yulin Zhang, Gang Xu, Wentao Zhu

arXiv:2610.09940v1cs.ROcs.CV

TL;DR

Predictive latents could help VLA models, but their use across pretraining, policy learning, and deployment faces embodiment mismatch, interference with action learning, and teacher miscalibration under shifts. Juno uses one action-conditioned JEPA across those stages; its evaluations report higher success and shift resilience, with a 68.5% SimplerEnv average versus Qwen3GR00T’s 60.9%.

  • Problem

    Making predictive latents useful across VLA pretraining, policy learning, and deployment must address embodiment mismatch, interference with action learning, and teacher miscalibration under shifts.

    Source: §1 · p. 2

  • Method

    Juno uses one action-conditioned JEPA as a control-aligned representation backbone, predictive teacher, and adaptable dynamics model across pretraining, policy learning, and deployment.

    Source: §1 · p. 2

  • Results

    Across SimplerEnv, RoboCasa-GR1, and real-world manipulation, Juno shows higher success and shift resilience; on SimplerEnv it reaches 68.5% average success versus Qwen3GR00T’s 60.9%.

    Source: §4.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    Juno links future-state prediction with action generation and improves robustness through deployment experience.

    Source: §5 · p. 9

Abstract

from arXiv · show

Joint-embedding predictive architectures (JEPAs) predict masked or future observations in representation space, offering a natural source of predictive latents for vision-language-action (VLA) models. Yet making these latents useful across pretraining, policy learning, and deployment requires addressing three failures: mismatch with embodiment-specific control, interference with action learning, and teacher miscalibration under distribution shifts. We introduce Juno, a unified framework built around one action-conditioned JEPA that serves as a control-aligned representation backbone, a predictive teacher, and an adaptable dynamics model. During pretraining, we train it on embodiment-matched trajectories and use a dynamic CLS loss to transfer motion-weighted patch dynamics to a compact global state. During policy learning, we fuse current-frame JEPA patches into VLA perception and use a decoupled reasoning branch with separate transformation parameters to distill future latent states for action generation. During deployment, we adapt the world model on all observed transitions, including failed rollouts, freeze the adapted teacher, and re-align the policy on verified executions using LoRA adapters and a trainable action head, without expert corrections or task rewards. On SimplerEnv, Juno raises average success from $60.9\%$ to $68.5\%$ over Qwen3GR00T, the strongest baseline, and test-time adaptation further reaches $72.7\%$; on a real robot, it retains $70\%$--$75\%$ success under background, height, and object shifts where the base policy collapses to $0\%$.

1 INTRODUCTION

Juno treats predictive latents as a lifecycle problem: embodiment mismatch, interference with action learning, and teacher miscalibration obstruct their use across pretraining, policy learning, and deployment. It uses one action-conditioned JEPA across these stages, with control-aligned pretraining, decoupled predictive reasoning, and dynamics-first adaptation.

1 INTRODUCTION

  • A single action-conditioned JEPA addresses the three lifecycle obstacles as a control-aligned backbone, predictive teacher, and adaptable dynamics model.Juno organizes these roles around Control-Aligned Pretraining, Decoupled Predictive Reasoning, and Dynamics-First Test-Time Adaptation.Source: §1 · p. 2
  • Dynamic CLS loss transfers motion-weighted patch dynamics to the compact global state, addressing its weak response to local interaction changes.The figure reports that LeWM (CLS) stays nearly constant during pick-and-place, while LeWM+Ldyn becomes sensitive.Source: §1 · p. 2, Figure 2 · p. 3
  • A decoupled reasoning branch distills future teacher latents while receiving action gradients, training aligned states to support action generation.Current-frame JEPA patches are fused into VLA visual tokens, and the branch uses separate transformation parameters under shared causal attention.Source: §1 · p. 2
  • Juno-TTT adapts the world model on all observed transitions, freezes the teacher, then re-aligns the policy on verified executions without expert corrections or task rewards.Failed actions are excluded as policy targets even though their consequences provide dynamics supervision.Source: §1 · p. 2

2 RELATED WORK

Prior work uses predictive representations and world-model signals for embodied prediction and VLA policy learning, but leaves open which predictive state fits a robot controller and whether future alignment yields action-decodable states. Existing robotics test-time updates also assume a reliable frozen teacher.

Latent Predictive World Models

  • Action-conditioned JEPAs extend representation prediction to embodied planning, but prior work does not determine which predictive state matches a particular robot controller.JEPAs predict future representations without pixel reconstruction, and related work also explores latent-action abstractions from unlabeled video.Source: §2 · p. 3

World Action Models

  • VLA world-model methods align future representations or expose predicted dynamics through tokens and subgoals, but future alignment alone does not establish action decodability.The cited approaches span representation alignment, foresight tokens, latent subgoals, and aligned visual–action tokens.Source: §2 · p. 3

Test-Time Training in Robotics

  • Robotics test-time training adapts policies or memory using self-supervised signals, yet these methods assume a reliable frozen teacher that may be miscalibrated under severe shifts.The reviewed updates include prompts, adapters, policy components, and adaptive memory.Source: §2 · p. 3

3 METHOD

Juno uses an action-conditioned JEPA trained on embodiment-matched trajectories, then integrates its representations into VLA perception and action generation. At deployment, it adapts the world model on observed transitions and re-aligns the policy using verified executions.

3.1 PRELIMINARIES

  • A JEPA encodes observations and predicts future latent states; a VLA policy maps visual observations and language instructions to actions.The action-conditioned world model uses a CLS token for latent prediction and retains patch tokens as spatial evidence.Source: §3.1 · pp. 3–4

3.2 CONTROL-ALIGNED PRETRAINING

  • LeWM is trained on embodiment-matched trajectories, aligning its action-conditioned representation with the downstream control distribution.Its patch features respond more locally to robot–object interactions than V-JEPA 2, while its action-conditioned objective supports later adaptation.Source: §3.2 · p. 4

3.2.1 DYNAMIC CLS LOSS

  • Dynamic CLS training uses a frozen patch supervisor and motion-weighted patch changes to make the compact CLS state more responsive to localized changes.The auxiliary loss preserves LeWM’s single-CLS interface while transferring patch-level change information to the CLS representation.Source: §3.2.1 · pp. 4–5

3.3.1 PATCH-LEVEL FUSION

  • Frozen JEPA patch features are projected and fused into VLA visual tokens through cross-attention and a gated residual, preserving spatial information without changing the token sequence.Source: §3.3.1 · p. 5

3.3.2 FUTURE LATENT-STATE DISTILLATION AND ACTION GENERATION

  • A decoupled reasoning branch with separate non-embedding parameters predicts future JEPA states, while its hidden states also condition the action decoder.The branch shares the causal attention graph with the vision-language branch, whose action-generation representation is preserved.Source: §3.3.2 · p. 5
  • During training, future observations provide detached JEPA CLS targets matched one-to-one to reasoning queries; inference removes the future-target path.The joint action-generation pathway makes the aligned states decodable into actions rather than merely predictive of future observations.Source: §3.3.2 · p. 5

3.4 DYNAMICS-FIRST TEST-TIME ADAPTATION

  • Under severe shifts, Juno adapts the world model on all observed transitions, including failures, then freezes it and re-aligns the policy on verified executions.The policy update uses LoRA adapters and a trainable action head; failed actions are not policy supervision targets.Source: §3.4 · p. 6

4 EXPERIMENTS

Juno is evaluated across simulation benchmarks, real-robot transfer, and deployment adaptation. It outperforms the base VLA on both simulation benchmarks, generalizes across real-robot shifts, and improves further with test-time training.

4.1 EXPERIMENTAL SETUP

  • Evaluation covered WidowX SimplerEnv, 24 RoboCasa-GR1 tasks, and real-robot manipulation; the real-robot conditions tested held-out positions, with shifted conditions varying appearance or physical factors.The real-robot frozen-policy comparison used 20 trials per condition; adaptation results were reported separately because they used additional deployment interactions and optimization.Source: §4 · p. 6, §4.1 · pp. 6–7

4.2 SIMULATION BENCHMARK RESULTS

  • 68.5% average success on SimplerEnv beat Qwen3GR00T’s 60.9%, while RoboCasa-GR1 reached 59.6% versus Qwen3GR00T’s 47.8%.Across RoboCasa-GR1’s 24 tasks, Juno outperformed Qwen3GR00T on 21 and scored highest on 16.Source: Table 1 · p. 7, §4.2 · p. 7, Table 2 · p. 8

4.4 TEST-TIME ADAPTATION RESULTS

  • 72.7% average SimplerEnv success after test-time training, up from 68.5%, while robot adaptation raised success from 40% to 65% under noise and from 55% to 70% under dynamic lighting.The real-robot adaptation used 40 rollouts per condition; the SimplerEnv evaluation was nominal, without distribution shift.Source: Table 1 · p. 7, Table 4 · p. 8, §4.4 · p. 9

4.3 REAL-WORLD ROBOT EVALUATION

  • In frozen-policy real-world evaluation, Juno achieved 100% success in-domain and 75% with an out-of-distribution background, versus the unnamed baseline’s 40% and 0%, respectively.Juno also scored 70% under each of the added-height and combined height-plus-object shifts, while the baseline scored 0% in both.Source: Table 3 · p. 8, §4.3 · p. 9

4.5 ABLATION STUDY

  • Removing fusion alone reduced SimplerEnv average success to 53.9%, while removing fusion and the MoT branch together recovered it to 60.4%.Without alignment loss, success was 60.8% versus the 60.9% base VLA; dynamic CLS contributed a smaller 2.9-point gain.Source: Table 5 · p. 9, §4.5 · p. 9

5 CONCLUSION

Juno unifies predictive world models across VLA pretraining, policy learning, and test-time adaptation. Experiments report higher success and resilience to distribution shifts.

5 CONCLUSION

  • Juno integrates predictive world models into VLA pretraining, policy learning, and test-time adaptation.It combines control-aligned representations, decoupled predictive reasoning, and dynamics-first adaptation.Source: §5 · p. 9
  • Experiments in SimplerEnv, RoboCasa-GR1, and real-world manipulation show higher success and resilience to distribution shifts.Source: §5 · p. 9
  • Future work will scale training and evaluation across larger datasets, broader tasks, and multiple embodiments.Source: §5 · p. 9

SUPPLEMENTARY MATERIAL CONTENTS

Figure 5 presents final-layer attention and patch-embedding PCA for V-JEPA 2, LeWM, and the control-aligned encoder. Its color comparisons are valid only within each encoder.

SUPPLEMENTARY MATERIAL CONTENTS

  • LeWM focuses attention on the manipulation region, while the control-aligned encoder concentrates attention more along robot and object boundaries.Source: Figure 5 · p. 16
  • Figure 5 compares final-layer attention maps and PCA of patch embeddings across V-JEPA 2, LeWM, and the control-aligned encoder.Source: Figure 5 · p. 16
  • PCA colors are comparable only within each encoder because PCA is fitted separately on the same frames.Source: Figure 5 · p. 16

A ADDITIONAL REPRESENTATION ANALYSIS

The additional representation analysis examines spatial feature structure and whether control-aligned global-token transitions reflect motion patterns. Its visualizations assess teacher suitability in the studied setting, not isolated effects of individual training factors.

A ADDITIONAL REPRESENTATION ANALYSIS

  • The analysis compares frozen V-JEPA 2 and the control-aligned encoder, but does not isolate pretraining data, architecture, and objective as independent factors.The control-aligned encoder is trained on robot trajectories with an action-conditioned JEPA objective and Dynamic CLS loss.Source: A ADDITIONAL REPRESENTATION ANALYSIS · p. 16

A.1 ATTENTION AND PCA VISUALIZATION

  • Compared with LeWM, the control-aligned encoder shows more concentrated attention along robot and manipulated-object edges while retaining foreground–background separation in PCA maps.The maps show where information is represented, not which pixels the downstream policy uses.Source: §A.1 · p. 16

A.2 CLS TRANSITION SIMILARITY ACROSS ACTION PATTERNS

  • Transitions with the same action pattern have higher similarity, while the illustrated translation-to-gripper-closing comparison has low similarity.The visualization concerns transition direction, not magnitude, and does not by itself isolate the Dynamic CLS loss effect.Source: §A.2 · p. 17, Figure 6 · p. 17

B.1 TRAINING DATA

The supplementary material specifies training data and optimization recipes, control-aligned encoder pretraining, and evaluation protocols for SimplerEnv and RoboCasa-GR1.

B.1 TRAINING DATA

  • SimplerEnv examples use one 224 × 224 RGB view, instructions, 16-step action chunks, and eight future images at stride 2; real-robot fine-tuning uses 60 demonstrations.Bridge/Fractal sampling gives each source probability 1/2; RoboCasa trains across 24 tasks without proprioception.Source: §B.1 · pp. 17–18

B.2 POLICY OPTIMIZATION

  • Policy optimization uses Qwen3-VL-4B-Instruct with a GR00T flow-matching decoder, and alignment loss is scheduled alongside action loss.The PiPER recipe retains alignment loss and updates LoRA adapters plus the action decoder and remaining fusion, projection, and query parameters.Source: §B.2 · p. 18, Table 7 · p. 18

B.3 CONTROL-ALIGNED ENCODER PRETRAINING

  • The control-aligned encoder is trained on four-frame Bridge/Fractal windows with action-conditioned stride-two transitions and Dynamic CLS loss.The policy uses the epoch-36 export of the encoder run.Source: §B.3 · p. 19

C.1 SIMPLERENV

  • SimplerEnv evaluation uses four WidowX Bridge tasks, 96 episodes per task across four runs, and reports the unweighted mean success rate.Source: §C.1 · p. 19

C.2 ROBOCASA-GR1

  • RoboCasa-GR1 evaluation covers all 24 Tabletop Tasks with 50 episodes per task and reports per-task rates and their unweighted mean.Success is recorded if the simulator success flag becomes true at any step.Source: §C.2 · p. 19

C.3 REAL-WORLD EVALUATION

The evaluation uses a PiPER robot setup and reports success under held-out and progressively shifted conditions. RoboCasa-GR1 results cover 24 tabletop tasks, with average success defined as their unweighted mean.

C.3 REAL-WORLD EVALUATION

  • The PiPER setup uses only top-view RGB input resized to 224×224, and executes 16-step action chunks at 15 Hz.Actions are delta end-effector targets in the robot base frame; workspace or joint-limit violations are rejected.Source: §C.3 · p. 19
  • RoboCasa-GR1 evaluation covers 24 tabletop tasks, with success-rate entries and an unweighted mean across tasks.The caption identifies the best result in each row as bold but supplies no cell values.Source: Table 8 · p. 20
  • Evaluation conditions use held-out cube positions; background, height, and object conditions add cumulative appearance, workspace, and object changes.The supplied description specifies a 10 cm work-plane increase for the height condition and a grape substitution for the object condition.Source: §C.3 · p. 20
  • Success means placing the designated object in the designated bowl without operator intervention.Source: §C.3 · p. 21

D TEST-TIME TRAINING

Juno-TTT separates test-time dynamics adaptation from policy re-alignment: all deployment transitions inform the world model, while only verified successful executions supervise policy updates. The section also describes simulation and robot adaptation procedures and paired qualitative rollouts.

D.1 PROCEDURE

  • Stage A adapts the world model on every valid deployment transition, including failures; Stage B freezes that teacher and updates the policy using successful data only.The policy uses LoRA adapters while its pretrained weights remain frozen, and the action decoder is fully trainable.Source: §D.1 · p. 21

D.2 SIMPLERENV

  • SimplerEnv adaptation uses frozen-policy rollouts, with a 10-epoch world-model stage followed by 400 policy-update steps.Stage A uses an episode-level 90/10 split; Stage B uses a 16-step action chunk and eight future frames at stride 2.Source: §D.2 · p. 21

D.3 REAL ROBOT

  • The dynamic-lighting robot procedure collects 40 rollouts, using all valid transitions for Stage A and verified successful executions for Stage B.The lighting updates are separate from the frozen-policy evaluation in Table 3.Source: §D.3 · p. 21

E QUALITATIVE RESULTS

  • The qualitative section pairs failed and successful executions, comparing Juno with the base VLA in some figures and Juno before versus after TTT in others.Frames are ordered from left to right; the lighting example compares models, while the Gaussian-noise example compares before and after TTT.Source: E QUALITATIVE RESULTS · p. 21

E.1 SIMULATION EVALUATION

The simulation examples show a base-VLA failure alongside Juno completing spoon placement, and a separate failed-versus-successful Juno pair for eggplant placement before and after TTT.

E.1 SIMULATION EVALUATION

  • Juno places the spoon on the towel in the illustrated rollout, while Qwen3GR00T fails to place it there.The sequence shows Juno picking up and transporting the spoon before releasing it on the target surface.Source: §E.1 · p. 21, Figure 8 · p. 22
  • The eggplant sequence before TTT lifts the eggplant but does not complete its transfer into the basket.The supplied prose for this comparison ends before describing the post-TTT sequence; the figure caption identifies the successful post-TTT placement.Source: §E.1 · p. 21, Figure 9 · p. 22
  • For eggplant placement, the figure contrasts a failed Juno execution before TTT with a successful placement into the yellow basket after TTT.Source: Figure 9 · p. 22

E.2 REAL-WORLD EVALUATION

The real-robot examples contrast Juno before and after TTT under Gaussian observation noise, and compare the frozen base policy with frozen Juno under flickering colored lighting. The selected rollouts illustrate behavior but do not establish rates or isolate failure causes.

E.2 REAL-WORLD EVALUATION

  • Under Gaussian observation noise, the illustrated Juno rollout fails before TTT and completes cube placement inside the bowl after adaptation.The noise-corrupted observations appear in the figure insets.Source: §E.2 · p. 22, Figure 10 · p. 23
  • Under flickering colored illumination, the frozen base policy leaves the cube outside the bowl while frozen Juno completes placement.The separate dynamic-lighting Juno-TTT gain is reported as 55% to 70%.Source: §E.2 · p. 22, Figure 11 · p. 23
  • Selected rollout pairs do not establish success rates or isolate the causes of individual failures.Source: §E.2 · p. 22
Loading 2610.09940v1…