PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboJEPA: Scaling Robotic Latent World Models

Artem Zholus, Nicolas Beltran-Velez, Jianhao Yuan, Sarath Chandar, Tushar Nagarajan, Daniel Severo, Koustuv Sinha, Michal Drozdzal, Adriana Romero Soriano, Jeannette Bohg, Nicolas Ballas, Mahmoud Assran

arXiv:2610.10515v1cs.AIcs.RO

TL;DR

RoboJEPA studies how robotic latent world-model capabilities scale with model size, data, and compute, a question not characterized for prior latent world models. It trains an action-conditioned predictor across robotic embodiments and fits compute scaling laws to its prediction error. The error follows a second-order compute power law and is associated with downstream planning performance, while the models also plan toward image goals on real robots.

  • Problem

    How latent world-model quality changes with data, parameters, and compute has not been characterized, despite the scarcity and expense of high-quality robotic interaction data.

    Source: §1 · p. 2

  • Method

    RoboJEPA is an action-conditioned latent world-model family trained on multi-embodiment robotic data to predict future representations, with model scale varied over a fixed encoder.

    Source: §1 · p. 2, §5 · p. 15

  • Results

    Latent forward-prediction error follows a second-order compute power law on real-robot and simulated holdouts, and is a strong predictor of downstream planning performance.

    Source: §1 · p. 3, §5 · p. 15

  • Takeaways & Limitations

    RoboJEPA models plan toward single-image goals on real robots, with real-robot success rates showing an improvement trend as model size increases.

    Source: §1 · p. 3

    The scaling laws characterize the predictor on a frozen V-JEPA 2.1 representation, not jointly scaled encoder and predictor models.

    Source: §5 · p. 15

Abstract

from arXiv · show

Latent world models have shown a remarkable ability to predict future states and to plan in the real world. In practice, however, we lack a principled way to estimate how their capabilities scale with model size, data, and compute, an open problem that slows progress in the field. In this work we present RoboJEPA, a world model based on the Joint Embedding Predictive Architecture (JEPA) and trained on a large-scale dataset spanning 12 robotic embodiments. We show that RoboJEPA's imagination error, the error of its latent rollouts, follows a second-order power law in compute, allowing us to predict model quality well beyond the scale at which the law is fit. We further show that downstream robotic planning performance improves predictably with compute, and that imagination error is strongly correlated with it, making it a reliable proxy for real-robot evaluation. Finally, we demonstrate that latent world models can be deployed zero-shot as robotic agents, planning toward a single goal image to solve tasks requiring long-horizon planning on real hardware. We release all model checkpoints together with our training and robot deployment code. To our knowledge, this is the first work to establish scaling laws for multi-embodiment robotic world models trained on real robot data, and RoboJEPA, at 8B parameters, is the largest JEPA predictor model trained to date.

1 Introduction

RoboJEPA addresses the lack of scaling laws for robotic latent world models, where scarce, costly interaction data makes predicting capability at a given compute budget useful. It scales predictably with compute, with offline forward-prediction loss tracking downstream planning performance, and is evaluated as an image-goal planner on real and simulated robots.

1 Introduction

  • The study targets the uncharacterized scaling of latent robotic world models, a question made consequential by scarce and expensive high-quality robot interaction data.A scaling law could help practitioners choose between a larger model, more demonstrations, or more compute, and identify data-budget saturation.Source: §1 · p. 2
  • RoboJEPA is an action-conditioned latent world-model family trained on a unified mixture of 23 public manipulation datasets spanning 12 embodiments and 26 action spaces.It predicts the next latent state deterministically in one forward pass using a frozen V-JEPA 2.1 encoder’s representation space.Source: §1 · pp. 2–3
  • RoboJEPA’s out-of-distribution latent forward-prediction loss follows a compute scaling law on both DROID and RoboCasa holdouts, and predicts downstream planning performance.On RoboCasa, only models trained above the stated compute threshold solve object-interaction tasks via non-greedy planning.Source: §1 · p. 3
  • Real-robot success rates improve with model size across planning episodes, while models with at least 2B parameters keep the gripper closed after grasping in object-manipulation tasks.The evaluation spans more than 50,000 episodes across two robot platforms and includes grasping, lifting, and pick-and-place.Source: §1 · p. 3
  • The authors release all model checkpoints and training and robot-deployment code, and report an 8B-parameter JEPA predictor as the largest trained to date.They describe the work as the first to establish compute-optimal scaling laws for action-conditioned robotic world models trained on large-scale, multi-embodiment real-robot data.Source: §1 · p. 3

2 Method and Data

RoboJEPA predicts future visual features from robot actions, proprioceptive states, and encoded camera views, then rolls those predictions forward to plan toward a goal image. Its architecture and training support heterogeneous robotic inputs, while its decoder renders imagined features as video for qualitative inspection.

2 Method and Data

  • RoboJEPA plans toward a single encoded goal image by searching for actions whose imagined latent rollout reaches the goal.It uses receding-horizon cross-entropy method planning and rolls the model forward under candidate action sequences.Source: §2 · pp. 3–4

2.1 RoboJEPA Architecture

  • The frozen V-JEPA 2.1 encoder supplies multiview latent features, and the predictor uses visual, action, and proprioceptive tokens to deterministically predict future features.The architecture is designed to accommodate differing view counts, controller types, and proprioceptive information; its attention is causal and local to at most eight past timesteps.Source: §2 · p. 3, Figure 3 · p. 4, §2.1 · pp. 4, 6
  • Training combines one-step next-feature prediction with autoregressive prediction from a shorter visual prefix, using the same per-token ℓ1 objective.The rollout loss trains on the same target sequence while supplying the full action sequence; this is intended to improve robustness to model errors and suppress error accumulation.Source: Figure 3 · p. 4, §2.1 · p. 6

2.2 Data, Training and Inference

  • Models range from 22M to 8B parameters, while training accommodates variable camera views and resolutions and planning uses shared hyperparameters across embodiments and tasks.A rolling KV-cache and multigpu CEM make planning take several seconds even for an 8B model, though it is not real-time.Source: §2.2 · pp. 7–8

2.3 Diffusion Decoder

  • The diffusion decoder turns predicted latent features into pixel video and imagines motion between the world model's more sparsely produced feature frames.It expands features produced at 5 Hz into an RGB stream at 20 FPS.Source: §2.3 · p. 8

3 Experiments

RoboJEPA’s prediction error follows a second-order power law, and planning performance improves with compute, including on long-horizon tasks when training uses longer rollouts. Real-robot results and decoded latent rollouts show scaling-related improvements in task success and predicted visual detail.

3.1 World Model Scaling Laws

  • The second-order power law extrapolates best: held-out errors are 0.6 and 1.4×10−3 on DROID and RoboCasa, roughly 2–3× below the standard power law.Fits use 22M–2B models and evaluate extrapolation on held-out 4B and 8B models.Source: Table 2 · p. 9, §3.1 · p. 9

3.2 Downstream Planning Scaling

  • Greedy RoboCasa planning forms clear compute-performance frontiers: end-effector reaching emerges around 10^20 FLOPs and moving while holding an object around 3 × 10^20 FLOPs.These simple tasks saturate early.Source: Figure 6 · p. 10, §3.2 · p. 10

3.3 Better Scaling for Long Horizon Planning

  • Short-horizon K=2 training yields weak, frontier-less obstacle-reach scaling; K=10 cooldown improves prediction error and restores predictable rewards, with avoidance emerging near 10^21 FLOPs and saturating near 10^22 FLOPs.Long-horizon cooldown adds higher-resolution inputs and is run from flat-learning-rate checkpoints rather than retraining from scratch.Source: Figure 8 · p. 11, §3.3 · p. 11

3.4 Real World Robotic Capabilities

  • On DROID, RoboJEPA success generally rises with model size across grasp, lift, and pick-and-place; π0.5’s 53% pick-and-place and 0% lift results are contextual, not directly comparable baselines.The VLA comparisons differ in goal specification and training; RoboJEPA plans toward a single goal image.Source: Figure 10 · p. 12, §3.4 · p. 12

3.5 Qualitative Analysis

  • Visualizations hold the encoder and decoder fixed while varying model size; larger predictors track dynamics and detail better, with gripper-object contact and structured folding appearing only at sufficient scale.Source: §3.5 · p. 13
  • After DROID fine-tuning, the 8B model imagines three views simultaneously at 720p, with side-camera content propagating into wrist-camera predictions.Source: §3.5 · p. 13

4 Related Work

Related work establishes scaling for many model families, but predictive scaling laws for action-conditioned robotic world models trained on large-scale, multi-embodiment real-robot data remain an open area. RoboJEPA distinguishes itself by fitting explicit laws for training compute and connecting prediction accuracy to downstream planning.

Joint-Embedding Predictive Architectures

  • Prior latent JEPA work includes action-conditioned prediction for zero-shot image-goal manipulation, while RoboJEPA scales the predictor using a unified, large-scale robot-data mixture.The authors state that the methods are not directly comparable because their training corpora differ.Source: §4 · pp. 13–14

World Models for Robotics

  • RoboJEPA predicts in a frozen V-JEPA representation space, contrasting with pixel-space world models that reconstruct or generate visual outputs.The paper presents latent prediction as avoiding the computational cost of pixel-level reconstruction at every step.Source: §4 · p. 14

Robotic Foundation Models from Demonstrations

  • RoboJEPA uses the same broad robot-demonstration data substrate as policy work but trains a goal-agnostic predictive model rather than a task-conditioned policy.The paper says the world model can evaluate or improve policies and plan zero-shot for new goals at test time.Source: §4 · p. 14

Scaling Laws

  • Robotics scaling laws remain largely open because high-quality interaction data is scarce, motivating systematic study of action-conditioned models on multi-embodiment robot demonstrations.The cited work characterizes forward-prediction loss against predictor parameters and demonstration tokens, and reports transfer to downstream planning.Source: §4 · pp. 14–15

Scaling world models

  • RoboJEPA scales training compute for a pure world model and fits predictive laws for future-embedding accuracy, unlike work reporting only empirical scaling trends.The paper contrasts predictive extrapolation with trends that do not quantify future improvement, saturation, or compute allocation.Source: §4 · p. 15

5 Conclusion

RoboJEPA scales action-conditioned latent prediction from 22M to 8B parameters and establishes compute-based laws for real and simulated data. Its offline imagination error tracks online planning performance, while the study’s laws apply to a fixed encoder and a corpus already near data saturation.

5 Conclusion

  • Offline prediction scaling laws correlate with downstream planning performance, making imagination error a reliable proxy for real-robot evaluation while reducing its computational cost.The laws are established on DROID real-world and RoboCasa simulated data.Source: §5 · p. 15
  • RoboJEPA scales its predictor from 22M to 8B parameters and fits second-order power laws for world-modeling error against training compute.The 8B model is described as the largest JEPA predictor trained to date.Source: §5 · p. 15
  • Distinct manipulation capabilities emerge at separate compute thresholds, ordered from end-effector control through object and scene reasoning to scene dynamics.On real robots, planning success rate and qualitative behavior improve as the world model scales.Source: §5 · p. 15
  • The study scales the predictor over a fixed V-JEPA 2.1 encoder and a fixed corpus whose fits are already close to data saturation.The authors identify more diverse interaction data as a likely requirement for pushing the frontier further.Source: §5 · p. 15

A Model Architecture: Additional Details

The appendix details the predictor’s architectural choices, cross-embodiment inputs, and data-unification setup. It describes axis-specific positional encoding, view conditioning, bounded-window attention, and a controlled sweep across eight predictor scales.

A Model Architecture: Additional Details

  • RoboJEPA shares V-JEPA 2’s backbone family and next-frame latent prediction, while adding normalization, position-encoding, and conditioning changes for cross-view, cross-embodiment modeling.The added components include RMSNorm and QK-norm, a fourth view RoPE axis, and explicit per-embodiment and per-view conditioning.Source: A Model Architecture: Additional Details · p. 24, Table 4 · p. 25

A.1 Factorised multi-axis RoPE

  • Factorised RoPE assigns channel sub-blocks to time, height, width, and view axes, with independent rotations that encode relative positions across axes.The paper says this relative-position property supports generalization across views and training and deployment patch grids.Source: §A.1 · pp. 25–26

A.2 Per-block view bias under grouped-query attention

  • A learnable per-layer view bias supplements view-axis RoPE to provide camera identity information to visual-token queries and keys.The zero-initialized bias is added after RoPE and is not applied to action or state tokens.Source: §A.2 · pp. 26–27

A.3 Attention masking

  • Attention is causal across timesteps with a local window of eight, while tokens within each timestep attend omnidirectionally.The bounded window keeps per-step attention cost constant and enables a KV cache of O(Wt · V · Hp · Wp) keys per layer.Source: §A.3 · p. 27

A.4 Per-embodiment input projection

  • Per-embodiment action and state projections let one predictor handle heterogeneous input conventions, while token-style conditioning allows step-dependent action effects.Action and state tokens attend to visual and other state/action tokens within the same step and local temporal window.Source: §A.4 · p. 27

A.5 Robot-id breakdown

  • The 12 physical platforms map to 26 robot IDs by treating distinct platform and action-source or state-convention pairs separately.The breakdown comprises 12 default state-delta IDs and 14 extra splits.Source: §A.5 · p. 27, Table 5 · p. 28

A.6 Predictor scaling sweep

  • The eight predictor variants span 22M to 8B parameters, changing depth, width, and head counts while holding the remaining block recipe fixed.The head dimension and MLP hidden ratio are also held constant.Source: §A.6 · p. 28, Table 6 · p. 29

B Data: Additional Details

  • The data appendix describes converting heterogeneous datasets to a common temporal grid, padded action and state widths, and semantic camera vocabulary.It also documents the storage format, dataset-specific conversions, sampling, and per-dataset breakdown.Source: B Data: Additional Details · p. 29

B.1 Dataset mixture and unification

The data pipeline combines varied robot demonstrations by normalizing action conventions, sampling, storage, and sensor representations. The appendix records both dataset-level composition and conversions needed to produce a shared input format.

B.1 Dataset mixture and unification

  • Training benefits from multiple action parameterizations of the same demonstration, and the mixture spans 12 physical platforms represented by 26 platform/action-space IDs.The authors report that varying action parameterizations improved the world model at fixed training compute.Source: §B.1 · pp. 29–30

B.2 Per-dataset breakdown

  • Table 7 provides the full per-dataset breakdown of the 23-dataset, 26-embodiment mixture, including synchronized-proprioception hours and per-rank sampling weights.Table 1 is the compressed main-text view.Source: Table 7 · p. 30, §B.2 · p. 30

B.3 Standardised on-disk format

  • Each trajectory is repackaged into a shared object-storage layout with camera video, robot-state data, optional controller commands, and episode metadata.The sampler uses an index of episode paths, lengths, and camera availability; clips sample 16 frames at the world-model rate.Source: §B.3 · pp. 30–31

B.4 Per-dataset postprocessing

  • Dataset-specific postprocessing reconciles video, kinematics, pose, gripper, and multi-arm representations before data enter the shared stream.Examples include reconstructing end-effector pose from joint encoders, converting orientations to xyz Euler form, and normalizing gripper signals to a common [0, 1] scale.Source: §B.4 · p. 31

B.5 View-mixed-rank training: mixture composition

The 1B training mixture combines one- and two-view clips while standardizing temporal stride, action inputs, image preprocessing, and camera identities across heterogeneous sources.

B.5 View-mixed-rank training: mixture composition

  • Two view-groups share predictor parameters and contribute 768 view-clips (512 clips) per optimizer step, with both view counts represented in every step.The mixture is split 50/50 by clip, while two-view clips contribute two view-clips each.Source: §B.5 · pp. 32–33

B.6 Frame rate, decimation and the per-frame mask

  • Source videos are decimated to a common world-model rate, and padded frames are masked out of the prediction losses.The mask is shared across views within a clip; deployment bridges the 5 Hz model rate to the 10 Hz Franka controller with two substeps per prediction.Source: §B.6 · pp. 33–34

B.7 Action conditioning details

  • Actions use state deltas for most sources and concatenated high-rate controller commands for DROID and RoboCasa, then are padded to a 64-dimensional input slot.The native state-delta widths vary by platform; high-rate command chunks preserve controller signals within the fixed world-model stride.Source: §B.7 · p. 34

B.8 Image preprocessing

  • Training clips are randomly resized and center-cropped to 240×320, with geometry-distorting augmentations disabled before frozen-encoder feature extraction.The encoder produces a 15×20 patch grid with 1664 channels per patch.Source: §B.8 · pp. 34–35

B.9 Per-camera enumeration

  • Physical cameras map to seven semantic camera ids, so cameras assigned the same role share a learned view bias across datasets.The mapping is based on per-dataset regex rules; the predictor consumes the semantic id rather than the original camera name.Source: §B.9 · p. 35, Table 9 · p. 35

C Training: Additional Details

The appendix details the batch and cooldown procedures used to train RoboJEPA, plus a separate high-resolution, three-view DROID finetuning run.

C Training: Additional Details

  • For 4B and 8B long-horizon finetuning, data and tensor parallelism split sequences across two GPUs to accommodate K=10 rollouts.Pretraining and short-horizon prediction instead use FSDP2 for these model sizes.Source: C Training: Additional Details · p. 35

C.1 Per-rank batch decomposition and view groups

  • The 1B run processes 512 clips and 768 clip-views per optimizer step across equal-sized one- and two-view groups.The encoder processes each view independently, while the predictor handles each clip as a single sequence.Source: §C.1 · pp. 35–36

C.2 Cooldown phase

  • Cooldown resumes the stable-phase checkpoint, raises the autoregressive horizon from K=2 to K=10, and linearly decays learning rate to zero over 5,000 steps.It also splits training across four view-by-resolution groups; weight decay remains 0.1 throughout.Source: §C.2 · p. 36

C.3 Training hyperparameters

  • Table 10 summarizes the 1B predictor’s additional loss, optimizer, and cooldown settings, omitting settings already listed in Table 6.The loss is defined in Equation (5), and AdamW follows Loshchilov and Hutter (2019).Source: §C.3 · p. 36, Table 10 · p. 37

C.4 DROID 3-view Finetuning at 720×1280

  • The DROID finetuning run uses three fixed exterior-left, exterior-right, and wrist views at 720×1280 over 16 timesteps, yielding about 170,000 input tokens per sequence.It uses a K=10-step autoregressive rollout and trains for 2,000 optimizer steps.Source: §C.4 · p. 37

D Diffusion Decoder: Additional Details

A diffusion decoder turns frozen V-JEPA features into visualizations of world-model rollouts, while qualitative filmstrips compare predictions across model sizes and against recorded video.

D Diffusion Decoder: Additional Details

  • The decoder is visualization-only: its training uses frozen V-JEPA features, and it is not invoked inside CEM or trained jointly with the world model.The V-JEPA encoder, predictor, and Cosmos tokenizer are also frozen; only the decoder parameters are trainable.Source: D Diffusion Decoder: Additional Details · p. 38, Table 11 · p. 40
  • V-JEPA features condition a diffusion transformer that denoises Cosmos latents aligned one-to-one with V-JEPA frames.The features are spatially upsampled and concatenated with noisy Cosmos inputs; the decoder reconstructs appearance and imagines intervening RGB frames.Source: D Diffusion Decoder: Additional Details · p. 38
  • The decoder predicts clean signals with unweighted x0-MSE and independently sampled noise levels for each frame.The training uses a log-SNR cosine schedule and zero terminal SNR; the per-frame noise sampling supports prefix-conditioned generation at inference.Source: D Diffusion Decoder: Additional Details · pp. 38–39

E Qualitative World-Model Imagination Examples

  • Filmstrips show decoded rollouts for model sizes from 22M to 8B beside ground truth, sampled at five equal-time points across the first 20 seconds.Large figures use 480×640 resolution and small figures use 240×320; camera-view count is fixed per embodiment.Source: E Qualitative World-Model Imagination… · p. 42

E.1 Large resolution (480×640)

Qualitative rollouts illustrate how imagined manipulation varies with scale and resolution; the appendix also details CEM planning and its Franka deployment.

E.1 Large resolution (480×640)

  • At 480×640, folding structure appears only at larger scales, gripper-object contact develops only for larger models, and finer details are preserved as scale increases.The examples include AgiBot, Open X-Embodiment Bridge, and RoboCasa rollouts with one or two views.Source: §E.1 · p. 42, Figure 12 · p. 43, Figure 13 · p. 44

E.2 Small resolution (240×320)

  • At 240×320, contact with the manipulated object develops only in larger models, while Bridge-V2 rollouts show a clear progression with model scale.The examples also include two-view LeRobot and Franka/DROID rollouts.Source: §E.2 · p. 46, Figure 16 · p. 47, Figure 17 · p. 48

F.1 Full CEM algorithm

  • CEM repeatedly samples action sequences, rolls them through the world model, scores final-step goal distance, and refits a Gaussian to elite sequences.Across environment steps, it warm-starts by shifting the previous plan and reuses its tail.Source: §F.1 · p. 51, §E.2 · p. 51

F.2 Distributed elite selection

  • Distributed CEM gathers samples and rewards across ranks so every rank selects elites from the same global pool.This allows the elite count to exceed the per-GPU sample count; the DROID configuration uses 16 samples per GPU and 20 elites.Source: §F.2 · pp. 51–52

F.3 Spherical translation sampling

  • Spherical translation sampling gives the proposal noticeable mass near small displacements, helping stabilize refinement near a warm-started plan.Its radius-angle parameterization also lets norm clipping preserve angular content.Source: §F.3 · p. 52

F.4 Translation ℓ2-ball projection

  • Translation actions are projected onto an ℓ2 ball to enforce per-step end-effector displacement limits despite aggressive elite refits.The stated bounds are 0.05 m for DROID and 0.10 m for RoboCasa; gripper actions are clipped separately.Source: §F.4 · p. 52

F.5 Multi-view goal handling

  • Multi-camera goal distances use configurable per-view weights, and unavailable goal views are handled by setting their weights to zero.This supports using a multi-view checkpoint for single-camera DROID episodes without retraining.Source: §F.5 · p. 52

G.1 Franka: DROID Platform

  • On the Franka, the planner targets a goal image using external and wrist-camera views, then executes translation and gripper actions with orientation held fixed.The blocking control loop waits for each commanded target before returning the next observation.Source: §G.1 · p. 53, Figure 20 · p. 53

G.2 Real-robot tasks and evaluation protocol

The real-robot protocol evaluates RoboJEPA on three Franka/DROID tasks using image goals, fixed planning settings, and manually scored episodes. The deployment planner searches over end-effector translation and gripper commands while holding orientation fixed.

G.2 Real-robot tasks and evaluation protocol

  • RoboJEPA is evaluated on Grasp, Object Lift, and Pick and Place, with success criteria defined separately for each task.Pick and Place success does not require satisfying the Object Lift criterion.Source: §G.2 · p. 54
  • For each model–task pair, RoboJEPA is run for 30 episodes and VLA baselines for 50; RoboJEPA models use the most-compute checkpoint at each size and shared planning settings.Progress is mean episode score, while success rate is the fraction of episodes with 100% progress.Source: §G.2 · p. 54
  • Each RoboJEPA episode receives a single final image as its goal, without intermediate image subgoals or task-specific fine-tuning.The VLA baselines receive text instructions and also receive no task-specific fine-tuning.Source: §G.2 · p. 54

G.3 Real robot planning hyperparameters

  • The real-robot planner searches 3D end-effector translation and a 1D gripper command, with rotation frozen, scores imagined rollouts by latent-feature distance, and replans after executing actions.The hyperparameter table gives the configuration used across the real-robot experiments.Source: §G.3 · p. 54, Table 13 · p. 55

G.4 RoboCasa Custom Environments

These appendices specify RoboCasa tasks and planning settings, show qualitative real-robot behavior, and describe inference optimizations. Across model sizes, larger models show more adaptive recovery; the inference engine targets dispatch overhead, cache reuse, and a smaller FP8 speedup.

G.4 RoboCasa Custom Environments

  • ObjectPush tests contact-rich, non-prehensile dynamics by moving an object through gripper-closed contact, with push direction restricted to a 60° cone around the forward axis.Reach isolates free-space motion; ObstacleReach requires an arc around an appliance; ObjectReach moves a scripted-grasped banana, can, or mug toward its goal.Source: §G.4 · pp. 56, 59

G.5 RoboCasa planning hyperparameters

  • Across the four RoboCasa tasks, the closed-loop CEM planner shares its settings except for predictor horizon; ObjectReach additionally searches over the gripper after a scripted grasp.Reach, ObstacleReach, and Push search over 3D end-effector translation.Source: §G.5 · p. 59, Table 14 · p. 60

G.6 Real-Robot Task Examples

  • Real-robot examples show ten episodes each for Grasp, Lift, and Pick-and-Place, with goals specified by a single image from external and wrist cameras.The initial robot pose and goal pose coincide, so the visible difference is the manipulated object’s configuration.Source: §G.6 · p. 60

G.7 Real World Robotic Capabilities

  • Larger RoboJEPA models shift from smooth but single-attempt behavior toward more adaptive execution and recovery after unsuitable approaches.Small models’ main weakness is gripper control; 300M and 1B models show intermediate behavior. The comparison with VLA models is limited by differences in goal specification, training data, and controller interfaces.Source: §G.7 · p. 60

H Key-Value Cache: Additional Details

  • The KV cache is bounded by the local temporal window, replaces the oldest frame as prediction advances, and shares observation-context keys and values across CEM candidates.Later predictions retain candidate-specific caches because their action sequences differ.Source: H Key-Value Cache: Additional Details · p. 64

I.1 Where the time goes (why the model is not matmul-bound)

  • At steady state, attention and fused pointwise operations take more GPU time than GEMMs, while dispatch adds about 31% of wall time.The reported decomposition is for batch size 16, a full cache, and two views; dispatch dominates even more at batch size 1.Source: §I.1 · p. 64, I Inference Engine: Additional Details · p. 64

I.2 AOT Inductor compilation: from dispatch to deployment

  • AOT compilation reduces launched kernels from 8,975 to 1,291, cuts dispatch overhead 23×, and reduces cold start from 15–20 minutes to 1.2–1.5 seconds.Shared weights reduce resident memory from about 108 GB to 14 GB.Source: §I.2 · p. 65, Table 15 · p. 65

I.3 Static FP8 path: the last small lever

  • Static FP8 is a small final lever: its advantage is largest at low cache occupancy and shrinks to about 5% when attention dominates at full cache.The FP8 outputs match the BF16 reference within the spatial-sensitivity benchmark’s noise floor.Source: §I.3 · pp. 65–66, Table 16 · p. 66

I.4 Optimisations that did not work

Several inference optimizations failed to improve the workload because their overheads outweighed their benefits or existing compilation already handled the target bottleneck.

I.4 Optimisations that did not work

  • Dynamic FP8 took 177 ms versus about 100 ms for BF16 because per-call activation scaling added more cost than the GEMM saved.Static, pre-calibrated scales eliminate this overhead.Source: §I.4 · p. 66
  • Weight-only INT8 and W8A8/F8A8 variants were incompatible with the required KV-cache dtype or ran about 2× slower due to dequantization overhead.Source: §I.4 · p. 66
  • CUDA graphs were slower than the non-graph path, while TensorRT was slower and produced incorrect outputs; RoPE precomputation gave 0% speedup.AOT Inductor already fuses RoPE into surrounding Triton kernels.Source: §I.4 · p. 66
  • Although cuDNN attention was 2.4× faster than xformers in isolation, it was only about 3% faster in the AOT pipeline.Export caused AOT Inductor to lower SDPA variants to flash-attention regardless of the context-manager hint.Source: §I.4 · p. 66

J Scaling Laws: Additional Parametric Forms

This appendix details alternative scaling-law fits, training-compute accounting, and frontier selection. Long-horizon cooldown training yields a better high-compute frontier, while noisy planning outcomes require filtering to estimate that frontier.

J Scaling Laws: Additional Parametric Forms

  • For extrapolation, the appendix fits BNSL and UNSL on 47 small-model checkpoints (22M–2B) and reports mean absolute held-out error across 33 large-model checkpoints (21 at 4B, 12 at 8B).The prediction target is mean rollout-ℓ1 over horizons 1–9.Source: J Scaling Laws: Additional Parametric… · pp. 66–67

J.1 Broken Neural Scaling Law (BNSL)

  • The single-break BNSL treats loss as a function of total compute, with an irreducible floor and a smooth transition that changes its log–log slope.The listed fitted parameters are for the DROID and RoboCasa holdouts.Source: §J.1 · p. 67, Table 17 · p. 67

J.2 Unified Neural Scaling Law (UNSL)

  • The fitted UNSL is a surface over model size and data, specialized for single-epoch runs and lacking a closed-form compute-optimal envelope.Its stated simplifications collapse dataset size and steps into one data axis and omit the overfitting and hyperparameter terms.Source: §J.2 · pp. 67–68, Table 18 · p. 68

J.3 Scaling Laws for Short- and Long-Horizon Prediction Training

  • On both DROID and RoboCasa, long-horizon cooldown training crosses below short-horizon training at about 5×10^20 FLOPs and continues to a lower floor.The long-horizon cooldown uses K=10 rollout and mixed resolutions; it starts from flat-learning-rate checkpoints and adds no new data.Source: §J.3 · p. 68

J.4 Per-Step Training Compute across Stages

  • Training compute is accumulated as per-step FLOPs times optimizer steps, with per-step totals including gradient accumulation, data-parallel replicas, and configured rollouts.The long-horizon recipe’s per-step cost differs from the short-horizon recipe, and 8B finetuning costs about 19× a long-horizon step and 62× a short-horizon step.Source: §J.4 · pp. 68–69, Table 19 · p. 69, Table 20 · p. 70

J.5 Stochastic upper-envelope selection

  • The stochastic upper-envelope procedure rejects sufficiently low points before fitting, avoiding a frontier distorted by downward outliers or by fitting the full scatter.It adapts a running mean and variance over compute-sorted points; rejection thresholds vary by task to reflect evaluation noise.Source: §J.5 · pp. 70–72

K Encoder Choice and World-Model Scaling

Restated encoder benchmarks show V-JEPA 2.1 substantially outperforming V-JEPA 2 on dense tasks, motivating the authors’ hypothesis that dense features matter for scaling robotic world models. The authors also report instability when scaling a predictor on V-JEPA 2.

K Encoder Choice and World-Model Scaling

  • Scaling the predictor on V-JEPA 2 often led to collapse beyond a certain model size, while increasing dataset size sometimes caused instability or divergence.The authors hypothesize that this behavior may be attributable to the encoder’s features.Source: K Encoder Choice and World-Model Scaling · p. 73
  • In restated published evaluations, V-JEPA 2.1 is on par in classification, lower-scoring on IntPhys2, and substantially stronger on dense tasks; this is not a controlled ablation.The table reports prior evaluations rather than evaluations conducted in this work.Source: K Encoder Choice and World-Model Scaling · p. 73, Table 21 · p. 73
  • The authors hypothesize that high-quality dense features are more important for scaling robotic world models than features associated with physical understanding.The hypothesis follows the reported dense-task performance gap; model sizes and pretraining recipes differ, so the comparison is not a controlled ablation.Source: K Encoder Choice and World-Model Scaling · p. 73, Table 21 · p. 73

Related papers

Loading 2610.10515v1…