Source-linked AI summary
Does LeWorldModel Preserve Local Action Geometry?
Vasil Vasilev, Siddharth Nayak
TL;DR
Accurate prediction at a sampled action does not establish whether nearby action directions have the correct relative effects. The paper compares pretrained LeWM and simulator action sensitivity across scales, finding substantial directional mismatch at the smallest scale; whether this matters for control remains open.
Problem
Accurate prediction at a sampled action does not establish that nearby action directions have the correct relative effects, motivating evaluation alongside aggregate prediction and control scores.
Source: §1 · p. 1Method
Results
At the primary scale, the most sensitive directions differ by median 36° on Reacher and 42° on Cube; a tenfold perturbation increase reduces median normalized Gram discrepancy by about 70% and 47%, respectively.
Source: §3 · pp. 2–3Takeaways & Limitations
Action-sensitivity agreement depends on perturbation scale and output metric, and held-out response prediction is established only within the declared measurement space and scale.
Source: §3 · p. 3, §4 · p. 4Additional checkpoints are needed to estimate training-seed variation, and the documented Cube replay-protocol change limits reproducibility claims.
Source: §4 · p. 4
Abstract
from arXiv · showhide
We examine how closely pretrained LeWorldModel (LeWM) reproduces the effects of small action changes. Using matched simulator and model responses, we measure action sensitivity with Gram matrices at three perturbation scales. Across nearly a thousand states from Reacher and OGBench Cube, the most sensitive action directions differ by median angles of about $36^\circ$ and $42^\circ$, respectively, at the smallest scale. Increasing the perturbation tenfold reduces median normalized Gram discrepancy by roughly 70\% on Reacher and 47\% on Cube. Geometry measured in pixels and embeddings varies more across scales than the chosen physical-state geometry. On held-out directions, environment Grams achieve median embedding response errors of 17\% and 28\%, compared with 26\% and 38\% for an isotropic reference with the same trace. LeWM's errors are higher at this scale. These findings support scale-specific evaluation of action sensitivity; its effect on control remains an open question.
1. Introduction
World models and action-conditioned predictive systems support planning, but accurate prediction at a sampled action does not establish whether nearby action directions have the correct relative effects. This motivates evaluating action sensitivity alongside aggregate prediction and control scores.
1. Introduction
- World models support planning through learned predictive dynamics, and joint-embedding predictive architectures include action-conditioned systems such as V-JEPA 2 and LeWorldModel.The cited examples include Ha and Schmidhuber (2018), Hafner et al. (2019, 2024), Assran et al. (2023, 2025), Bardes et al. (2024), and Maes et al. (2026).Source: §1 · p. 1
- Accurate prediction at a sampled action does not establish whether nearby action directions have the correct relative effects.Source: §1 · p. 1
- The study evaluates pretrained LeWM on Reacher and OGBench Cube with simulator counterfactuals, a shared frozen encoder, calibration references, and held-out action directions across three output metrics.It distinguishes encoder-output sensitivity from physical response and goal-directed action choice.Source: §1 · p. 1
2. Measuring Finite-Scale Action Geometry
The method compares model and simulator responses to shared finite action perturbations, then evaluates their relative action geometry across scales and output spaces. It uses trace-normalized Gram discrepancy alongside calibration references and separate measures of gain and alignment.
2. Measuring Finite-Scale Action Geometry
- Each perturbation applies one shared normalized offset to all five actions in the recorded block, while the model predicts the next embedding and the simulator encodes the rendered endpoint from a restored state.Both maps use the same output space.Source: §2 · pp. 1–2
- The shared offset defines d action coordinates; response magnitudes are motivated by the local Jacobian approximation, which is tested at each scale.Contacts and rasterization can limit local linearity.Source: §2 · p. 2
- Relative sensitivity is measured by trace-normalizing each Gram matrix and computing the Frobenius discrepancy between model and environment matrices.The comparison discards common multiplicative gain; absolute gain, condition numbers, and eigenspace angles are reported separately.Source: §2 · p. 2
- Evaluation uses one released checkpoint per environment, 458 Reacher anchors and 500 Cube anchors, at perturbation scales 10^-3, 3 × 10^-3, and 10^-2.Five wrapper steps span 0.20 seconds on Reacher and 0.25 seconds on Cube; trajectories were regenerated after an audit of alignment, output coordinates, and state restoration.Source: §2 · p. 2
3. Results
LeWM captures some action structure but remains misaligned, and agreement varies with perturbation scale and output metric. Results include comparisons with calibration references and held-out response prediction in embedding space.
3. Results
- 36° on Reacher and 42° on Cube are the median primary-scale differences between the most sensitive action directions in the model and environment.Median normalized Gram discrepancy is 0.465 on Reacher and 0.508 on Cube.Source: §3 · p. 2, Figure 1 · p. 3
- Reacher’s model and environment condition numbers are both near 4.5 despite their directional mismatch, while Cube’s model median is roughly four times the environment value.About a third of Cube environment matrices are affected by the numerical floor, so the upper tail requires care; misalignment also remains for well-separated leading eigenvalues.Source: §3 · p. 2
- Cube has lower primary-scale discrepancy than all three references, while Reacher improves on rotation and shuffle but is closer to isotropy.The paired model-minus-shuffle median differences are −0.059 for Cube and −0.047 for Reacher.Source: Table 1 · p. 3, §3 · p. 3
- A tenfold increase in perturbation reduces median normalized Gram discrepancy by about 70% on Reacher and 47% on Cube.Physical-pose Gram changes are below one millionth on Reacher and approximately 0.03 on Cube, compared with pixel changes of about 0.11 and 0.08 and embedding changes of about 0.43 and 0.49, respectively.Source: §3 · p. 3
- Rendering, encoding, and nonlinear dynamics may each contribute to output-metric differences, but their individual effects remain unresolved.All three scales are reported because changing the perturbation changes the response under study.Source: §3 · p. 3
- Held-out embedding response WAPE is lower for environment Grams than isotropy, while model Grams have higher error.The paired median advantage of environment Grams is roughly six to eight percentage points on Reacher and eight to twelve on Cube.Source: Table 2 · p. 4, §3 · p. 4
- Physical-pose prediction has median WAPE below 0.04% in both environments, while pixel errors are about 5% on Reacher and 15% on Cube.For embedding displacements, median errors rise from 17% to 32% on Reacher and from 28% to 43% on Cube when predicting one-sided rather than centered responses.Source: §3 · p. 4
4. Discussion and Conclusion
LeWM captures part of the environment’s action structure, with agreement depending on scale and output metric. Whether improved sensitivity estimates help action selection remains open.
4. Discussion and Conclusion
- Held-out tests give the Gram a concrete interpretation: it predicts response magnitudes within the declared measurement space and scale.The original 50-state subsets gave identical repeated results, but later image differences required a documented change to the Cube replay protocol.Source: §4 · p. 4
- Additional checkpoints are needed to estimate variation across training seeds, and the Cube replay-protocol change limits reproducibility claims.Source: §4 · p. 4
- The implications for training and control remain open; signed, goal-directed responses and closed-loop performance would test whether better sensitivity estimates help action selection.Equal Grams can occur for opposite outcome directions, as in J_model = −J_env.Source: §4 · p. 4
A.1. Trace-normalized metric discrepancy
This section defines trace-normalized Gram discrepancy and complementary measures for comparing action sensitivity, then calibrates those measures against reference geometries. It also states numerical and interpretive limits on Gram agreement.
A.1. Trace-normalized metric discrepancy
- Trace normalization compares relative Gram geometry while removing common sensitivity scale; DG is the Frobenius distance between normalized model and environment Grams.Zero or non-finite traces are invalid, and the primary additive constant is zero.Source: §A.1 · p. 6
A.2. Eccentricity
- Eccentricity compares model and environment condition numbers, with positive Dκ indicating amplification and negative Dκ indicating suppression.Source: §A.2 · p. 6
A.3. Directional and subspace alignment
- Leading-direction and subspace angles compare Gram eigenspaces, but near-tied eigenvalues make directions unreliable; gap checks assess angle sensitivity.The maximum principal angle is used for Cube’s leading two-dimensional subspaces.Source: §A.3 · p. 6
A.4. Calibration and uncertainty
- Isotropy, random orthogonal rotation, and shuffled-state matching remove different information; uncertainty intervals quantify sampled-anchor uncertainty, not training-seed variation.Rotation preserves the environment spectrum while randomizing action-coordinate orientation; shuffling tests whether matching the current state matters.Source: §A.4 · pp. 6–7
A.5. Anchor drift in the new bank
- Cube’s new bank samples somewhat different trajectories because live states can drift from archived states; Reacher replay is nearly identical.Cube’s median maximum coordinate differences are 5.88 × 10−4 in qpos and 4.10 × 10−4 in qvel; all high-drift anchors are retained.Source: §A.5 · p. 7
A.6. Environment-relative interpretation
- Environment-relative geometry expresses model sensitivity in coordinates where the environment metric is locally Euclidean, but inverse metrics can amplify noise near singularity.This motivates using DG and principal-subspace angles as primary empirical comparisons.Source: §A.6 · p. 7
Appendix B. What Gram Agreement Does Not Establish
- Gram agreement captures relative response magnitudes, not response direction, overall scale, or goal-directed control performance.Opposite latent responses can have identical Grams, and isotropy as a training target would suppress genuine sensitivity differences in anisotropic environments.Source: Appendix B · p. 7
Appendix C. Bank Construction and Robustness
The appendix documents a revised current-simulator bank with matched finite differences, then checks robustness across perturbation scales, state drift, and numerical conditions. The results support treating scales separately and limit contact-specific conclusions.
Appendix C. Bank Construction and Robustness
- Eligible anchors must reproduce the live context image exactly after restoration, and paired unperturbed branches must agree in pixels and in qpos/qvel.Source: Appendix C · p. 7
- Cube is measured across a ten-fold perturbation range because rendering and contacts can introduce finite-scale effects; convergence to a smooth derivative remains untested.Each scale is its own finite-response measurement.Source: Appendix C · p. 8
- +0.083 (95% interval [0.064, 0.108]) for Reacher and −0.108 ([−0.128, −0.094]) for Cube were primary-scale paired model-minus-isotropic DG differences.The corrected Reacher measurements do not support the historical direction-preserving/eccentricity-amplifying interpretation; its median dominant-direction angle is 36.03°.Source: Appendix C · p. 8
- Historical gripper-contact labels are not transferred to regenerated trajectories, so contact-specific effects require event measurements on the live counterfactuals.Source: Appendix C · p. 8
- On Cube’s first 50 anchors, repeats were identical, but cross-scale DG ICC was 0.220 and the environment’s dominant direction shifted by a median 38.99° from 10−3 to 10−2.Reacher repeats were also exact; its median DG fell from 0.465 to 0.308 to 0.141 across scales, while dominant angles fell from 36.03° to 20.70° to 8.89°.Source: Appendix C · pp. 8–9
- The primary-scale directional mismatch persists after restricting to well-separated eigenvalues, though the condition-number floor limits interpretation of Cube’s upper tail.Both Cube top-1 gaps ≥0.1 retain 449 states with median angle 41.09°; the corresponding Reacher check retains 454 states with median angle 35.80°.Source: Appendix C · p. 9
- The revised bank regenerates trajectories in the current simulator from original sampling anchors, branches counterfactuals from live step 10, and uses matched finite differences for both systems.The new protocol supersedes historical estimates; the original collection runtime remains unavailable.Source: Appendix C · p. 9
D.1. Separate output metrics
The study constructs separate action-response Grams in physical pose, pose-plus-velocity, pixel, and embedding spaces. Each space uses its own scaling, so cross-space gain comparisons require care.
D.1. Separate output metrics
- Physical-pose responses use MuJoCo’s generalized-position tangent, scaled by 0.1 m for translation and 1 rad for rotation, then divided by √nv.The authors note that these units and coordinate weights are an explicit, non-unique choice.Source: §D.1 · pp. 9–10
- The supplemental pose-plus-velocity metric appends scaled velocity displacement over a five-step horizon, while pixels use flattened RGB values divided by 255√NRGB.The pose-plus-velocity concatenation is divided by 2.Source: §D.1 · p. 10
- Embedding Grams use the frozen encoder’s Euclidean metric, and gain comparisons across output spaces require care because units and coordinate weights differ.Grams are constructed independently in each space.Source: §D.1 · p. 10
- Within each output metric, scale dependence is measured by the Frobenius change between normalized environment Grams at the compared perturbation scales.The supplied definition compares the 0.01 and 0.001 scales.Source: §D.1 · p. 10
D.2. Held-out directions and error definitions
Held-out action directions test whether coordinate Grams predict response magnitudes on new simulator endpoints. The same sampled directions are reused across scales, and paired uncertainty is summarized across sampled states and episodes.
D.2. Held-out directions and error definitions
- Eight independent normalized Gaussian directions are sampled per anchor in raw half-range coordinates, with the same directions reused across perturbation scales.The seed is 20261021 plus sample ID, with an additional 100000 for Reacher.Source: §D.2 · p. 10
- Held-out endpoints do not construct the coordinate Gram or determine its gain, keeping the prediction test separate from fitting.Source: §D.2 · p. 10
- Each state contributes one WAPE combining its eight directions, while paired median-difference intervals use episode-cluster bootstrap draws with directions held fixed.These intervals quantify uncertainty across sampled states and episodes, not training seeds or direction resampling.Source: §D.2 · p. 10
- The one-sided response magnitude is measured separately at both positive and negative perturbations, since a centered secant can cancel effects present in one-sided displacement.The two tests are not equivalent.Source: §D.2 · p. 10
D.3. Magnitude calibration and all-scale results
The held-out magnitude test compares model Grams with isotropic and oracle-gain references, alongside all-scale measurements of embedding response error. Retained outliers and distinct output-space measures qualify the interpretation of median results.
D.3. Magnitude calibration and all-scale results
- The magnitude reference is isotropy with the environment Gram’s trace, and WAPE includes gain and finite-scale approximation error on new directions.Both references use environment trace information.Source: §D.3 · pp. 10–11
- Table 5 reports median embedding-response WAPE at all scales, including a one-sided environment-Gram column.Source: Table 5 · p. 11
- At the primary scale, model WAPE exceeds isotropy by 0.046–0.069 on Reacher and 0.138–0.189 on Cube; after oracle gain correction, differences are 0.007–0.031 and −0.065 to −0.036.The paired median differences are reported as WAPE fractions with 95% intervals; the oracle correction requires environment trace information.Source: §D.3 · pp. 10–11
- One Reacher state reaches about 117% physical-prediction WAPE while the other 457 states stay below 1%, and Cube’s median physical-pose WAPE rises to about 1.47% at ϵ = 0.01.All such outlier states remain in the analysis; the physical cause of several large angular responses is unresolved.Source: §D.3 · p. 11
- Figure 3 contrasts environment-Gram change across output spaces with held-out embedding-response magnitude error, displaying medians.Its replay protocols use each environment’s original LeWM checkpoint.Source: Figure 3 · p. 12
D.4. Replay protocol amendment and verification
The response-validation run uses the retained anchor bank and a separately versioned Cube replay protocol. A threshold-exceeding replay was documented and retried, while accepted-output checks stayed within specified bounds.
D.4. Replay protocol amendment and verification
- Response validation retains all 458 Reacher and 500 Cube anchors, distinct from the primary discrepancy bank and its 50-anchor repeatability checks.Fresh environment coordinate endpoints are evaluated alongside held-out endpoints.Source: §D.4 · p. 12
- Cube’s v3 protocol permits at most one RGB-level context or baseline difference and applies 1% embedding-response gates before recomputing model responses from fresh context.Partial v1/v2 runs are archived and not pooled with production.Source: §D.4 · p. 12
- Production v3 exceeded the 1% baseline embedding replay threshold at anchor 292, then completed after an unchanged production resume; the cause remains unknown.The thresholds and full anchor set remained unchanged.Source: §D.4 · p. 13
- Accepted v3 outputs had at most 0.02525% relative model-response change and 4.34×10−5 normalized model-Gram change, with zero repeated qpos/qvel errors.The earlier replay failure remains part of the reproducibility record.Source: §D.4 · p. 13