PaperlayerView exact revision on arXiv

Source-linked AI summary

ARC-Bench: Closed-Loop Replanning Masks Broken Action Ranking in Frozen JEPA World Models

Zhengshu Zhang, Zhiyuan Li

arXiv:2609.05461v1cs.AIcs.LGcs.RO

TL;DR

Reward-free latent world models assume latent distance can rank actions by true cost, but this paper shows that assumption fails across navigation and manipulation. ARC-Bench audits released JEPA-style objectives with fixed candidates, and the results show severe misranking that persists across visual backbones while frequent replanning masks it.

  • Problem

    Frozen reward-free world models rely on latent closeness to rank candidate actions, but this action-rankability assumption had not been directly benchmarked for released JEPA-style objectives.

  • Method

    ARC-Bench uses a no-leak fixed-candidate protocol to compare frozen latent-objective rankings with true executed costs, with controlled encoder-swap audits and explicit safeguards.

  • Results

    Severe, structural misranking appears across navigation and manipulation, persists across DINOv2 and V-JEPA backbones, and closed-loop success collapses when replanning is reduced.

  • Takeaways & Limitations

    Closed-loop success rates can overstate frozen-representation rankability because frequent replanning overwrites ranking errors before they compound.

  • Takeaways & Limitations

    The controlled V-JEPA extension covers Push-T and MetaWorld reach-wall encoder swaps rather than every released video-JEPA planner or the full space of video-control domains.

Abstract

from arXiv · show

Reward-free latent world models plan by scoring candidate actions with distances in a frozen latent space: an action is preferred if its predicted future embedding lands closer to the goal embedding. This silently assumes that latent closeness is action-rankable, i.e., that ordering candidates by latent distance agrees with ordering them by true cost. We audit this assumption directly. We introduce ARC-Bench, a no-leak, fixed-candidate protocol that measures whether frozen JEPA-style objectives rank candidate actions correctly, and apply it to official released JEPA-WM checkpoints across navigation and manipulation-style control. The assumption fails, severely and structurally: on the official manipulation audits the top-scored candidate is almost always suboptimal, and the same inversion appears in the maze domains. A controlled visual-backbone extension shows that the defect persists when DINOv2 is replaced by video-pretrained V-JEPA 1 and V-JEPA 2 encoders at ViT-L/ViT-G scale. Provenance, undertraining, matched-budget backbone controls, and metric-circularity controls rule out trivial explanations. We then explain why this defect has stayed invisible: closed-loop replanning masks it. When we reduce the planner's replanning frequency, success collapses in both a navigation and a manipulation domain, and the episodes rescued by frequent replanning are enriched for severe first-plan ranking failures in the PointMaze first-plan diagnostic. Closed-loop success rates therefore systematically overstate the rankability of frozen latent representations. ARC-Bench supplies the measurement, and the masking mechanism the explanation, for methods that adapt, amortize, or replan around latent-space planners without directly auditing released JEPA-WM action rankability.

1 Introduction

This paper directly audits whether frozen latent distances correctly rank candidate actions, finding severe, structural misranking across navigation and manipulation that persists across visual backbones and is masked by frequent replanning.

  • 1 Introduction: The latent-closeness assumption fails because candidates that look closer in embedding space can have worse executed outcomes than farther candidates.The paper names an anchor with this inversion a Mirage.
  • 1 Introduction: Reducing replanning exposes the defect: PointMaze success falls from 89.6% to 61.8%, and Push-T success falls from 36.1% to 12.5%.Episodes rescued by frequent replanning are enriched for severe first-plan failures, showing how closed-loop control can mask scorer errors.
  • 1 Introduction: ARC-Bench audits action rankability with a no-leak fixed-candidate protocol and reportable metrics for frozen world-model objectives.The protocol includes provenance, undertraining, and circularity controls.
  • 1 Introduction: Official JEPA-WM objectives misrank fixed candidates across navigation and manipulation, with wrong-anchor rates of 96.8–100% in two 750-anchor audits.Strong Mirage inversions also appear in the maze domains.
  • 1 Introduction: The failure persists after replacing DINOv2 with frozen V-JEPA 1 and V-JEPA 2 encoders: Push-T top picks remain wrong on 91.1–97.3% of useful anchors, while MetaWorld reach-wall Hit@1 and Hit@3 remain zero.The downstream predictor, objective, candidate protocol, and training budget are controlled in the encoder-swap audits.
  • 1 Introduction: 100% of anchors have wrong raw terminal-Euclidean rankings in topology-stress audits; Manhattan matches U-Maze and only partially improves MiniGrid/FourRooms.These results indicate the misranking is not confined to one checkpoint or layout.

2 Related Work

Related work addresses latent-distance unreliability through value functions, trajectory straightening, amortized action prediction, adaptive replanning, or test-time adaptation rather than directly measuring released JEPA-style scorer rankability.

  • 2 Related Work: Value-driven world models score latent futures with learned rewards or values, while reward-free models use latent distance as the planning objective.The paper positions ARC-Bench as a direct audit of the latter scorer.
  • 2 Related Work: Trajectory-straightening methods improve Euclidean progress proxies by learning spaces where feasible latent trajectories are less curved.This treats latent-distance unreliability through representation geometry.
  • 2 Related Work: Amortized planners, adaptive replanning, and test-time adaptation route around adjacent failures through action prediction, cadence changes, or model updates.These approaches address planning without directly establishing released scorer rankability.

3 ARC-Bench: Auditing Action Rankability

ARC-Bench audits frozen world-model scorers by ranking fixed candidate action sequences against offline ground-truth terminal costs, with held-out evaluation and provenance controls. The protocol spans navigation and manipulation environments and includes controlled encoder-swap diagnostics to test whether stronger visual backbones repair rankability.

  • 3.1 Problem formulation and metrics: Each audit anchor pairs a context, goal, and fixed candidate set with offline terminal-state costs that remain hidden from deployable scorers.The protocol defines candidate quality by distance between executed terminal and goal terminal states.
  • 3.1 Problem formulation and metrics: ARC-Bench reports top-1 regret, Hit@1 and Hit@3, reciprocal rank, pairwise accuracy, Spearman correlation, wrong-anchor rate, and strong-Mirage rate.Uncertainty is quantified with anchor-level paired bootstrap intervals.
  • 3.2 Evaluation controls: Learned scorers are selected on validation data and evaluated once on a locked held-out split, while oracle diagnostics remain non-deployable and separate.Pure topology-stress audits use fixed non-learned scorers without model selection or held-out tuning.
  • 3.3 Backbones, environments, and provenance: The audit covers PointMaze and Wall navigation diagnostics alongside Push-T and MetaWorld reach-wall manipulation control using official checkpoints where available.The PointMaze checkpoint matches the official repository byte-for-byte, and retraining controls do not recover ranking quality.
  • 3.3 Backbones, environments, and provenance: Controlled encoder-swap audits compare DINOv2 with V-JEPA 1 and V-JEPA 2 at ViT-L and ViT-G scales while holding downstream components, objectives, protocols, and training budgets fixed.These diagnostics test whether stronger video-pretrained visual features make latent distance action-rankable.
  • 3.4 A terminal-ranking probe, with circularity controls: A lightweight second-stage probe predicts candidate-induced terminal displacement from actions and context, then re-ranks candidates without using true terminals at test time.Its substantial prediction errors and weak cost correlations are used to control for metric circularity rather than to recover oracle terminals.

4 Fixed-Candidate Results: The Ranking Failure Is Structural

Across official, controlled-backbone, localizability, and topology-stress audits, latent-distance objectives repeatedly fail to rank fixed candidates by true cost. The failure persists across domains and backbones, while a diagnostic probe recovers part of the held-out ranking structure.

  • Official objectives: 96.8% of Push-T anchors and 100% of MetaWorld reach-wall anchors selected suboptimal top candidates under the official objective.Push-T Hit@1 was 3.2%, while MetaWorld Hit@1 was 0.0% and Hit@3 approximately 0.1%.
  • Video-pretrained encoders: 91.1–97.3% of controlled encoder-swap Push-T anchors remained wrong, while every MetaWorld backbone was wrong on every anchor.Hit@1 stayed below 9% on Push-T, and no MetaWorld model placed a good candidate in the top three.
  • Video-pretrained encoders: Positive global Spearman correlations for two V-JEPA 2 variants did not imply useful top-ranked actions because Hit@1 and Hit@3 remained zero.The correlation was largely a between-family effect, while within-family concordance remained near chance.
  • Localizability probe: The terminal-ranking probe reduced PointMaze top-1 regret from 0.496 to 0.195 and increased Hit@1 from 15.2% to 48.8%.The probe was evaluated on locked held-out candidate pools as a localizability diagnostic, not a proposed replacement planner.
  • Topology-stress audits: Terminal-Euclidean scoring was wrong on 100% of anchors with Hit@1 = 0 in both U-Maze and MiniGrid/FourRooms topology-stress audits.These supplementary audits isolate terminal-distance geometry rather than evaluating official JEPA-WM checkpoints.

5 From Ranking to Planning: Why the Failure Stayed Hidden

Finite-budget sampling does not remove scorer differences, and the gap between fixed-candidate rankability and online planning is measurable. Frequent replanning masks ranking errors, whereas longer open-loop commitments expose failures in navigation and manipulation.

  • Sampling budget: Increasing CEM-style sample budgets improves all scorers but preserves their ordering, so scorer quality controls selection quality at every tested budget.The finite-budget audit used B ∈ {2, 4, 8, 12} on cached score tables rather than online rollouts.
  • Fixed-candidate-to-closed-loop gap: Raw CEM succeeded on 10/12 episodes, while pure probe-guided CEM succeeded on 0/12 and mixed objectives remained near raw performance.On online-sampled candidates, probe ordering had mean Spearman near zero and argmin agreement of 5.6–11.1%.
  • Masking mechanism: Frequent replanning overwrites step-level ranking errors, but longer open-loop blocks expose them as task failures.The PointMaze first-plan diagnostic predicts stronger failures on episodes with severe initial ranking errors.
  • Closed-loop masking: Success fell from 89.6% to 61.8% on PointMaze and from 36.1% to 12.5% on Push-T when replanning was reduced.Discordant paired outcomes favored frequent replanning by 44–4 on PointMaze and 35–1 on Push-T.
  • Encoder-swap cadence: Every visual backbone lost a large fraction of Push-T success when forced to commit for six steps, while some MetaWorld models were already near failure at k = 1.The controlled cadence audit used fixed planner horizon and search budget with seed-level bootstrap intervals.
  • First-plan diagnostic: 41.7% of high-Mirage PointMaze episodes were rescued by frequent replanning versus 8.3% of low-Mirage episodes.Rescued episodes also had roughly twice the prior Mirage regret of non-rescued episodes: 4.35 versus 2.14.

6 Discussion

Closed-loop success can conceal severe single-step action-ranking failures in frozen latent planners. The paper argues that rankability should be reported alongside success and interpreted relative to replanning cost.

  • Closed-loop success conflates representation quality with the control loop’s ability to suppress ranking errors.
  • V-JEPA backbones can change closed-loop success without making latent distance a reliable candidate-ranking signal.
  • Reward-free planning methods should report single-step rankability alongside closed-loop success and state the replanning budget prominently.
  • Amortized search, adaptive replanning, and test-time adaptation implicitly work around the action-sensitive latent-geometry deficit measured by ARC-Bench.
  • When replanning is cheap, the ranking deficit acts as an efficiency tax; when replanning is costly or decisions are hard to reverse, it can become task failure.

7 Limitations

The evaluation has defined methodological and scope boundaries across diagnostics, official audits, backbone extensions, and masking experiments. The authors also limit the generality of their localizability and enrichment claims.

  • The localizability probe is a diagnostic on two maze domains, not a planning method, and its action-only deployable form does not transfer to closed-loop control.
  • Official Push-T and MetaWorld results audit generated-candidate scoring rather than full end-to-end CEM planner reruns.
  • The V-JEPA extension covers encoder-swapped Push-T and MetaWorld reach-wall models, not every video-JEPA planner or video-control domain.
  • Masking experiments cover two domains under reduced-budget CEM configurations, while Mirage-enrichment stratification is established only on PointMaze.

8 Conclusion

The paper finds that latent closeness is not reliably action-rankable across official frozen JEPA-WM checkpoints and controlled visual backbones. Closed-loop replanning commonly hides this defect by overwriting ranking errors before they accumulate.

  • Latent closeness is severely and structurally non-rankable across navigation and manipulation-style control on official released checkpoints.
  • The failure persists across controlled DINOv2 and V-JEPA visual backbones.
  • The defect is measurable in fixed-candidate audits, localizable, and immune to sampling budget.
  • Closed-loop replanning ordinarily makes the defect invisible by overwriting it faster than it can be observed.

A.1 Candidate-ranking audit data

The candidate-ranking audits use held-out fixed candidate pools and offline true costs that remain hidden from deployable scorers. Official manipulation audits use generated candidate rollouts aligned with released action-chunk formats.

  • Maze audits use locked held-out pools with 256 anchors per domain and 12 candidate action sequences per anchor.
  • Each candidate receives an offline true cost based on terminal-state distance to the goal, hidden from deployable scorers at test time.
  • Push-T and MetaWorld audits use generated candidate rollouts grouped in the action-chunk formats of the released JEPA-WM evaluators.

A.2 Environment routes and online planning settings

The closed-loop masking experiments vary only the replanning interval across PointMaze and Push-T while holding the planning configuration fixed. The paper also documents the provenance of its reproduced analyses and the scope of materials included for reproducibility.

  • Environment routes: PointMaze uses the official PointMaze CEM configuration on gym 0.23.1 with mujoco-py, while Push-T uses the official JEPA-WM online evaluation route.
  • Online planning settings: The experiments manipulate only the replan interval k across k ∈{1, 2, 4, 6}, using horizon 6, 5 CEM iterations, 64 samples, and 8 elites.Episode indices are paired across the sweeps.
  • Analysis provenance: The reported tables and figures reproduce official generated-candidate summaries, controlled V-JEPA encoder-swap evidence, fixed-candidate ranking comparisons, and topology-stress diagnostics.
  • Reproducibility materials: The arXiv source package includes manuscript source and figures but excludes large raw outputs, full logs, local environments, official checkpoints, and model weights.Paper-level evaluation code, plotting scripts, and sufficient CSV/JSON summaries are planned for public release after preparation.
Loading 2609.05461v1…