WorldGuide: Learning Success-Failure Boundaries in Latent World Models for Vision-Language-Action Policies

• VLA imitation learning supervises expert actions but offers limited explicit supervision about their effects; latent world models add future-representation prediction as an auxiliary objective.Prior latent world-model approaches use this objective to encourage action-relevant state-transition representations.Source: §1 · p. 1
• Success-heavy predictors may map failed interactions near successful futures, and failure trajectories alone do not ensure that outcome-relevant differences become prominent in latent space.Small contact or alignment discrepancies can change task outcome while contributing little to latent distance.Source: §1 · pp. 1–2
• WorldGuide studies failure data and matched success–failure supervision through failure-rich predictive pretraining, contrastive fine-tuning, and differentiable reward-guided policy optimization.The method pairs visually similar same-task trajectories and uses the trained predictor to guide policy and visual-encoder updates.Source: §1 · p. 2
• WorldGuide is competitive with WAM, sets a new SOTA among VLA baselines, and reaches 96.8% on LIBERO-100 while reducing inference latency by 2.2× to 19.9× with zero world-model deployment overhead.The predictor is discarded after training.Source: §1 · p. 2

• Generalist VLA work includes tokenized sequence modeling, generative action heads for continuous control, efficient sequence models, and policies scaled to humanoids.World-action models also couple action generation with explicit future-state prediction for long-horizon planning.Source: §2 · pp. 2–3
• World models support VLA learning as simulators, jointly learned state-and-action predictors, or JEPA-style latent objectives, with hallucinations or inference-time rollouts posing stated trade-offs.The related-work discussion says these paradigms are trained strictly on successful data without capturing failure boundaries.Source: §2 · p. 3
• WorldGuide uses failure-rich pretraining, matched success–failure contrastive learning, and boundary rewards backpropagated into action and visual modules.The overview depicts this as three sequential stages, with a frozen world model supplying the Stage 3 reward.Source: §2 · p. 3, Figure 2 · p. 3

3.1 PRELIMINARIES

• The latent world model predicts future semantic features from observations and actions, but expert-only training can map failures near successful latents, producing optimistically low distance penalties.This success bias motivates training on failure outcomes and separating outcome-relevant representations.Source: §3.1 · p. 4

3.2 STAGE 1: FAILURE-RICH PRETRAINING

• Stage 1 jointly trains the encoder and predictor on expert demonstrations and failures from simulator perturbations and intermediate policy checkpoints.Perturbations provide broad examples of coarse errors, while checkpoint failures supply subtle decision-point discrepancies near the success–failure boundary.Source: §3.2 · p. 4

3.3 STAGE 2: BOUNDARY-AWARE CONTRASTIVE FINE-TUNING

• Stage 2 pairs successful segments with visually similar failures from the same task near annotated failure onsets, then separates their latent representations.Segments without a valid matched failure use only the predictive loss, and post-failure retry loops are truncated to avoid dominating training.Source: §3.3 · pp. 4–5, Figure 3 · p. 5
• Stage 2 weights contrastive distances using temporal salience and channel masks to emphasize execution-critical frames and outcome-decisive features.A one-sided repulsive hinge loss pushes matched pairs beyond a margin.Source: §3.3 · p. 5

3.4 STAGE 3: END-TO-END OPTIMIZATION WITH DIFFERENTIABLE REWARD

• Stage 3 freezes the world model while reward gradients optimize the VLA policy end-to-end, then discards the model at deployment.The gradients reach the action head and vision-language model, with the action head favoring goal-directed chunks and the vision-language model aligning representations with safety-critical semantics.Source: §3.4 · pp. 5–6

4.1 SIMULATION SETUP AND BASELINES

• Evaluation covers LIBERO, RoboTwin, SimplerEnv, and real-world tasks, with each benchmark using its specified training stages and trajectories.Real-robot experiments use 300 expert and 60 failure trajectories, followed by 30K/10K Stage 1/2 steps and 40K fine-tuning steps.Source: §4 · p. 6, §4.1 · p. 6

4.2 SIMULATION EVALUATION

• 98.4% overall average on LIBERO; WorldGuide sets a new SOTA among standard zero-overhead VLAs and reaches 96.8% on LIBERO-100.It matches LingBot-VA at 98.5% while reaching 225 ms per step, a 2.2×–19.9× speedup.Source: §4.2 · p. 6
• 85.7% clean and 86.1% domain-randomized are WorldGuide's best average success rates on RoboTwin 2.0 across 12 tasks.It ranks among the top two on 8 of 12 tasks; Turn-Switch is the exception, where π0.5 performs best.Source: Table 2 · p. 7, §4.2 · p. 7
• +6.8% on Google Robot and +7.5% on WidowX Robot versus VLA-JEPA; WorldGuide's SimplerEnv results outperform its closest method.The paper attributes the gap to expert-only training's difficulty recognizing early failure signs under domain shifts.Source: §4.2 · p. 7

4.3 REAL-WORLD EVALUATION

• 18.3% pooled real-world success is a +57% relative gain; WorldGuide outperforms π0.5 on all three tasks and beats or ties π0.Improvements align with geometric sensitivity, but all policies frequently repeat unsuccessful motions after initial errors.Source: Table 3 · p. 7, §4.3 · p. 7

4.4 FURTHER ANALYSIS AND ABLATION STUDY

• Stage 2 raises episode-level AUC from 0.81 to 0.87 and critical-window AUC from 0.74 to 0.79 in episode-held-out probes.A near-failure-trained boundary probe reaches 0.87 AUC before errors and 0.74 AUC on held-out post-failure windows.Source: §4.4 · p. 8, Table 4 · p. 8
• Stage 1 failure rollouts improve policy metrics over success-only training, while Stage 2 adds gains, especially on RoboTwin and SimplerEnv.LIBERO gains are more modest because its baseline is near saturation.Source: Table 5 · p. 8, §4.4 · p. 9
• The boundary reward achieves 0.85 AUC and improves RoboTwin to 85.7/86.1% and LIBERO to 98.4%, unlike goal similarity or prediction MSE rewards.The paper reports those alternatives stay near chance AUC even in the failure-rich latent space.Source: §4.4 · p. 9

• Jointly learning from success and failure trajectories with boundary-aware contrastive learning yields failure-sensitive representations, and the world model is discarded at inference.Source: §5 · p. 9
• WorldGuide improves robustness and task success with no additional inference overhead.Source: §5 · p. 10
• The authors state that source code, evaluation protocols, and model checkpoints required to reproduce reported results will be publicly released upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 ARCHITECTURE AND BASE MODELS

• The world model uses a ViT visual encoder shared with the policy and a lightweight transformer predictor; LIBERO and RoboTwin use discrete OFT action tokens, while SimplerEnv uses a DiT-B flow-matching head.Source: §A.1 · p. 12
• Causal teacher forcing predicts next-frame latents in parallel over sliding contexts, eliminating autoregressive rollouts during training.Source: §A.1 · p. 12
• The world model is discarded at deployment, leaving only the VLA policy to execute.Source: Table 6 · p. 13

A.2 TRAINING CONFIGURATION

• All stages use AdamW with cosine learning-rate decay, gradient clipping at 1.0, bf16 mixed precision, and gradient checkpointing across 8× NVIDIA H20 GPUs.Source: §A.2 · p. 13
• Stage 2 uses a hinge margin of 0.7 and contrastive loss weight of 0.11, with temporal salience and channel gating enabled by default.Source: §A.2 · p. 13
• SIGReg regularizes predicted latents toward an isotropic standard Gaussian using 17 evaluation knots, 1024 random projection directions, and weight 0.09 across environments.Source: §A.2 · p. 13

B.1 SIMULATION BENCHMARKS

• LIBERO evaluation covers four suites; RoboTwin covers 12 tasks in clean and domain-randomized settings, while SimplerEnv tests real-to-sim visual generalization.Policies execute with action chunks of 8 for LIBERO, 50 for RoboTwin, and 16 for SimplerEnv; WorldGuide latency is 225 ms per chunk on the deployment path, with no world-model inference.Source: §B.1 · pp. 13–15

B.2 REAL-WORLD SETUP

• Real-robot tests use 300 demonstrations for three tight-tolerance pick-and-place tasks, with π0, π0.5, and WorldGuide evaluated in 20 trials per task under identical conditions.The experiments use an ARX LIFT2 dual-arm platform teleoperated with a PICO headset; only one arm is used because the tasks can be completed with a single arm.Source: §B.2 · p. 15

C.1 TWO COMPLEMENTARY SOURCES

• The failure set combines simulator perturbations that produce broad, salient errors with under-converged policy checkpoints that produce subtle failures at critical decision points.Perturbations include noisy control, randomized object placements, and scripted skill violations; checkpoint failures include missed grasps and premature releases.Source: §C.1 · p. 15

C.2 VLM KEYFRAME ANNOTATION OF tf

• Qwen3.5-VL-27B annotates each failure event’s onset from its video, and validated timestamps are converted to frame indices at 20 fps with random-sample manual review.Source: §C.2 · pp. 15–16

C.3 FAILURE-ONSET-BASED TRUNCATION

• Failure episodes are truncated after the annotated onset, retaining two additional action chunks to avoid prolonged post-failure retry loops.Source: §C.3 · p. 16

C.4 MATCHED PAIRING FOR STAGE 2

• Stage 2 matches each successful segment to the nearest feature-distance failure segment from the same task, filtered by proximity to annotated failure onset.Candidates must fall within the temporal tolerance and an action-chunk-sized window around the failure onset.Source: §C.4 · p. 16

Evaluation set

• The held-out set contains 109 timestamped failure episodes and 100 task-matched successful episodes from four LIBERO suites.Each probe input uses 8 context frames, 1 target frame, and 8 executed ∆qpos actions.Source: D Q1 PROBING PROTOCOL · p. 17

Window sampling

• Each failure episode contributes 16 windows: 8 sampled across task progress and 8 concentrated in the decision-critical region.Successful episodes use the corresponding progress grid; failure windows are also indexed by time relative to the annotated failure point.Source: D Q1 PROBING PROTOCOL · p. 17

Latent readout

• A 64-dimensional PCA and logistic-regression probe uses episode-level GroupKFold(5), preventing windows from the same episode crossing train and test.Predicted futures are normalized within each task before PCA; the protocol reports both window-level and episode-level mean scores.Source: D Q1 PROBING PROTOCOL · p. 17

Phase-wise and boundary-score analysis

• Stage 2 leaves the success-only progress control essentially unchanged at 0.87 → 0.86, while phase-wise analysis separates early, mid, near-failure, and post-failure windows.A separate boundary probe trains on the final 7.5 seconds before failure and near-goal success windows, then evaluates also on unseen post-failure windows.Source: D Q1 PROBING PROTOCOL · p. 17

E.1 SEPARABILITY OF FAILURE AND SUCCESS IN SINGLE IN-DISTRIBUTION TASKS

• 1.000 episode-level AUC and 0.993 window-level AUC separate 17 failing from 92 successful RoboTwin trajectories; phase-wise AUCs range from 0.982 to 0.997.A phase-matched boundary probe reaches 0.98 AUC before failure and 0.996 AUC on unseen post-boundary windows, with every failing episode above the success median.Source: §E.1 · pp. 17–18, Figure 9 · p. 18

E.2 CONTRASTIVE-NEGATIVE AND WEIGHTING VARIANTS

• Replacing progress-anchored counterpart selection with random same-task failures costs 7.4 points on RoboTwin and 2.9 points on LIBERO.With matched counterparts, plain ℓ2 yields 84.6/84.1; temporal salience contributes more than the channel gate, consistent with sparse outcome-decisive frames.Source: §E.2 · pp. 18–19, Table 8 · p. 18

E.3 HYPER-PARAMETER SENSITIVITY

• Doubling β to 0.2 drops performance by 3.3 points, while reducing λc below 0.06 yields 83.1; margins above 0.35 plateau at 85.4–85.7.The horizon sweep from W=12 to 100 declines by 0.6 points; on randomized placements, ∆=6 loses 2.2 points.Source: §E.3 · pp. 19–20, Figure 10 · p. 19

E.4 SPATIO-TEMPORAL FACTORIZATION IN dm

• Near failure, temporal salience peaks in the final ∼15 frames, while the channel gate retains ∼5.9% of 2048 channels above 0.6.Removing temporal salience yields 85.0/84.6 versus 85.3/85.0 without the channel gate; the spatio-temporal metric also serves as the Stage-3 reward.Source: §E.4 · p. 20, Figure 11 · p. 20

F MORE VISUALIZATIONS

• Across SimplerEnv and 12 Robotwin tasks, attention focuses on end-effectors, manipulated objects, contact points, and placement targets rather than background clutter.The visualizations include kitchen and study-room scenes, tool operations, object alignment, and bimanual manipulation.Source: F MORE VISUALIZATIONS · pp. 20–21, Figure 12 · p. 21, Figure 13 · p. 21

F.1 IMPLICATIONS FOR RECOVERY PERFORMANCE

• Stage 2 relies on finely annotated data and requires collecting failure trajectories using the model, making data construction time-consuming.The authors identify addressing this limitation as future work.Source: §F.1 · p. 22