Source-linked AI summary
Beyond Task Success: Stage-Wise Reliability of World Model Planning under Sensing Degradation
Geonmyeong Lee, Byoung-Tak Zhang
TL;DR
Final task success alone cannot reveal where sensing disturbances attenuate or persist in world model planning. The paper tracks 10 visual and temporal degradations through representation, prediction, planner preference, and outcome in paired evaluations. Their effects vary across stages, and temporal responses depend on information position, freshness, and planner exposure.
Problem
Final task outcomes reveal success or failure but not where sensing disturbances attenuate or how far they persist through a world model planner.
Source: §1 · p. 1Method
The authors pair clean and degraded conditions across 10 visual and temporal degradations and track changes from representation through prediction and planner preference to task outcome.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Stage-wise diagnosis can distinguish where sensing disturbances attenuate or persist and help narrow targets for model verification and sensing mitigation.
Source: §5 · p. 7The study uses controlled synthetic degradations and a limited set of world model planning settings, so broader models, tasks, and real sensing conditions remain to be tested.
Source: §5 · p. 7
Abstract
from arXiv · showhide
In world model planning, sensing inputs pass through an encoder and predictor before affecting planner decisions, so final task success alone cannot reveal where sensing disturbances attenuate or persist in the pipeline. We apply 10 visual and temporal sensing degradations to a world model planner and track their effects across representation, future prediction, planner preference, and physical outcome using paired evaluation on the same 50 tasks. The relative impact of degradations was not preserved across stages: large representation shifts could attenuate downstream, while smaller initial shifts could persist to the outcome, and internal-response ordering did not directly match physical-outcome ordering. Temporal degradations also showed distinct patterns: even with similar overall changes in observation history, responses differed substantially with the location of corrupted information and the planner's actual exposure. This non-uniform stage-wise response was also observed in secondary evaluations with another manipulation task and a different world model. Stage-wise diagnosis can therefore identify where sensing disturbances attenuate or persist and help prioritize subsequent model verification and sensing mitigation.
1 Introduction
The paper asks whether final task outcomes alone can assess sensing robustness in a multi-stage world model planner, and proposes paired, stage-wise evaluation across sensing degradations. It finds that degradation effects vary across stages, with temporal responses depending on where information is corrupted and what the planner encounters.
1 Introduction
- The study pairs clean and degraded conditions for 10 visual and temporal degradations, including random frame loss, history substitution, and fixed delay.It also examines stage-wise responses on another manipulation task and a separate world model.Source: §1 · p. 1
- The paired analysis tracks each disturbance from representation through future prediction and planner preference to task outcome.This makes downstream propagation comparable across sensing degradations.Source: §1 · p. 2
- Sensing degradations did not retain their relative impact across stages: some attenuated after representation, while others persisted, and internal metrics or outcomes alone were insufficient.Large planner responses did not necessarily correspond to lower success-rate point estimates.Source: §1 · p. 2
- Similar history-level changes produced different temporal responses depending on corrupted information’s position, freshness, and the planner’s actual exposure.The paper therefore cautions against relying only on aggregate history change or nominal corruption.Source: §1 · p. 2
2 Related Work
Prior work examines sensing robustness across vision, robot learning, VLA systems, and world models, alongside diagnostics for prediction and planning. This paper focuses on how sensing disturbances propagate through world-model planning’s downstream stages.
2 Related Work
- Prior studies assess robustness to visual perturbations in robot learning and VLA systems [7] [5], and examine missing observations and delay in temporal control.The paper focuses on downstream propagation, not individual degradation mechanisms or their mitigation.Source: §2 · p. 2
- World-model diagnostics include action-conditioned predictive consistency under visual perturbations [1] and planning-relevant quality across encoder, predictor, and planner [9].These works build on findings that prediction quality and downstream control need not align [10]; this paper applies the stage-wise perspective to sensing degradation.Source: §2 · p. 2
3 Sensing Degradation and Stage-Wise Evaluation
The evaluation applies synthetic visual and temporal shifts, measures their effects at four functional stages, and pairs clean and degraded conditions in primary and secondary settings.
3.1 Sensing degradation conditions
- Ten synthetic conditions cover appearance and visibility changes as well as missing, substituted, or delayed observations in recent history.Visual examples include intensity changes, motion blur, transient occlusion, and lens-like water-drop effects.Source: §3.1 · p. 3
- Temporal conditions include random frame loss, substitution at a specific history position, and fixed observation delay.Random frame loss drops each new frame with 40% probability; the evaluation also compares restoring only the latest observation with all-delay and clean histories.Source: §3.1 · p. 3
3.2 Stage-wise analysis
- Stage 1 normalizes latent representation shifts by a clean trajectory’s one-step change; temporal tests also measure the latest observation’s distance.This accounts for differing representation scales across models and separates whole-history change from current-observation change.Source: §3.2 · p. 3
- Stage 2 compares predicted future representations under the same action sequence, using D5/Dcontext to measure whether discrepancies attenuate or persist.The shared sequence is selected by the degraded planner; values near zero indicate attenuation, while larger values indicate greater persistence.Source: §3.2 · pp. 3–4
- Stage 3 evaluates clean-winner ranking over the same 300 candidate sequences in clean and degraded contexts, separating preference shifts from candidate-pool differences.Metrics include the degraded-context rank and how often the clean winner remains rank 1.Source: §3.2 · pp. 3–4
- Stage 4 independently runs clean and degraded planners, executes their selected actions, and compares paired task success rates and success-rate differences.The evaluation reports 95% Tango score confidence intervals for paired binary proportions.Source: §3.2 · pp. 3–4
3.3 Evaluation setup
- The primary evaluation uses DINO-WM on OGBench OGBScene Drawer at goal offset 20, pairing clean and degraded conditions across 50 replan-eligible tasks.The selected operating point avoids an overly easy single-shot setting and exposes replanning and temporal-degradation effects.Source: §3.3 · p. 4
- Secondary evaluation tests OGB-Cube with DINO-WM and LeWM, using low-light, motion blur, and Delay-10 on tasks solved under clean sensing.OGB-Cube relocates an independent cube to a specified 3-D target position, unlike the articulated drawer task.Source: §3.3 · p. 4
4 Results
Across the primary Scene evaluation, sensing degradations changed differently across pipeline stages, and internal responses did not reliably predict task outcomes. Temporal responses depended on corrupted-history position and planner exposure; stage-wise heterogeneity also appeared in a second task and model.
4.1 Sensing degradations show distinct propagation profiles
- 1.384 vs 2.224 Stage 1 shifts for low-light and overexposure fell to 0.054 and 0.083 at Stage 2, while Substitution-Current’s 0.792 persisted at 0.571.Despite similar Stage 2 values and median clean-winner ranks, overexposure and water-drop differed in same-top counts (34/50 vs. 23/50) and worst ranks (8 vs. 26).Source: §4.1 · pp. 4–5, Figure 2 · p. 5
- Stage 4 success fell from the clean baseline of 60.0% to 52.0% for low-light and 50.0% for overexposure, while Delay-5 and Delay-10 each reached 66.0% despite large internal shifts.All paired 95% confidence intervals for primary conditions included zero, so point estimates establish neither superiority nor improvement, nor no effect or equivalence.Source: §4.1 · p. 5
- 32/150 appearance cases versus 3/150 temporal cases had terminal-error increases of at least 40 mm; 35 of 36 clean-success-to-failure transitions fell in these large-error cases.The 40 mm threshold was a post-hoc diagnostic, not a new success criterion; large-error cases did not share one physical failure pattern.Source: §4.1 · p. 5
4.2 Temporal degradations show distinct propagation profiles
- 0.463 and 0.790 Stage 1 shifts for random frame loss and Substitution-Prior fell to 0.045 and 0.037 at Stage 2, unlike Substitution-Current’s similar 0.792 shift and 0.571 residual.Figure 3 compares Prior–Current and Delay-5–Delay-10 across stages and includes a same-state counterfactual restoring the latest observation.Source: §4.2 · p. 6, Figure 3 · p. 6
- 46/50 tasks had dropped frames enter planner history, but the latest observation was dropped in only 21; Substitution-Prior displaced the clean-best action from rank 1 in 22/50 tasks.Increasing delay from 5 to 10 steps increased changes at every stage.Source: §4.2 · p. 6
4.3 Stage-wise evidence across another task and model
- Stage-wise heterogeneity reappeared on OGB-Cube with DINO-WM and LeWM: appearance sensitivity differed by model, while Delay effects persisted into prediction and planner preference in both.Table 2 reports secondary Cube results for low-light, motion blur, and Delay-10; its Stage 1 values are normalized within each model.Source: Table 2 · p. 7, §4.3 · p. 7
4.2 Temporal degradations show distinct propagation profiles
- 7 of 10 same-state DEV pairs moved planner decisions toward clean after restoring the latest observation; median distance fell 16.63% for the first action and 22.86% for the full sequence.This counterfactual does not establish recovery of the subsequent robot trajectory or task success after fresh observations return.Source: §4.2 · p. 7
5 Conclusion
The conclusion emphasizes that sensing degradations show distinct attenuation and persistence across world-model-planning stages, with temporal responses depending on information position, freshness, and planner exposure. Stage-wise diagnosis can help target model verification and sensing mitigation, though the study’s controlled degradations and limited settings constrain how broadly the patterns are established.
5 Conclusion
- Distinct attenuation and persistence across representation, prediction, planner preference, and task outcome make stage-wise diagnosis useful for targeting model verification and sensing mitigation.Temporal responses also depend on information position, freshness, and the planner’s actual exposure.Source: §5 · p. 7
- The study uses controlled synthetic degradations and a limited set of world-model-planning settings, leaving broader models, tasks, and real sensing conditions for future testing.Source: §5 · p. 7