PaperlayerView exact revision on arXiv

Source-linked AI summary

Events, Not Pixels: Where World Models Get the Return Wrong

Haoyue Liu, Xiaoying Tang

arXiv:2610.29435v1cs.AI

TL;DR

World models are judged largely by how faithfully they render frames, although planners use their imagined returns to rank actions and visual fidelity may not track return accuracy. This paper uses transition substitutions with a budget-matched random control to locate return error, finding that event patches outperform random patches across reliable Atari pairs and that the remaining error is structured.

  • Problem

    Visual rendering quality is only loosely tied to what a planner needs, and prior analyses do not reveal whether return error is spread across a rollout or concentrated at a few transitions.

    Source: §1 · p. 1

  • Method

    A transition-level diagnostic replaces selected imagined transitions with real ones and compares return-error reductions against an equally sized random patch.

    Source: §1 · p. 1

  • Results

    Event patches beat random patches in 39 of 44 reliable Atari model–collection pairs, with a median +44-point lift; restoring events removes a median 70% of return error.

    Source: §1 · p. 2, §6.2 · p. 6

  • Takeaways & Limitations

    Event patches improve plan ranking, and the same concentration pattern appears on control tasks with rare reward events.

    Source: §8 · p. 9

Abstract

from arXiv · show

World models are increasingly used for planning, where a planner ranks candidate actions by returns imagined by the model. However, these models are judged mainly by how well they render frames, even though a visually faithful rollout can still assign the wrong return. To identify where this mismatch arises, we introduce a transition-level diagnostic that replaces imagined transitions with real ones and measures the reduction in return error against a budget-matched random control. Our analysis yields two findings. (1) Event transitions, the few steps where a reward is earned or an episode ends, are rendered as well as other steps (median pixel-error ratio 0.85), yet restoring them removes a median of 70% of the return error. In contrast, restoring transitions with the largest MSE, SSIM or LPIPS error removes less than a random choice. Across the 44 Atari-100k model--collection pairs that pass our reliability checks, including three released families, event patches outperform random patches in 39 (sign test p=1.4 x 10^{-7}), with a median gain of +44 percentage points. (2) The remaining error is structured: a complementary patch separates errors at true events from those elsewhere, while an eight-step timing tolerance recovers few events missed by our deterministic models. This structure also guides repair: a rule recorded before four repair outcomes were known predicts when a diffusion model repairs these failures. Event patches improve plan ranking, and the same concentration appears on control tasks with rare reward events.

1. Introduction

World models let planners rank imagined action sequences by return, but visual fidelity does not necessarily track the decision-relevant error. This paper localizes that error by replacing selected imagined transitions with real ones and finds it concentrates at reward and termination events.

1. Introduction

  • World-model progress is commonly measured by frame-rendering fidelity, yet a rollout can render well and still misprice a return after missing a reward or termination event.Prior analyses found visual-quality rankings can disagree with task success, but did not locate return error within a rollout.Source: §1 · p. 1
  • A transition-level substitution diagnostic compares event patches with equally sized random patches while holding the action fixed.Real and imagined rollouts share context and recorded actions; complementary and all-real patches provide additional controls.Source: §1 · p. 1, Figure 1 · p. 2
  • Across 44 reliable Atari model–collection pairs, event patches beat random patches in 39, with a median +44-point gain.Restoring event transitions removes a median 70% of return error, although their median pixel-error ratio is 0.85.Source: §1 · pp. 1–2
  • A complementary patch separates errors at true events from errors elsewhere, while timing tolerance explains little of the events deterministic models miss.The paper also reports that event patches improve plan ranking and that the concentration pattern carries over to control tasks with rare reward events.Source: §1 · p. 2

2. Related Work

Prior work documents gaps between visual fidelity and task success and uses oracle interventions or perturbation-based attribution to analyze model and agent failures. This paper distinguishes its contribution as locating decision-relevant error within released world-model rollouts.

2. Related Work

  • This work locates decision-relevant error inside rollouts, complementing studies that weight error by state dimensions and DIAMOND’s finding that visual details matter.The figure compares event and random patches across budgets, with matched-coverage lift shown separately.Source: §2 · p. 2, Figure 2 · p. 3
  • Prior oracle analyses substitute true states or intervene on whole models, while counterfactual debugging assigns Shapley credit across time steps under injected faults.The paper instead replaces individual transition pairs in released world-model rollouts under a budget-matched random control.Source: §2 · p. 3

3. Diagnostic Setup

The diagnostic estimates return from transition pairs, then measures how selected real-information substitutions reduce return error. Its event mask targets reward or termination transitions and compares them with a same-budget random mask.

3. Diagnostic Setup

  • The diagnostic uses rendered frames and applies to released checkpoints without access to model internals.A frozen, memoryless detector shared across a game's models reads each transition pair and action for reward sign and termination.Source: §3 · p. 3
  • Imagined and emulator returns use the same recorded actions and stop at the first termination; Atari uses a 60-step horizon and γ = 0.99.Rewards enter through their sign, putting all games on one scale.Source: §3 · p. 3
  • Observation patches replace both frames of selected transitions before detection, while reward or termination interventions replace the corresponding predicted values.The reported score is the fractional reduction in mean absolute return error; the all-frame reference gives the detector only real transition pairs.Source: §3 · p. 3
  • The event mask selects real reward or termination transitions and their one-step neighbors; the random mask selects the same number of steps per rollout.The complementary mask measures errors away from selected events and, contrasted with the event patch, classifies each model's failure pattern.Source: §3 · p. 4

4. Localising the Return Error

Across patch budgets, event targeting generally outperforms a same-budget random control, and the random comparison is needed to isolate targeting from coverage. Widening the event neighborhood also reveals timing effects.

4. Localising the Return Error

  • In 8 of 9 sweeps, the event curve stays well above the random curve as the event neighborhood widens across three games and three released families.For DIAMOND on Freeway, patching 2.8 of sixty steps removes 69% of error versus −4% for random patches and 86% when every frame is real.Source: §4 · p. 4
  • Without dilation, DIAMOND's Freeway event patch is negative, consistent with the reward digit appearing after the transition that earns it.The event neighborhood is therefore treated as part of the measurement.Source: §4 · p. 4
  • On DIAMOND and Simulus Seaquest, both patches remove more error as coverage grows while lift stays flat, so event-patch reduction alone overstates targeting.Table 1 reports each game at a one-step neighborhood, while Figure 3 shows events beyond Atari.Source: §4 · p. 4

5. Missing Events and a Model-Class Repair

The section asks why imagined events are missed and whether another model class can draw them, extending localization from where return error lies to possible causes and repair. It uses timing-tolerant recall and a pre-specified rule to assess missed events and diffusion-model repair.

5. Missing Events and a Model-Class Repair

  • Localization identifies where return error lies, but the section turns to why imagined events are missed and whether another model class draws them.Source: §5 · p. 4

5.1. Absent or Mistimed Events

  • A missed event may occur at a neighboring imagined step or may not be drawn in a form the detector reads; timing tolerance distinguishes these explanations by their recall curves.Mistiming predicts recall rising faster with alignment tolerance than a displaced-prediction null; unreadable events predict low recall at every tolerance.Source: §5.1 · p. 4
  • Table 1 reports the percent of return error removed by event, complementary, and budget-matched random patches, averaged over three seeds.Rows are sorted by event coverage; grey pairs and games whose detector fails the gate are omitted.Source: Table 1 · p. 5

5.2. A Rule Recorded Before the Outcomes

  • As a repair, the authors replace the deterministic MSE predictor with a diffusion model trained on the same data, sampling each next observation instead of regressing it.Because sampled futures may differ from realized ones, they evaluate event recall, return error, and plan ranking.Source: §5.2 · p. 5
  • Before observing the remaining outcomes, the authors pre-specify that baseline event recall below 0.50 predicts return-error reduction above the game's floor, while recall at or above 0.50 predicts none.The prospective score covers Pong and three unrun sampling arms; Freeway had already been scored.Source: §5.2 · p. 5

6. Experiments

The experiments ask whether return error concentrates at event transitions, whether missed events are absent or mistimed, whether a diffusion model repairs them, and whether the same transitions drive plan choice. The study uses 400 Atari probes and detector reliability checks, with additional control-task and environment evaluations.

6.2. A1: Return Error Concentrates at Events

  • 39 of 44 reliable pairs show positive event-patch lift (sign test p = 1.4×10−7), with a median lift of +44 points.Each released family and every counted game except MsPacman has a positive median lift.Source: §6.2 · p. 6
  • Event frames have a median pixel-error ratio of 0.85, yet the largest MSE, SSIM, or LPIPS errors identify transitions whose patches remove less than random patches.Pixel metrics therefore do not locate the return error.Source: Table 2 · p. 5, §6.2 · pp. 6, 8
  • At q above 12%, median lift remains +40, while all Krull pairs stay positive at q = 58.5%.Below 6%, median lift is +59.Source: §6.2 · p. 8
  • The complementary patch classifies 33 pairs as event-dominated, 5 as non-event-dominated, and 6 as unclassified; DIAMOND on Boxing is event-dominated under its own reward model.Under the detector, that pair is non-event-dominated: event patches remove 12% and the complementary patch 72%.Source: §6.2 · p. 8

6.3. A2: Missed Events Persist Under Tolerance

  • At τ = 8, at least 76% of the deterministic model’s recall deficit remains across five games, indicating that most missed events are absent rather than merely mistimed.On Freeway, MSE recall rises from 0.091 at τ = 0 to 0.153 at τ = 8, against 0.028 for the null.Source: Figure 5 · p. 6, §6.3 · p. 8

6.4. A3: A Diffusion Model Repairs as Predicted

  • The pre-specified repair rule scores 4/4, including 3/3 on games whose detector passes the attribution gate.All four comparisons clear their declared floors.Source: §6.4 · p. 8, Table 3 · p. 8
  • The repair comes from the diffusion network rather than sampling: its deterministic squared-error version has lower error than the MSE baseline on all five games.Sampling recalls more events on Pong (0.80 against 0.69).Source: §6.4 · p. 8

6.5. A4: Event Transitions Drive Plan Selection

  • Event patches improve pairwise plan accuracy in all 7 reliable pairs, with median top-1 regret falling from 0.565 to 0.141.In 15 additional games, all 8 further pairs improve; the median gain is +0.156, compared with +0.012 for random patches.Source: Figure 7 · p. 7, §6.5 · p. 8

7. Cross-Task Transfer and Discussion

The diagnostic is also applied beyond Atari, including control tasks and matched-action rollouts in MiniGrid and Crafter.

7. Cross-Task Transfer and Discussion

  • The three control-task comparisons with reliable detectors yield +54/+54 points of lift on sparse cartpole across two architectures and +8 on ball-in-cup.The MiniGrid and Crafter rollouts show scene transitions under matched recorded actions.Source: §7 · p. 8

8. Conclusion

A transition-level substitution diagnostic locates where world models’ imagined returns go wrong. Across Atari pairs, event transitions concentrate return error despite looking no worse in pixel metrics; event patches also improve plan ranking, and the pattern appears on control tasks with rare reward events.

8. Conclusion

  • In 39 of 44 reliably measured Atari-100k model–collection pairs, return error concentrates at reward and termination transitions that pixel metrics do not single out.A rule recorded in advance predicts where a diffusion model class helps, and event patches improve plan ranking in every counted pair.Source: §8 · p. 9

Related papers

Loading 2610.29435v1…