PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling What Changes: Sparse, Residual World Models for Object-Centric Manipulation

Param Thakkar, Parsika Paresh Shah, Manisha Sushant Gote

arXiv:2609.02046v1cs.ROcs.AI

TL;DR

Monolithic world models re-predict static scene elements and provide no explicit account of what changes. This paper evaluates an object-centric sparse/residual model that gates changed objects and applies residual pose updates, finding stronger efficiency, change detection, rollout, transfer, and preliminary planning results than a dense baseline.

  • Problem

    Monolithic predictors re-predict unchanged objects, spending capacity on static scene content, injecting errors, and lacking an explicit handle for which objects changed.

  • Method

    The model uses a shared per-object change gate and masked residual delta head, carrying ungated objects forward unchanged.

  • Results

    The sparse/residual model outperforms dense prediction on accuracy and parameter efficiency, preserves change-detection F1 across object counts, reduces rollout drift, and separates from dense in preliminary planning.

  • Takeaways & Limitations

    Modeling what changes is a useful object-centric bias for avoiding error injection and exposing an inspectable change mask, though delta regression remains near no-op quality.

  • Takeaways & Limitations

    The study is object-centric and structured-state rather than pixel-space, omits a dense-residual baseline, and evaluates planning narrowly with one planner, task, and three objects.

Abstract

from arXiv · show

Monolithic world models predict the entire next state at every step, spending capacity re-predicting the static majority of a scene and injecting error into it. We ask whether explicitly modeling change (a per-object change gate plus a residual delta head that perturbs only the objects the gate flags) is a more effective and interpretable bias for physical prediction and control. On a MuJoCo tabletop pushing benchmark scaling from 3 to 8 objects, the sparse/residual model predicts next-state poses 2.5 to 4.6 times more accurately than a dense multilayer perceptron at 8.6 to 11.1 times fewer parameters, sustains change-detection F1 of 0.80 to 0.87 where the dense baseline is degenerate, transfers across object counts with zero retraining (99.4 percent F1 retention), and reaches about 90 percent of its full-data accuracy with a quarter of the data. In autoregressive rollout it compounds far less error, hugging the no-motion floor while the dense model drifts. Finally, inside a sampling-based planner, prediction-only models fail (though a true-simulator oracle solves the task with the identical planner, confirming the planner is sound), but once featurized and trained for the states a planner visits, the sparse model begins to plan (0.23 plus or minus 0.06 success over three seeds) while the dense monolith stays at zero at every seed. Modeling what changes, rather than re-predicting the whole world, is a simple, effective bias for object-centric physical AI; code, data generators, and all checkpoints will be released upon publication.

I. INTRODUCTION

The paper proposes an object-centric world model that explicitly identifies changing objects and applies residual updates only to them, addressing capacity waste, error injection, and interpretability limits in dense prediction.

  • Core proposal: A per-object change gate selects moving objects, while a residual delta head updates only gated poses and copies all other objects unchanged.The design shares weights across objects and exposes an inspectable change mask.
  • Reported contributions: 2.5–4.6× higher overall accuracy is reported than dense prediction at 8.6–11.1× fewer parameters.The stated advantage is concentrated on suppressing errors for static objects, with changed-object error near no-op quality.
  • Reported contributions: 0.80–0.87 change-detection F1 is sustained where the dense baseline is degenerate.The explicit mask provides a direct handle on which objects the model expects to move.
  • Reported contributions: 99.4% F1 retention is achieved when transferring across object counts without retraining.The count-independent, shared per-object structure supports this transfer.
  • Reported contributions: 0.23 ± 0.06 planning success is obtained after featurization for planner states, versus 0.00 for dense at every seed.The sparse model is separated from dense, though the result is not significantly better than random.

II. RELATED WORK

The paper positions its approach at the intersection of dense world models, object-centric dynamics, sparse computation, and learned pushing models, combining their structures around explicit change prediction.

  • Dense world models and model-based control: Dense world models predict full states or latents monolithically, allowing prediction error to spread across the whole state.They support imagination, planning, and policy learning but do not explicitly identify inert scene components.
  • Object-centric and structured world models: Object-centric dynamics models share computation across entities and can generalize across entity counts, but typically still predict every object densely.The proposed model adds explicit per-object change decisions and residuals to this family.
  • Sparse coding and conditional computation: Conditional-computation methods provide input-dependent sparsity through learned gates, but not specifically over a world model’s change structure.The paper transfers sparsity from intermediate routing to the prediction output.
  • Pushing dynamics: Prior pushing models include analytic mechanics, real-world datasets, learned forward/inverse models, and soft-mask rigid-motion prediction.The paper instead uses a discrete change gate with an explicit sparsity objective.

III. METHOD

The method builds shared per-object features, predicts a hard change gate and residual pose increment, and trains these components with change supervision, conditional regression, and sparsity regularization.

  • Sparse/residual model: Each object receives features from its pose, velocity, goal and pusher positions, action, and a permutation-invariant aggregate of other objects.These features feed the gate and delta heads.
  • Sparse/residual model: A gate predicts whether an object changes, while a delta head predicts a 3D residual pose increment used to form the next pose.The residual is masked by the gate rather than applied to every object.
  • Sparse/residual model: A discrete Gumbel straight-through gate enables end-to-end training while enforcing an exactly zero update at inference when off.Shared heads and count-independent layers allow a model trained at one object count to run at another.
  • Training objective: The loss combines class-balanced gate BCE, residual L2 regression supervised on changed objects, and a penalty on mean gate activation.The components separately supervise change detection, changed-object motion, and sparsity.
  • Training objective: As sparsity weight increases, precision rises from 0.94 to 0.97 while recall falls from 0.81 to 0.71 and F1 degrades only gently.The prediction setting uses λs=0.2.

IV. EXPERIMENTAL SETUP

Experiments use procedurally generated MuJoCo tabletop pushing scenes with structured object states, hard-motion evaluation, dense and no-op baselines, and metrics spanning accuracy, detection, efficiency, and visual displacement.

  • Evaluation: An 80/10/10 split with configuration-leakage protection is evaluated on a hard subset containing real motion.Metrics include pose L2, change-detection precision/recall/F1, parameters, operations, and latency.
  • Baselines: The study compares a dense MLP against a no-op predictor that assumes no change.The dense baseline regresses all object poses at the next step under L2 loss.
  • Qualitative evaluation: Fig. 2 visualizes per-object displacement arrows from current to predicted next positions for ground truth, sparse, and dense predictions.The key comparison is whether unchanged objects remain stationary rather than receiving hallucinated motion.

V. PREDICTION ACCURACY

The sparse model’s headline accuracy advantage over dense prediction comes primarily from avoiding corruption of unchanged objects, not from better motion regression. This capacity-efficient gain does not translate into wall-clock speed.

  • 2.5–4.6× lower overall L2 than dense at every object count, using 8.6–11.1× fewer parameters.At N=5 and N=8, sparse overall L2 is at or below the no-op baseline.
  • The capacity-matched ladder shows that the sparse model’s win is not explained by parameter count alone.A dense MLP reduced to the sparse parameter budget still performs worse.
  • 0.80–0.87 change-detection F1 is sustained where dense detection is degenerate.Sparse’s advantage is attributed to preserving unchanged objects; delta regression remains roughly no-op quality.
  • The model’s efficiency claim concerns capacity rather than latency.Per-object heads reduce the operation-count ratio from 3.8× to 1.1×, while dense’s single matrix multiplication is faster.

VI. WHY IT WINS

Ablations identify the change gate as the largest single contributor to the sparse model’s advantage, while oracle gating shows that delta regression remains the bottleneck. Rollout behavior and diagnostics therefore separate detection from motion-prediction quality.

  • 3–5× lower overall L2 and much higher F1 persist at matched parameters, showing dense degeneracy is not a capacity artifact.These diagnostics use seed 0 on the test split.
  • 1.7–2.0× is the gate’s largest single ladder improvement, and it is the only rung that reaches below no-op.Featurization contributes 35–47% of the L2 gap, while residual contributes somewhat more than featurization.
  • Oracle gating barely changes changed-object L2, identifying delta regression rather than detection as the bottleneck.One-step contact-driven deltas, especially rotation, remain difficult.
  • At horizon 20, sparse rollout is 3.4–6.4× better than dense at every object count but remains worse than no-op overall.The mover-only curve is added to distinguish static-object preservation from motion accuracy.

VII. IT IS A WORLD MODEL, NOT JUST A REGRESSOR

The sparse model’s world-model properties are strongest in relative robustness: it drifts less than dense, transfers across object counts, and uses data efficiently. However, its rollout error remains above the no-op floor overall.

  • 3.4–6.4× better than dense at horizon 20, yet sparse remains worse than no-op overall across N=3, 5, and 8.The reported overall L2 values are 0.373 vs. 0.277, 0.293 vs. 0.180, and 0.182 vs. 0.108 for sparse versus no-op.
  • 99.4 percent F1 retention is achieved when transferring a count-invariant sparse model across object counts with zero retraining.Off-diagonal transfer F1 is 0.827 versus 0.832 on the diagonal; dense cannot run off-count.

VIII. DOWNSTREAM PLANNING

The planning study swaps only the forward model inside an otherwise fixed receding-horizon planner. Prediction-trained models initially fail, while planner-state-aware featurization and retraining improve sparse-model success but leave it statistically indistinguishable from random.

  • Planning setup: A fixed CEM planner uses each model as its forward simulator for a 15-step, receding-horizon push-to-goal task.The setup uses 256 samples, three refit iterations, and replanning every step; success means the target reaches within 5 cm of its goal.
  • Round 1: prediction-trained models: 1.00 oracle success and 0.95 scripted success show that the planner and task are solvable.The true-simulator oracle solves every episode in about 11 steps.
  • Round 1: prediction-trained models: 0.00 success for both learned models shows that prediction-trained sparse and dense models cannot plan in the initial setting.Both learned models perform worse than random because the planner queries out-of-distribution states.
  • Round 2: diagnosed-cause intervention: 0.23 ± 0.06 sparse success follows contact-aware, velocity-free features and diverse mixed-policy retraining, while dense remains at 0.00 at every seed.The sparse model also halves final distance, but its success is not significantly better than random at 0.15.
  • Round 2: diagnosed-cause intervention: The robust planning result is separation from dense, not a statistically significant improvement over random.The weakest sparse seed equals random at 0.15, whereas dense stays at zero across seeds.

IX. LIMITATIONS

The paper’s conclusions are bounded by structured-state experiments, metric-specific caveats, incomplete attribution, and preliminary planning evidence. These limitations constrain how broadly the results should be interpreted.

  • Scope: The results apply to object-centric structured state rather than pixel-space, so they do not transfer directly to perception-first settings.The paper explicitly scopes its claims to structured state, not pixels.
  • Metrics and baselines: Dense F1 degeneracy is partly definitional because ungated masks use the same pose-change threshold, so deadband and precision–recall analyses are needed.A nonzero regression can force recall to 1 under the thresholded-mask metric.
  • Metrics and baselines: A missing dense-residual non-object-centric baseline confounds attribution between residual prediction and object-centric structure.This prevents cleanly separating those two design contributions.
  • Evaluation scope: Single-seed results in Sec. VI and Figs. 3b,c limit the strength of those comparisons.The limitation is explicitly identified alongside the baseline-attribution caveat.
  • Planning scope: Planning evidence is preliminary and narrow: one CEM planner, one three-object task, and sparse success not significantly above random.Sparse achieves 0.23 ± 0.06 versus random 0.15, below scripted 0.95, while the robust result is separation from dense.

X. CONCLUSION AND FUTURE WORK

The study concludes that sparse change modeling is a capacity-efficient change detector that suppresses error on static objects, while delta regression remains the main bottleneck. Its scope is structured object states in small tabletop pushing tasks, with future work aimed at improving dynamics prediction, planning-state coverage, and representation flexibility.

  • Conclusion: Sparse matches no-op overall and on changed objects while retaining change-detection F1 of 0.80–0.87 where dense detection is degenerate.The authors characterize the result as change detection that avoids corruption, not accurate mover prediction.
  • Future work: The authors flag dense-residual modeling, distributional delta prediction, DAgger on planner states, multi-step training, and slot-based encoders as next steps.These directions address the reported delta-regression bottleneck, planner-state mismatch, and structured-state restriction.
  • Conclusion: Parameter efficiency is the durable efficiency advantage, whereas dense achieves lower wall-clock latency and the operation-count benefit erodes as object count grows.The operation-count ratio falls from 3.8× to 1.1× because sparse per-object heads scale with N.
  • Conclusion: The gate contributes the largest single improvement in the capacity-matched ladder, while oracle gating shows changed-object error is bottlenecked by delta regression.These analyses separate the contribution of object-change detection from residual prediction.

APPENDIX G PER-SEED PLANNING RESULTS

Per-seed planning results show that sparse planning remains separated from dense at every seed, although its average margin over random does not hold at the weakest seed.

  • Per-seed results: Dense succeeds in 0/20 episodes throughout, while the sparse advantage over dense holds at every training seed.The table reports 20 episodes per seed.
  • Per-seed results: Sparse exceeds random on average, but the margin does not hold at the weakest seed.The paper cautions that sparse planning is not significantly better than random overall.
Loading 2609.02046v1…