PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Instance Slots: Semantically Rich World Models for Physical Interaction Planning

Juntao Cheng, Jingkai Wang, Yijun Shen, Xiansheng Chen, Zhiwei Yu

arXiv:2608.22294v1cs.RO

TL;DR

SR-WM addresses the limited task semantics of conventional world models and object slots by binding visual entities to five functional roles. It uses the resulting role state for semantic dynamics, candidate generation, selection, and suffix repair, with evaluation covering end-to-end performance, transfer, diagnostics, and failure boundaries.

  • Problem

    World models and instance slots do not explicitly specify which task roles entities play or whether actions establish and preserve required relations.

  • Method

    SR-WM binds soft visual entities to gripper, target, goal, relation, and phase roles, then uses one role state for action generation, semantic transition prediction, selection, and violation-aware suffix repair.

  • Results

    The evaluation separates representation, generation, selection, repair, transfer, efficiency, and physical execution across LIBERO suites and cross-suite settings.

  • Takeaways & Limitations

    SR-WM treats object-centric prediction as a semantic interface connecting visual dynamics with planning-oriented decision making.

  • Takeaways & Limitations

    The five-role ontology may be insufficient for bimanual interaction, deformable objects, tool use, or multiple simultaneous targets, and physical deployment requires careful supervision and safeguards.

Abstract

from arXiv · show

World models for physical interaction are typically trained to predict future observations or latent features; however, a planning-oriented model must answer a fundamentally different question: whether a candidate action produces a task-consistent future while preserving essential relations.Monolithic state representations obscure the underlying entities, while standard instance-level object slots merely identify \emph{what} is present without specifying \emph{what role} each entity plays in the task context. To bridge this gap, we present the Semantically Rich World Model (SR-WM), a task-conditioned world model structured around five functional roles: gripper, target, goal, relation, and phase.Within SR-WM, a visual entity encoder extracts soft entity hypotheses from pretrained patch features, allowing segmentation masks to serve as optional proposal priors without mandating them as required state representations or inference inputs.A role binder subsequently maps these hypotheses to task-specific roles, while an action-conditioned dynamics model predicts role transitions alongside fine-grained semantics, including grasp/contact, predicate establishment, relation preservation, fixture state, and phase change.Crucially, this unified role state grounds downstream multi-candidate action generation, stage-aware reranking, and violation-aware suffix resampling.Our comprehensive evaluation protocol spans all four LIBERO simulation suites, cross-suite transfer, perception diagnostics, and action-sensitivity analysis.Ultimately, this formulation transforms object-centric prediction into a semantic interface linking visual dynamics with planning-oriented decision making.

1 Introduction

SR-WM addresses the gap between plausible prediction and task-consistent planning by binding visual entities to functional roles and using shared semantic states for generation, prediction, selection, and repair.

  • World models must evaluate whether actions establish required states while preserving satisfied constraints, not merely predict plausible futures.
  • Exchangeable object slots identify entities but leave task roles, predicates, grasp status, and release timing to downstream policies.
  • SR-WM binds soft visual entities to five roles: gripper, target, goal, relation, and phase.Relation represents geometric and symbolic predicates, while phase tracks progress.
  • The shared role state conditions both multi-candidate action generation and semantic transition prediction for predicate establishment, preservation, and phase progress.
  • Stage-aware selection and violation-aware suffix repair support a receding-horizon planning loop that executes prefixes and repairs violating suffixes.

2 Related Work

Prior work improves world-model prediction, object-centric control, or action proposal, but does not explicitly represent the task predicates that actions must establish or preserve.

  • Visual world models support test-time planning through predicted pixels, features, or latents, while LeWM reports up to 48× faster planning than foundation-model world models.
  • Object-centric methods expose exchangeable entities or segmentation-derived slots for exploration, visuomotor control, and planning.
  • Candidate diversity or value estimation alone does not ensure required transitions, motivating role-conditioned generation and selectors trained on transition semantics and hard negatives.

3 Problem Formulation

The formulation constructs task-conditioned role states from multimodal observations, samples candidate action chunks, predicts semantic rollouts, and selects actions under receding-horizon control.

  • RGB history, proprioception, and language instruction define the role state used for prediction, generation, and selection.
  • The flow-matching head samples K horizon-H action chunks from the role state and instruction using independent Gaussian noise.
  • Action-conditioned dynamics predict role states and semantic transitions at J sub-horizon boundaries for each candidate.
  • A stage-aware selector chooses among candidates, after which the selected prefix executes under receding-horizon control.
  • Instruction-conditioned binding converts task-agnostic entity hypotheses into functional roles used for predictive dynamics and action planning.

4 Semantically Rich World Model

SR-WM separates exchangeable visual entities from task-conditioned role states, using five non-exchangeable roles to represent task semantics, geometry, embodiment, and progress. The shared role state conditions action generation and transition prediction, including semantic events and selective repair.

  • From Exchangeable Entities to Task-Conditioned Role States: SR-WM first extracts exchangeable entities, then transforms them into a task-conditioned predictive state rather than assigning fixed meanings to entity slots.The encoder uses pretrained patch tokens, multi-view learned queries, and soft entity hypotheses; role binding supplies task-specific meaning afterward.
  • Candidate Generation, Selection, and Repair: The role state directly conditions both the Flow-Matching action head and action-conditioned dynamics, allowing task structure to influence generated trajectories.All candidates share network weights and differ through independent noise, separating candidate diversity from candidate-specific policies.
  • LeWM Entity Tokens and Soft Visual Entities: Optional segmentation proposals act only as additive attention biases, while patch context and dropout support mask-free inference and robustness to missing or merged proposals.Setting λseg = 0 yields patch-only inference; proposal identities are not persistent object addresses.
  • Task-Conditioned Role Binding: Five non-exchangeable roles—gripper, target, goal, relation, and phase—encode embodiment, task entities, predicates, and temporal progress.Relation aggregates target–goal and gripper–target geometry, while phase aggregates role histories.
  • Task-Conditioned Role Binding: Instruction-conditioned role queries can bind the same visual entity as target, goal, or distractor depending on the task.Ground-truth role labels supervise binding during training, while inference uses images, proprioception, and language.
  • Semantic Transition Supervision: Semantic heads predict contact, grasp, predicate establishment and preservation, conjunction satisfaction, fixture state, and phase-transition validity across ordered action segments.Intermediate transition prediction localizes violation onset and supports suffix-repair localization; shuffled hard negatives test order, contact, and progress.
  • Candidate Generation, Selection, and Repair: The stage-aware selector combines progress, grasp, predicates, fixture validity, and uncertainty, while selective repair resamples only the suffix after the earliest predicted violation.The retained prefix is reevaluated under receding-horizon control, and full resampling is compared under a matched generation budget.

5 Experiments

Experiments evaluate SR-WM across end-to-end manipulation, semantic role fidelity, mask dependence, action sensitivity, ablations, transfer, and efficiency. Results show gains from explicit role structure, semantic selection and repair, mask-free inference, and cross-suite adaptation, while failures remain under occlusion and phase misalignment.

  • Action Prior and Coverage: SR-WM improves average Candidate 0 success from 45.5% to 62.8% and Oracle@4 coverage from 68.0% to 89.5%.Role conditioning improves both nominal generation and feasible-candidate coverage.
  • Closed-Loop Selection and Repair: Semantic selection reaches 75.3% top-1 success, while suffix repair raises success to 83.0%; pairwise ranking accuracy is 78.2%.Selection leaves a 14.2-point gap to Oracle@4 before repair.
  • Segmentation Prior Dependence: Patch-only inference retains 72.6% rollout success without inference-time masks, while corrupted masks reduce performance to 70.8% and oracle masks reach 78.4%.The evaluation compares patch-only, soft-prior, corrupted-mask, and oracle-mask variants.
  • Action-Conditioned Dynamics: The full model achieves a 0.157 shuffled-action gap and 78.2% ranking accuracy, whereas removing shuffled negatives reduces the gap to 0.018 and accuracy to 53.7%.Removing semantic heads lowers ranking accuracy to 60.8%; removing relation preservation or phase loss lowers it to 69.4% and 66.8%.
  • Semantic Role Structure: Closed-loop success rises from 55.0% with five instance slots to 62.0% with instructions, 68.0% with target–goal binding, 73.0% with relation roles, and 83.0% with the full architecture.The ablation progressively adds explicit semantic roles, including gripper and phase.
  • Cross-Task Generalization: Object-pretrained adaptation reaches 74.6% on Goal and 58.7% on LIBERO-10 versus 66.3% and 50.6% from scratch, while joint pretraining enables 78.4% on held-out tasks.Zero-shot transfer from the Object suite reaches 58.4% on novel spatial configurations, 52.7% on new semantic goals, and 27.9% on LIBERO-10.
  • Efficiency: The complete-system efficiency evaluation reports parameters, memory, training time, generation latency, scoring latency, and control rate on matched hardware.The published 15M LeWM architecture is treated as a reference point rather than a substitute for counting added SR-WM modules.
  • Qualitative Error Profiling: Remaining failures cluster around severe gripper occlusion during insertion and granular phase misalignment during complex sequential tasks.Occlusion can mislocalize the target center, while early release can cause dropped objects when fine-grained haptic feedback is unavailable.

6 Limitations and Scope

SR-WM’s scope is primarily single-arm, goal-directed manipulation, and several practical limitations constrain broader deployment and interpretation.

  • Scope: The five-role ontology may be insufficient for bimanual interaction, deformable objects, tool use, or multiple simultaneous targets.It is described as appropriate for many single-arm, goal-directed manipulation tasks.
  • Deployment: Predicate and phase supervision is inexpensive in simulation but requires careful labeling or estimation during physical interaction.
  • Deployment: Prefix-preserving suffix repair adds inference calls and requires collision checks and low-level safeguards for safety-critical execution.
  • Efficiency claims: The approximately 15M-parameter LeWM backbone does not establish a complete-system small-model or real-time claim because SR-WM adds role modules and an FM action head.Such claims depend on measured totals and matched-hardware results.
  • Evaluation: Oracle@K is an analysis upper bound, so high oracle coverage with weak top-1 selection indicates an unresolved selector problem rather than deployable performance.

7 Conclusion

SR-WM replaces exchangeable entity slots with a role-conditioned predictive interface that supports semantic dynamics, action generation, selection, and repair. Its evaluation is designed to distinguish improvements in representation, generation, and decision making.

  • 7 Conclusion: SR-WM converts visual entities into gripper, target, goal, relation, and phase roles using the compact LeWM encoder–predictor.
  • 7 Conclusion: The shared role state conditions both Flow-Matching action generation and action-conditioned semantic transition prediction for candidate selection and repair.
  • 7 Conclusion: The evaluation separates representation quality, action sensitivity, candidate coverage, selector quality, and physical execution.
  • 7 Conclusion: Role binding constructs a task-conditioned predictive state from entity hypotheses, language, and embodiment context rather than treating roles as post-hoc labels.

A.3 Experimental Claim Boundaries

The experimental design defines SR-WM’s information boundary, semantic supervision, transition targets, and training construction while distinguishing deployable inputs from training-only or oracle context.

  • Entity representation: Soft entity hypotheses encode appearance, geometry, motion, and confidence without assuming persistent identity through complete occlusion or re-entry.
  • Proposal dependence: Segmentation proposals act only as optional additive attention biases, with dropout and λseg = 0 supporting patch-only inference.
  • Role state: Role queries construct non-exchangeable gripper, target, goal, relation, and phase components, with relation representing interaction state and phase representing temporal progress.
  • Information boundary: During deployment, SR-WM uses RGB observations, proprioception, and language, while simulator state and task-derived labels supervise training targets rather than serving as rollout inputs.
  • Semantic dynamics: Intermediate action-conditioned states expose contact, predicate, preservation, fixture, and phase violations, providing boundaries for suffix repair.
  • Training and labels: Training uses event-based phase parsing, preservation-aware labels, hard negatives, and provenance-preserving pools that combine demonstrations, rollout deviations, and repair examples.

E.3 LIBERO Evaluation

The LIBERO evaluation controls candidate generation, decision rules, seeds, and adaptation settings to separate coverage, selection, transfer, and efficiency claims. It also makes explicit that generator comparisons are not selector-only comparisons.

  • Evaluation protocol: LIBERO-Object, LIBERO-Spatial, LIBERO-Goal, and LIBERO-10 are evaluated separately under matched rollout conditions.
  • Closed-loop inference: Closed-loop inference encodes the state, samples K candidates from independent Flow-Matching noise, predicts semantic transitions, selects the highest-scoring candidate, and repairs violating suffixes when needed.Failed repairs return to a safe stop or registered fallback.
  • Claim boundary: Generator comparisons cannot be interpreted as selector-only comparisons because candidate chunks necessarily differ across generators.
  • Transfer: Cross-suite transfer compares zero-shot evaluation, fixed-data adaptation, and training from scratch to assess suite-dependent target-data efficiency.
  • Metrics and controls: The main comparisons examine candidate outcomes, progressive predictive-state construction, mask dependence, action sensitivity, and matched-hardware efficiency.

F.1 Result Provenance

The supplement specifies provenance, qualitative annotation, and a staged validation protocol for simulation and physical evaluation. It also defines software safety gates that stop execution when state, timing, normalization, or actuation contracts fail.

  • Result provenance: Numerical supplement values reproduce main Tables 1 and 3–7 and link each cell to checkpoint, split, seed, decision, repair-budget, and aggregation metadata.The manifest records checkpoint hash, task split, initial states, environment seeds, candidate-noise seeds, decision rule, repair-call budget, and aggregation unit.
  • Result provenance: Quantitative rollout claims use K = 4, while registered K ∈{1, 2, 4, 8} sweeps retain candidate-budget and effective-coverage statistics.Dynamic resampling, rollout deviations, and repair windows define the complete training curriculum, whose joint contribution is evaluated end to end.
  • Qualitative analysis: Qualitative failures are labeled by earliest observable failure, with later consequences recorded as secondary tags.Severe perceptual occlusion and granular phase misalignment map to perception/binding and premature-release/phase categories.
  • Qualitative analysis: Qualitative examples include initial observations, instructions, binding attention, four candidate summaries, predicted events, selection, violations, and repaired suffixes.The set covers successes, selector and repair failures, and cases where Oracle@4 is zero.
  • Physical evaluation: Physical evaluation is separated from simulation and uses a sequence of offline replay, shadow execution, canary, and short closed-loop checks before full execution.Shadow execution requires at least 100 consecutive inference steps without sending commands.
  • Physical evaluation: The online provider reconstructs training context from synchronized camera frames, proprioception, end-effector pose, gripper state, and monotonic timestamps without ground-truth state or predicate labels.Metadata records action representation, control frequency, history size, horizon, normalization fingerprint, and checkpoint identity.
  • Physical evaluation: Recorded demonstrations are replayed chronologically to verify field ordering, timestamp and camera alignment, pose, gripper convention, and normalization before actuation.Every normalized dimension with |z| > 6 is inspected rather than automatically clipped.
  • Safety gates: A clipped action, stale timestamp, unsupported state, invalid normalization fingerprint, or failed actuation contract triggers a safe stop, and workspace bounds remain enforced.The emergency stop complements rather than replaces software safety constraints.

H Reproducibility Package

The reproducibility package is designed to make configurations, data splits, candidate comparisons, and generated results independently traceable. Tables should fail closed when required runs are missing, duplicated, or contaminated by test outcomes.

  • Package contents: The artifact package contains exact model, optimizer, sampler, evaluation, environment, dependency, predicate-parser, checkpoint, split, seed, and candidate-noise specifications.It also includes scripts covering Candidate 0, Random@K, fixed ID, selector, Oracle@K, full resampling, and suffix repair.
  • Package contents: Candidate comparison and evaluation scripts cover baselines, selection, Oracle@K, resampling, and suffix repair as distinct reproducible procedures.Candidate-noise manifests support paired selector comparisons.
  • Result generation: Generated tables fail closed when expected runs are missing, duplicated, or selected from test outcomes.Every cell should trace to a run manifest, checkpoint hash, task list, seed list, and aggregation script.
  • Supplementary tables: Table S10 organizes experimental controls by matched factors, jointly varied factors, and corresponding system-level reading.Table S11 records the minimum provenance needed to regenerate and interpret reported result families.

I Supplementary Limitations

The supplementary limitations bound SR-WM’s state interface, temporal association, perception, deployment reliability, and physical-evaluation scope. The design assumes simple role cardinality and does not eliminate occlusion, binding, selection, or inference-cost challenges.

  • State-interface scope: The five-component predictive-state interface assumes one dominant gripper, target, and goal, limiting direct scope for bimanual, multi-target, deformable-object, and tool-use tasks.Those settings may require a variable-cardinality state graph.
  • Perception and tracking: Temporal association is limited to finite observation history, and entity ordering is not persistent through occlusion or re-entry.Optional masks reduce dependence on a particular segmenter but do not eliminate occlusion or binding failures.
  • Planning and deployment: A high Oracle@K can remain non-deployable when selection is weak, while suffix repair adds inference cost.This separates candidate availability from reliable deployment behavior.
  • Evaluation scope: Table S12 standardizes qualitative failure annotation, while Table S13 defines physical deployment gates and requires robot-specific tolerances to be registered before inspection.Rollout counts and rates accompany aggregate use of qualitative categories.
Loading 2608.22294v1…