PaperlayerView exact revision on arXiv

Source-linked AI summary

ObjDream: Guiding Vision-Language-Action Models with Object-Level Understanding and Future State Prediction

Huailiang Ma, Yuxuan Li, Ziyi Zhao, Xinkai Wang, Mingyu Li, Xinyu Guo, Lifeng Zhu, Aiguo Song, Cewu Lu, Jianhua Sun

arXiv:2610.20223v1cs.RO

TL;DR

VLA models often use image-level representations that leave object identity and state entangled with appearance and context, a gap relevant to robustness under distribution shifts. ObjDream adds factorized object representations and action-conditioned future-state supervision to VLA control. It reports results across LIBERO, LIBERO-Plus, RoboTwin, and real-world manipulation, including 98.6% average success on LIBERO.

  • Problem

    Many VLA models retain image-level representations that entangle object identity and spatial state with appearance and context, a concern for robustness to distribution shifts.

    Source: §1 · p. 2

  • Method

    ObjDream separates persistent identity from dynamic object state, routes VLM attention to object tokens, and uses action-conditioned future-state supervision during training.

    Source: §1 · p. 3, §3.2 · p. 5

  • Results

    ObjDream reports 98.6% average success on LIBERO and competitive results across LIBERO-Plus, RoboTwin 2.0, and real-world tasks.

    Source: §4.1 · p. 7, §4.3 · p. 9, §5 · p. 9

  • Takeaways & Limitations

    Real-world experiments across four tasks support manipulation performance and robustness to scene variations in physical settings.

    Source: §5 · p. 9

Abstract

from arXiv · show

Vision-language-action (VLA) models have emerged as a powerful paradigm for robotic manipulation, enabling robots to generate actions from visual observations and language instructions. However, most existing VLA models still operate on image-level visual representations inherited from pretrained vision-language backbones, while lacking fine-grained object understanding for manipulation. As a result, their robustness can degrade under out-of-distribution object layouts, clutter, and background variations. To address these limitations, we introduce ObjDream, an object-centric VLA framework centered on object-level understanding and future object state prediction. ObjDream uses a spatio-temporal object encoder to organize visual observations into temporal object slots, where persistent object identity and dynamic object state are represented through separate channels. These structured object representations are inserted into the token sequence of the vision-language model (VLM), where identity-guided routing modulates VLM attention to the corresponding object tokens. In addition, an action-conditioned object dynamics module predicts compact future object states relevant to manipulation during training, explicitly supervising object state evolution under robot actions without requiring future prediction at inference. Experiments on LIBERO, LIBERO-Plus, and RoboTwin 2.0 show that ObjDream improves generalization under diverse distribution shifts and scene perturbations. Real-world robot experiments further demonstrate that ObjDream transfers to physical manipulation settings with novel object arrangements and background changes.

1 Introduction

Most VLA models rely on image-level visual representations that entangle object identity and state with appearance and context, limiting robustness to distribution shifts. ObjDream addresses this gap with factorized object representations and action-conditioned future object-state supervision.

1 Introduction

  • Image-level VLA representations entangle task-relevant object identity and spatial state with appearance and context, which can weaken robustness to shifts in object appearance, layout, and background.Source: §1 · p. 2
  • Prior object-centric methods support region focus, relational reasoning, memory, or long-horizon manipulation, but often combine persistent identity and changing state in one slot or token.Separating identity from dynamic appearance has improved temporal consistency in video perception (Tran et al., 2026).Source: §1 · p. 2
  • Action-conditioned prediction of compact future object states, including geometry and visibility, supervises manipulation training without requiring future-state prediction at inference.The paper motivates compact object-state prediction as avoiding irrelevant texture and background modeling associated with full-image reconstruction.Source: §1 · pp. 2–3
  • ObjDream separates persistent object identity from dynamic state, inserts the resulting slots into the VLM token sequence, and uses identity-guided routing to direct attention to corresponding object tokens.Source: §1 · pp. 2–3
  • Evaluations on LIBERO, LIBERO-Plus, RoboTwin 2.0, and real-world robot experiments report competitive manipulation performance across scene perturbations.Source: §1 · pp. 2–3

2 Related Work

Related work improves VLA control through stronger models, predictive objectives, and object-centric representations, yet image-level visual structure remains common. ObjDream combines persistent identity, dynamic object states, and compact future-state supervision.

2 Related Work

  • VLA research explores stronger action decoders and fine-tuning, cross-embodiment training, preservation of pretrained priors, and predictive objectives, while image-level representations remain common.Examples include action-decoder and fine-tuning work (Zhao et al., 2023; Kim et al., 2025; Hou et al., 2025; NVIDIA et al., 2025; Qu et al., 2025), cross-embodiment training (Cai et al., 2026; Team et al., 2026; Wu et al., 2026a,b; Wang et al., 2026b), and predictive objectives (Zhao et al., 2025; Zhang et al., 2025; Cen et al., 2025b,a; Li et al., 2026b).Source: §2 · p. 3
  • Robot world models provide action-conditioned dynamics for control, but pixel- or video-space prediction entails high-dimensional visual details and can add training or inference overhead.ObjDream instead supervises compact future object states during training and directly generates actions at inference without explicit future-state decoding.Source: §2 · p. 3
  • Object-centric manipulation work spans scene decomposition, object grounding, compact tokenization, dynamics, relational representations, and predictive object features (Locatello et al., 2020; Kipf et al., 2021; Zhu et al., 2023; Vo et al., 2025; Li et al., 2025b; Wei et al., 2026; Bendikas et al., 2025; Ferraro et al., 2025; Mosbach et al., 2024; Hanyu et al., 2025; Yang et al., 2026; Yu et al., 2026b; Wang et al., 2026a; Yu et al., 2026a; Liu et al., 2026).Source: §2 · p. 3
  • ObjDream combines persistent identity references with dynamic object states and identity-guided VLM attention, building on the value of separating identity from changing visual content (Tran et al., 2026).Source: §2 · p. 4, Figure 2 · p. 4

3 Method

ObjDream combines a spatio-temporal object encoder, object dynamics, and an action expert. It inserts factorized object representations into the VLM and trains with object-state supervision alongside action imitation.

3.1 VLA Policy Formulation

  • The policy conditions continuous action chunks on language, visual observations, proprioception, and object embeddings, then executes generated chunks in a receding-horizon manner.The VLM consumes all policy views, while the object encoder uses a designated view and causal history.Source: §3.1 · pp. 4–5

3.2 Object-Centric Tokenization and VLM Routing

  • The encoder separates persistent identity from dynamic object state; object, relation, and prospective tokens enter the VLM, where cached identity references bias attention to valid object tokens.Prospective tokens combine relational context with individual object features and are optimized through the action objective.Source: §3.2 · pp. 5–6

3.3 Object-Level Forward Dynamics Module

  • During training, an action-conditioned auxiliary branch predicts future geometry, visibility, and mask logits for valid non-robot objects, then is skipped at inference.Future predictions cover multiple horizons in parallel; robot motion is represented by the action chunk.Source: §3.3 · p. 6

3.4 Training Objectives

  • Encoder pretraining combines segmentation, geometry, visibility, and region–text supervision; VLA training adds action imitation, current- and future-state, proposal, and slot-gate losses.The object encoder remains frozen during VLA training, and future-state loss excludes robot, invalid, and padded slots.Source: §3.4 · p. 6

4 Experiments

ObjDream is evaluated on standard and shifted benchmarks, component ablations, and real-robot tasks. Results show strong performance across LIBERO, RoboTwin 2.0, and real-world evaluations, with gains varying by setting and perturbation.

4 Experiments

  • Evaluation covers LIBERO suites, seven controlled LIBERO-Plus shifts, a fixed eight-task RoboTwin subset, and four real-world tasks.RoboTwin reports Easy and Hard settings; real-world methods share adaptation demonstrations and are evaluated ID and OOD without additional fine-tuning.Source: §4 · p. 7, §4.3 · p. 8

4.1 Main Results

  • 98.6% average success on LIBERO and second overall at 84.4% on zero-shot LIBERO-Plus show strong nominal performance and shifted-setting robustness.ObjDream performs best under language (90.6%), lighting (98.4%), and background (98.5%) perturbations; camera viewpoint and robot initialization remain the main challenges.Source: §4.1 · p. 7, Table 1 · p. 8
  • 12.87 and 15.62 percentage-point gains over StarVLA on RoboTwin 2.0 Easy and Hard show larger improvement under Hard conditions.ObjDream averages 60.25% on Easy and 22.75% on Hard; gains are particularly evident on grab roller, open microwave, and click bell under Hard conditions.Source: §4.1 · pp. 7–8, Figure 4 · p. 8

4.2 Ablation and Analysis

  • On LIBERO-Plus, removing factorized injection or identity-guided routing drops success from 84.4% to 82.5% or 82.6%, while removing ObjDyn yields 83.2%.The full model beats the backbone baseline by 2.0 points on LIBERO-Long and 3.2 points on LIBERO-Plus; factorized injection, identity-guided routing, and future-state supervision provide complementary gains.Source: §4.2 · p. 8, Table 2 · p. 9

4.3 Real-World Experiments

  • On real-robot tasks, ObjDream averages 74% ID and 55% OOD success, exceeding π0.5 by 8 points ID and ControlVLA by 14 and 24 points ID and OOD.It is comparable to π0.5 in aggregate OOD performance and beats the best baseline by 10 points on two-fruit placement and tabletop sweeping.Source: §4.3 · pp. 8–9, Table 3 · p. 9

5 Conclusion

ObjDream combines structured object representations with action-conditioned future-state supervision for manipulation. It separates persistent identity from changing state, routes VLM attention using identity, and supervises object evolution during training without future-state decoding at inference.

  • ObjDream combines structured object representations with action-conditioned future object-state supervision.Source: §5 · p. 9
  • Its object slots represent persistent identity cues separately from time-varying state features.Source: §5 · p. 9
  • Identity-guided routing directs VLM attention toward task-relevant object tokens.Source: §5 · p. 9
  • Action-conditioned supervision models object-state evolution during training without requiring future-state decoding at inference.Source: §5 · p. 9

A Offline Annotation Pipeline

ObjDream constructs object-level pretraining supervision automatically from demonstration instructions and observations. The pipeline grounds task-relevant phrases, propagates object masks across frames, and derives geometry, visibility, and identity references without manual frame-wise annotation.

B Future Object State Prediction Analysis

The future-state decoder improves on current-state persistence, particularly for displaced objects, and its predictions respond to action conditioning. The analysis evaluates future object states across LIBERO tasks and includes qualitative examples from RoboTwin 2.0 and real-world tasks.

Related papers

Loading 2610.20223v1…