PaperlayerView exact revision on arXiv

Source-linked AI summary

Spotlighting Task-Relevant Features: Object-Centric Representations for Better Generalization in Robotic Manipulation

Alexandre Chapin, Bruno Machado, Emmanuel Dellandréa, Liming Chen

arXiv:2601.21416v2cs.RO

TL;DR

Robotic manipulation policies need visual representations that separate task-relevant structure from nuisance variation. This paper systematically evaluates slot-based object-centric representations against global and dense features, finding stronger robustness under distribution shifts and diagnosing capacity-related trade-offs.

  • Problem

    Global and dense visual features entangle task-relevant and irrelevant information, leaving it unclear whether structured slots improve control under realistic distribution shifts.

  • Method

    The paper systematically studies SOCR-based policies, using Slot Attention to extract object-like slots and evaluating them in a shared policy framework with standard and robotic-data pretraining.

  • Results

    SOCR-based policies consistently outperform traditional dense and global baselines, with benefits most pronounced under severe lighting and texture shifts.

  • Takeaways & Limitations

    Object-level abstraction is associated with robustness under distribution shift, while slot capacity creates a trade-off between in-domain performance and distractor robustness.

Abstract

from arXiv · show

The generalization capabilities of robotic manipulation policies are heavily influenced by the choice of visual representations. Existing approaches typically rely on representations extracted from pre-trained encoders, using two dominant types of features: global features, which summarize an entire image via a single pooled vector, and dense features, which preserve a patch-wise embedding from the final encoder layer. While widely used, both feature types mix task-relevant and irrelevant information, leading to poor generalization under distribution shifts, such as changes in lighting, textures, or the presence of distractors. In this work, we explore an intermediate structured alternative: Slot-Based Object-Centric Representations (SBOCR), which group dense features into a finite set of object-like entities. This representation permits to naturally reduce the noise provided to the robotic manipulation policy while keeping enough information to efficiently perform the task. We benchmark a range of global and dense representations against intermediate slot-based representations, across a suite of simulated and real-world manipulation tasks ranging from simple to complex. We evaluate their generalization under diverse visual conditions, including changes in lighting, texture, and the presence of distractors. Our findings reveal that SBOCR-based policies outperform dense and global representation-based policies in generalization settings, even without task-specific pretraining. These insights suggest that SBOCR is a promising direction for designing visual systems that generalize effectively in dynamic, real-world robotic environments.

I. INTRODUCTION

The introduction argues that global and dense visual features entangle task-relevant objects with irrelevant background, motivating slot-based object-centric representations as a structural bottleneck for robust visuomotor control.

  • Visuomotor policies map visual observations and goals to robot actions, making visual representation central to generalization across objects, environments, and tasks.
  • Global and dense features lack an explicit mechanism to separate task-relevant objects from background noise, encouraging overfitting to textures and lighting under distribution shifts.
  • The paper asks which structural properties visual representations must satisfy for policies to generalize under distribution shift.
  • SOCRs impose a fixed-slot, competitive-attention bottleneck that decomposes scenes into discrete object-level entities and may reduce reliance on spurious correlations.
  • The study systematically diagnoses SOCRs across simulated and real-world manipulation tasks, analyzing generalization, capacity-related failures, and mitigation guidance.
  • Large-scale pretraining can boost object-centric downstream performance, while slot merging and capacity limits are identified as key failure modes.

II. RELATED WORKS

Related work largely develops pretrained global or dense features, slot-based scene decompositions, and segmentation-driven alternatives, but robust control under realistic distribution shifts remains insufficiently tested.

  • Pretrained vision models for robot learning include large-scale, domain-specific, and self-supervised approaches, while dataset quality and emergent object segmentation can outweigh strict domain alignment.
  • SOCRs represent scenes as unsupervised structured slots and have expanded from generative models to pretrained backbones and temporal video modeling.
  • Prior work leaves unclear whether slot compositionality yields robust real-world control under distractors, novel object configurations, and altered backgrounds.
  • Segmentation-driven methods can be effective but often require annotated pretraining data, substantial compute, and explicit spatial prompts.

III. METHOD

The method extracts compact object-centric slots from dense visual features, integrates them into a common policy-learning framework, and compares standard and robotic-data pretraining.

  • Object-centric representation: Slot Attention converts dense backbone tokens into K slots through iterative cross-attention with competition, encouraging specialization across image regions.
  • Object-centric representation: Queries come from current slot representations, while keys and values come from dense feature tokens; iterative refinement produces final slots.
  • Object-centric representation: DINOSAUR* replaces DINO with DINOv2, reconstructs backbone features rather than pixels, and transfers slot information across timesteps with a Transformer.
  • Policy training: The policy framework combines an encoder, transformer observation trunk, and MLP policy head to predict actions from visual, proprioceptive, language, and temporal inputs.
  • Robotic pre-training: Robotic pretraining uses weighted real-world robotic video datasets, producing DINOSAUR* for COCO-based comparison and DINOSAUR-Rob* for robotic-data alignment.

IV. BENCHMARKS AND EXPERIMENTAL SETUP

The benchmark spans simulated and real-world manipulation settings, varied task complexity, and multiple pretrained visual encoders and representation baselines. Real-world evaluation uses four Franka-arm tabletop tasks, while pretraining-data mixtures are also examined.

  • Environments and tasks: The evaluation covers MetaWorld, LIBERO-90, and four real-world tabletop manipulation tasks.MetaWorld tests simple single-object generalization, LIBERO-90 emphasizes multi-object interactions, and the real-world benchmark uses a Franka arm.
  • Environments and tasks: LIBERO-90 evaluates multi-object reasoning and combinatorial generalization but reports only training-distribution results.The benchmark does not introduce distributional shifts for evaluation.
  • Environments and tasks: Each real-world task uses 50 teleoperated expert demonstrations, with success determined by task-specific final physical configurations.Success requires outcomes such as nested bowls, a closed drawer containing the screwdriver, cans inside the bin, or plates in rack slots.
  • Visual models and baselines: Seven pretrained encoders span ResNet and Vision Transformer architectures and supervised, self-supervised, contrastive, and distillation objectives.The comparison includes diverse visual-model families and training objectives.
  • Visual models and baselines: The study compares dense features with a segmentation-driven SAM+DINOv2 object-centric baseline and evaluates diverse robotic pretraining data.The pretraining-data evaluation reports DINOSAUR* global performance, while diverse robotic datasets are compared with individual datasets and COCO.

V. RESULTS

The results section frames the evaluation around whether SOCRs improve policy-learning efficiency, generalization under visual shifts, and understanding of failure conditions. Experiments use repeated rollouts across simulation and real-world settings.

  • Research questions: The evaluation asks whether SOCRs improve robot policy-learning efficiency over other visual representations.
  • Research questions: The evaluation asks whether SOCRs enhance policy generalization under visual distribution shifts.
  • Research questions: The study investigates when SOCRs fail and which factors affect their performance.
  • Evaluation protocol: Simulation results average success rates over three random seeds and 50 rollouts per task, whereas real-world evaluation uses 12 rollouts per task and generalization level.The combined real-world experiments comprise approximately 1,000 rollouts.

A. Q1: Do SOCRs improve manipulation policy learning ?

Across simulated and real-world benchmarks, object-centric policies perform competitively or best, including on complex multi-object tasks. Their results also reveal limitations for spatially impoverished segmentation features and benefits from object-centric structuration and robotic pretraining.

  • Overall performance: Object-centric policies, especially DINOSAUR-Rob*, achieve the highest overall performance across MetaWorld, LIBERO, and real-world benchmarks.They are on par with or surpass the best dense and global baselines across environments.
  • Representation failures: The SAM+DINOv2 representation prevents effective policy learning when it encodes object appearance without spatial information.The authors suggest that omitting bounding boxes or mask coordinates hampers modeling object relationships.
  • MetaWorld: Above 60% success is achieved by every MetaWorld model except the VC-1-based policy.Object-centric models perform comparably to the strongest baselines in this simple environment.
  • LIBERO: In LIBERO, SOCR-based policies perform on par with or better than dense-based Theia and DINOv2 policies in multi-object scenes.Global-based R3M and VC-1 models lag behind on fine-grained object interactions.
  • Real-world evaluation: 56% success is achieved by DINOSAUR-Rob* on real-world tasks, while DINOSAUR* reaches 48% and improves 20% over DINOv2.DINOSAUR* remains competitive even without robotic pretraining.
  • Overall conclusion: Object-centric models are effective and scalable across structured simulation tasks and noisy real-world environments.

B. Q2: Do SOCRs enhance generalization under visual distribution shifts?

SOCR-based policies show stronger robustness to visual distribution shifts than dense and global representations across simulated and real-world manipulation settings. Their advantage is especially pronounced under texture and lighting changes, while distractors remain a notable failure condition.

  • SOCR-based policies exhibit substantially better robustness to distribution shifts than dense and global representations across the evaluated environments.The evaluation covers MetaWorld, LIBERO, and real-world Franka manipulation, with shifts including distractors, novel textures, and lighting changes.
  • DINOSAUR-Rob* consistently achieves the highest overall performance and the smallest relative drop from in-domain to generalization settings.Figure 3 summarizes mean success rates and relative in-domain-to-generalization drops across environments.
  • SOCR-based models outperform all baselines by a large margin under MetaWorld texture and lighting shifts.In MetaWorld, DINOv2 and Theia outperform ResNet-based policies on average, but SOCR-based models excel specifically under texture and lighting changes.
  • In real-world evaluations, DINOSAUR-Rob* achieves a 41% success rate on average across shifts while other feature types experience substantial performance drops.Both SOCR-based models outperform other feature types across the two real-world generalization levels.
  • Distractor generalization remains weaker because clutter produces slot merging between task-relevant objects and other scene elements.Qualitative analysis shows that the gripper and target object, ideally isolated, become merged across distractor difficulty levels.

C. Q3: When do SOCRs fail, and what factors impacts their performance ?

SOCRs fail primarily when limited slot capacity forces distractors and task-relevant objects into shared representations. Increasing capacity improves out-of-distribution robustness but reduces in-domain performance, motivating adaptive and multi-granular designs.

  • Slot merging occurs when distractor features leak into task-relevant slots, polluting target-object or gripper state information.This failure is associated with the remaining performance drop under distractor shifts.
  • When objects outnumber slots, fixed-capacity SOCRs struggle to separate novel objects from task-relevant ones.The resulting bottleneck motivates testing whether increased slot capacity can mitigate overcrowding.
  • Impact of slot number: At K = 4, OOD success rate drops to 34% because forced slot merging makes the state representation ambiguous for the policy.Under-segmentation is especially problematic when the scene contains the robot, target, goal, and distractors.
  • Impact of slot number: At K = 20, OOD robustness reaches 0.60, while in-domain performance falls from 0.76 at K = 10 to approximately 0.61 at higher capacities.Additional slots isolate novel distractors but can split relevant objects into multiple parts in distractor-free scenes.
  • Impact of slot number: Larger slot budgets improve distractor isolation but leave the downstream policy with a noisier, higher-dimensional set of vectors to filter.The resulting in-domain degradation indicates that increased capacity is not a complete solution.
  • Future directions: Proposed future directions combine fine- and coarse-grained slots with adaptive capacity, selection mechanisms, and semantic alignment.These designs aim to separate visual noise from policy inputs while distinguishing active novel objects from passive background noise.

VI. CONCLUSION

The paper concludes that object-level abstraction, rather than backbone scale alone, improves robustness under distribution shift, especially severe lighting and texture changes. However, SOCR robustness depends on managing slot capacity and clutter-related trade-offs.

  • SOCR-based policies consistently outperform traditional dense and global baselines under distribution shift.The conclusion attributes this robustness to object-level abstraction rather than backbone scale alone.
  • Benefits are most pronounced under severe distribution shifts in lighting and texture.
  • SOCRs remain sensitive to high-clutter scenarios, showing that robustness is not automatic.
  • Slot capacity creates a fidelity–optimization trade-off: K < 10 merges distractors, whereas K > 10 preserves them but overloads the policy with noise.
  • Resolving stability and capacity trade-offs could support more reliable interaction with dynamic, real-world environments.
Loading 2601.21416v2…