PaperlayerView exact revision on arXiv

Source-linked AI summary

STORM: Slot-based Task-aware Object-centric Representation for robotic Manipulation

Alexandre Chapin, Emmanuel Dellandréa, Liming Chen

arXiv:2601.20381v3cs.RO

TL;DR

Dense visual foundation model features provide strong perception but lack explicit object-level structure needed for robust and controllable robotic manipulation. STORM adds lightweight task-aware slots to frozen features, trains them semantically before cautious policy adaptation, and reports improved control and generalization across visual shifts and deployment settings.

  • Problem

    Dense VFM features lack explicit object-level structure, requiring manipulation policies to implicitly attend to relevant entities and limiting robustness under visual variations.

  • Method

    STORM places a lightweight slot-based module on frozen VFM features, using visual-semantic pretraining followed by joint task adaptation with a manipulation policy.

  • Results

    STORM improves control performance and generalization to visual distractors compared with frozen VFM features, full fine-tuning, and end-to-end object-centric training in simulated and real-world settings.

  • Takeaways & Limitations

    Small, semantically grounded adapters can efficiently refine generic VFM features while injecting structural and semantic bias into policy learning.

Abstract

from arXiv · show

Visual foundation models provide strong perceptual features for robotics, but their dense representations lack explicit object-level structure, limiting robustness and controllability in manipulation tasks. We propose STORM (Slot-based Task-aware Object-centric Representation for robotic Manipulation), a lightweight object-centric adaptation module that augments frozen visual foundation models with a small set of task-aware slots for robotic manipulation. Rather than fully tuning large backbones on the task, STORM employs an efficient two-stage training strategy: few layers of object-centric representation are first trained on top of the frozen backbone through visual--semantic pretraining using language embeddings, then jointly adapted with a downstream manipulation policy for task alignement. This staged learning prevents degenerate slot formation and preserves semantic consistency while aligning perception with task objectives. Experiments on object discovery benchmarks and robotic manipulation tasks show that STORM improves control performance and generalization to visual shifts (distractors, textures, lighting) compared to directly using frozen or fine-tuned foundation model features, or existing object-centric representations. STORM serves not only as an efficient mechanism for refining generic foundation model features, but also as a novel way of injecting beneficial structural and semantic bias into policy learning.

I. INTRODUCTION

STORM adds compact, semantically grounded slots on frozen visual foundation model features to provide interpretable object-level representations for robotic manipulation. Its two-phase adaptation strategy supports reliable control, with validation showing improved generalization and control reliability over several baselines.

  • STORM addresses the lack of explicit object-level structure in dense VFM features, which can make downstream policies brittle under visual variations.Without object-level structure, policies must implicitly learn which entities matter.
  • Slot-based representations provide discrete object or part latents that support modular reasoning, interpretability, and compositional generalization, but prior approaches lack semantic control.Existing language-conditioned methods had not been adapted to robotic control scenarios in the cited discussion.
  • STORM is a lightweight slot-based module that converts frozen VFM features into semantically grounded, interpretable representations for control policies.The architecture uses a small number of slot-based layers and a mask entropy loss to avoid degenerate solutions.
  • The two-stage strategy first stabilizes slot formation through visual-semantic pretraining, then jointly adapts representations with a control policy.This process makes the representations task-aware and prevents degenerate slot formation during policy training.
  • Experiments in simulated and real-world deployments show STORM outperforming frozen VFM features, full fine-tuning, and end-to-end object-centric training.The reported improvements include generalization to visual distractors and overall control reliability.

A. Pre-trained visual foundation models

Visual foundation models learn robust, spatially structured features from large-scale self-supervised data, while vision-language models provide semantic supervision. Robotic control nevertheless faces efficiency and task-relevance challenges when using dense features, motivating compact task-aware slots.

  • Self-supervised methods trained on internet-scale unlabeled data produce robust visual foundation models with rich spatially structured features.Common training objectives include masked image modeling, contrastive learning, and hybrids of both.
  • Vision-language models such as CLIP align images and text, enabling open-vocabulary and semantic supervision for visual systems.
  • Directly using dense, high-dimensional VFM features for control can be computationally expensive and may retain task-irrelevant information.STORM instead produces compact, discrete slot-based latents that are task-aware and usable by downstream policies.
  • Object-centric representation learning decomposes scenes into modular latent slots, but stable, task-relevant assignments remain challenging and prior methods predominantly target static tasks.

III. METHOD

STORM adapts prior object-centric learning to robotic control with lightweight semantic conditioning, entropy regularization, and explicit spatial grounding. Its decoupled training strategy stabilizes semantic slot discovery before policy alignment.

  • STORM extends object-centric learning for dynamic robotic control through a decoupled two-phase adaptation strategy.The strategy stabilizes semantic slot discovery before aligning slots with policy gradients, preventing representation collapse.
  • The method uses a lightweight CLIP-text encoder instead of large language models for slot initialization.
  • An entropy-based slot usage penalty mitigates slot collapse in the simplified language-conditioning setup.
  • The policy combines task embeddings with slot representations augmented by spatial information such as mask-derived centers of mass.A dynamic alignment module supports stable slot formation across timesteps.

A. Two-phase learning for object-centric adaptation

STORM separates semantic slot formation from downstream policy adaptation because direct end-to-end training can produce unstable or degenerate assignments. Pretraining followed by cautious adaptation stabilizes object bindings while preserving task-specific flexibility.

  • Direct end-to-end training of object-centric layers and a control policy frequently produces unstable or degenerate slot assignments.
  • Stage 1: Visual-Semantic Slot Pretraining: Stage 1 trains the object-centric module independently on frozen DINOv2 feature maps using visual-semantic alignment with CLIP text descriptions.The module produces slots corresponding to candidate objects and includes regularization for spatial localization.
  • Stage 2: Joint Slot-Policy Training: Stage 2 integrates the pretrained module with a downstream policy through imitation learning and fine-tunes the object-centric layers with a small learning rate.Policy gradients are detached at the slot level rather than backpropagated into the object-centric module.

B. Visual-semantic slot pretraining

STORM pretrains a lightweight object-centric module on frozen visual features using textual conditioning, reconstruction, semantic contrastive learning, and entropy regularization. This stage aims to produce semantically grounded slots while preventing slot collapse.

  • B. Visual-semantic slot pretraining: The first phase decomposes input images into robust semantic entities through object-centric decomposition and language conditioning.The decomposition branch processes frozen DINOv2 features with Slot-Attention, while the language branch supplies textual prompts.
  • B. Visual-semantic slot pretraining: Object slots cross-attend to DINOv2 patch features, and a shared MLP decoder reconstructs the feature maps from the slots.The reconstruction objective indirectly guides slot attention toward objects through the structure of the visual foundation-model feature space.
  • B. Visual-semantic slot pretraining: STORM places a compact slot-based module on frozen DINOv2 features and removes the large-language-model requirement by using a lightweight CLIP-text encoder.The CLIP-text encoder has approximately 300M parameters, described as an approximately 23× reduction relative to the prior LLM2Vec-plus-CLIP setup.
  • B. Visual-semantic slot pretraining: Text-initialized slots pool localized CLIP visual features, which are trained contrastively against CLIP text embeddings for semantic alignment.A PatchCLIP branch is used because DINOv2 does not inherently align visual features with language; prompts come from labeled text-image pairs during first-stage training.
  • B. Visual-semantic slot pretraining: An entropy-based slot usage penalty aggregates and normalizes soft assignment masks to discourage collapse onto a single slot or uniform attention.Minimizing this penalty encourages balanced usage across slots and acts as a critical regularizer for the lightweight formulation.
  • B. Visual-semantic slot pretraining: The visual-semantic module is pretrained by minimizing a combined reconstruction, semantic, and penalty objective.The stated overall loss weights the semantic and penalty terms together by 0.1.

C. Joint Slot-Policy Training

The second phase integrates pretrained semantic slots into an imitation-learning policy. The policy combines slot representations with task instructions and proprioception, while visual components are refined separately during joint training.

  • C. Joint Slot-Policy Training: Expert demonstrations provide trajectories from which the policy predicts actions using camera observations, task instructions, and robot proprioception.Each observation yields a fixed set of slot representations, supplemented by mask-encoder information for spatial grounding.
  • C. Joint Slot-Policy Training: Task instructions are encoded with frozen CLIP text features, while parsed instruction nouns condition the object-centric module.The noun list is obtained automatically with spaCy, and “robot arm” is appended explicitly.
  • C. Joint Slot-Policy Training: Feature-level gradient detachment prevents imitation-learning loss from backpropagating into the visual backbone.The policy follows a standard imitation-learning objective, while object-centric components are refined independently with their visual losses at a reduced learning rate.

IV. EXPERIMENTS

The experiments first validate visual-semantic learning on object discovery and then evaluate STORM’s two-phase task-aware learning for robotic manipulation, including component ablations.

  • IV. EXPERIMENTS: Experiments compare STORM with existing object-centric models on object discovery, evaluate its robotic manipulation policy, and ablate framework components.The evaluation is designed to assess both the visual-semantic module and the task-aware two-phase learning scheme.

A. Object decomposition and grounding

The visual-semantic module is evaluated for object discovery on complex multi-object datasets using segmentation and overlap metrics. STORM achieves strong segmentation quality while retaining a lightweight decoder.

  • A. Object decomposition and grounding: Pretraining uses VG-COCO for 300k steps with DINOv2-B/14, seven slots, 256-dimensional slots, and a batch size of 64.Optimization uses AdamW with learning rate 4×10^-4, cosine decay, and 10,000 warmup steps on one V100 GPU.
  • A. Object decomposition and grounding: Performance is measured with FG-ARI for segmentation accuracy and object consistency, alongside mBO for pixel-wise mask overlap.Table I further distinguishes instance-level mBOi from class-level mBOc.
  • A. Object decomposition and grounding: Object discovery is benchmarked on PASCAL VOC 2012 and COCO, which contain complex multi-object scenes.This evaluation primarily validates whether the lightweight architecture separates distinct entities rather than targeting state-of-the-art 2D segmentation.
  • A. Object decomposition and grounding: Masks are extracted from Slot-Attention maps and resized to the original image dimensions before comparison with ground-truth masks.This provides the mask representation used for object-discovery evaluation.
  • A. Object decomposition and grounding: STORM surpasses all unsupervised models on FG-ARI and performs highly competitively with CTRL-O.It slightly trails some diffusion-based methods on mBO but exceeds them in segmentation quality.
  • A. Object decomposition and grounding: The lightweight original DINOSAUR MLP decoder produces less precise masks than heavier transformer or diffusion decoders, while the learned slots remain semantically accurate and structurally sound.The authors identify heavier decoders as a possible way to improve mask sharpness.

B. Robotic manipulation

STORM is evaluated across MetaWorld, LIBERO, and real-world manipulation settings, comparing ID performance with OOD robustness under visual shifts. It consistently outperforms frozen and fine-tuned DINOv2, DINOSAUR, and SAM+DINOv2 baselines, with especially large gains in LIBERO and real-world OOD conditions.

  • Training procedure: Robotic training jointly fine-tunes the object-centric layers with the policy while keeping the DINOv2 backbone frozen, using shared policy architecture and hyperparameters across models.Joint training runs for 150k steps on demonstration trajectories, with the policy optimized using action-prediction MSE loss.
  • Evaluation setup: The evaluation covers MetaWorld, LIBERO, and real-world robotic scenarios, measuring success rates in ID and OOD conditions.OOD conditions include novel distractors, lighting, and textures; real-world deployment uses a Franka robot.
  • MetaWorld: On MetaWorld, frozen DINOv2 falls from 73.8% ID to 39.6% OOD, while STORM reaches 74.8% ID and improves OOD robustness.Fine-tuned DINOv2 improves OOD performance by 4.2 points but sacrifices ID performance; DINOSAUR reaches 73.2% ID and 46.0% OOD.
  • LIBERO: On LIBERO, STORM outperforms the best baseline by 10.7 points in ID and 19.0 points in OOD settings.Frozen DINOv2 achieves 78.9% ID and 70.3% OOD, while fine-tuning slightly reduces performance and DINOSAUR does not surpass the frozen baseline.
  • Real-world scenarios: In real-world evaluation, STORM achieves 55.2% ID success and 41.9% OOD success, exceeding frozen DINOv2 by 33.2 and 41.8 points.Frozen DINOv2 achieves 22.0% ID and 0.1% OOD success, while fine-tuning and DINOSAUR reach about 40% ID success.

C. Ablation studies

The ablations show that STORM’s two-phase, task-aware training and explicit mask information are important for stable and effective manipulation. Decoupling representation learning from policy adaptation improves ID and OOD performance, while compact spatial cues outperform unnecessarily complex encodings.

  • Two-phase learning and task awareness: Naïvely training task-aware slots jointly with the policy from limited demonstrations causes a significant drop in both ID and OD performance.The ablation attributes this degradation to unstable joint learning of object decomposition and policy control.
  • Two-phase learning and task awareness: Pretraining task-aware object-centric layers before policy training partially alleviates instability, improving OD generalization while maintaining comparable ID performance to frozen VFM features.The staged strategy first learns the representation independently and then adapts it during policy training.
  • Two-phase learning and task awareness: Removing task-aware training produces the DINOSAUR setup, which achieves 73.2% ID success but drops to 46.0% under OD conditions.The comparison isolates the contribution of aligning slots with task embeddings.
  • Two-phase learning and task awareness: STORM’s task-aware two-phase strategy achieves the best ablation results by stabilizing optimization, preserving coherent object bindings, and adapting slots to task-specific cues.The reported gains are consistent across both ID and OD performance.
  • Mask representation: Removing mask cues causes an important drop in success rates, confirming that explicit spatial information contributes substantially to manipulation performance.Mask features encode object location, extent, and shape, complementing appearance features.
  • Mask representation: The simple center mask encoding performs best on MetaWorld and is comparable to the full mask on MetaWorld-OOD, whereas bounding boxes offer limited benefit.Full masks add complexity without consistently improving performance, while bounding boxes are described as coarse and noise-sensitive.

V. CONCLUSION

STORM adapts frozen foundation-model features into task-aware, object-centric representations through staged learning. This approach improves robustness and generalization while avoiding the need to fully tune large backbones.

  • STORM adapts frozen foundation-model features into compact, task-aware, object-centric representations for robotic manipulation.The module uses semantic slot pretraining followed by joint policy adaptation.
  • Staged learning stabilizes slot formation and produces representations that are semantically meaningful and useful for control.Semantic slot pretraining precedes cautious joint adaptation with the policy.
  • STORM improves robustness to visual distractors and generalization over frozen and naively fine-tuned baselines on simulated and real-world benchmarks.
  • Small, semantically grounded adapters can efficiently leverage large visual foundation models without tuning the entire backbone.The paper presents this as a practical path toward strong and robust downstream behavior.
Loading 2601.20381v3…