Source-linked AI summary
RoboMP-DINOv2: Prompts, Not Filters for Robust Robot Manipulation
Han Qi, Heng Yang
TL;DR
Robot manipulation policies need to handle visual shifts without losing scene context relevant to action. RoboMP-DINOv2 prompts selected regions while preserving the full scene, and its MCR variant localizes color randomization. Across simulated settings it improves performance under visual shifts, including unseen object colors.
Problem
Robot policies encounter visual shifts, but general-purpose encoders do not identify task-relevant regions, while mask-based filtering can remove context that may affect the correct behavior.
Source: §1 · pp. 1–2Method
RoboMP-DINOv2 adds learned region-specific prompts to dense full-scene DINOv2 features and contextualizes all tokens; MCR randomizes color only within selected object regions.
Source: §1 · p. 2Results
Across seven settings, RoboMP-DINOv2 improves over DP-DINOv2 under spatial and clutter shifts; for unseen colors, RoboMP-DINOv2-MCR scores 72.5% versus 35.1%.
Source: §1 · p. 2Takeaways & Limitations
The evaluated designs improve robustness under the considered visual shifts while preserving behaviorally relevant scene information.
Source: §1 · pp. 2–3The examples of imperfect predicted masks do not establish explicit robustness to controlled mask corruption.
Source: §3.2.1 · p. 9
Abstract
from arXiv · showhide
Robot manipulation policies must generalize across visual shifts while preserving scene context relevant to action. General-purpose vision encoders are not tailored to visuomotor control, while object-centric approaches often use segmentation masks as hard filters that discard potentially useful context. We propose RoboMP-DINOv2 (Robotics Mask-Prompted DINOv2), a full-scene vision encoder that treats masks as spatial prompts rather than visibility filters. It extracts dense DINOv2 features from the full observation, injects learned region-specific embeddings at masked locations, and jointly contextualizes prompted and unprompted tokens for action prediction. We further introduce masked-region color randomization (MCR) to improve appearance robustness, yielding RoboMP-DINOv2-MCR. Across seven simulated manipulation settings, RoboMP-DINOv2 achieves 60.7% success under spatial shifts and 59.7% under scene clutter, compared with 50.7% and 41.0% for a DINOv2-based Diffusion Policy. Under unseen object colors, RoboMP-DINOv2-MCR achieves 72.5% success versus 35.1% for the strongest color-randomized baseline. Additional experiments and representation analyses show improved robustness while preserving behaviorally relevant scene information. Code is available at https://github.com/han20192019/RoboMP_DINOv2.
1 Introduction
Robot manipulation representations must balance focus on task-relevant regions with access to scene context. RoboMP-DINOv2 prompts selected regions without filtering out the rest of the scene, and the evaluated designs improve performance under visual shifts.
1 Introduction
- Visual changes can differ in behavioral relevance: distractors may alter observations without changing the desired action, while an obstacle in the motion path may require a different trajectory.Object-centric filtering can also discard unmasked context that may matter for behavior.Source: §1 · pp. 1–2
- MCR randomizes RGB appearance only within selected object regions, while leaving the rest of the scene unchanged.This is intended to discourage reliance on training-specific object colors while preserving unmasked contextual cues.Source: §1 · p. 2
- 60.7% versus 50.7% under spatial shifts and 59.7% versus 41.0% under scene clutter: RoboMP-DINOv2 outperforms DP–DINOv2 across seven simulated settings.It also achieves higher observed success than DP–DINOv2 in all seven settings under both regimes.Source: §1 · p. 2
- 72.5% versus 35.1% under unseen object colors: RoboMP-DINOv2-MCR outperforms the strongest evaluated alternative on six settings and matches the highest observed performance on one.The paper reports the strongest overall performance among the evaluated encoder and augmentation combinations.Source: §1 · pp. 2–3
- RoboMP-DINOv2 adds learned region-specific prompts to dense full-scene DINOv2 features, contextualizes prompted and unprompted tokens, and aggregates them for action prediction.Unlike hard object-centric filtering, it preserves the complete observation, including unmasked scene content.Source: §1 · pp. 2–3
2 Method
RoboMP-DINOv2 uses ordered masks as spatial prompts over dense full-scene DINOv2 features, retaining all tokens while contextualization and aggregation produce policy conditioning; MCR optionally randomizes selected-region colors during training.
2.2 RoboMP-DINOv2: Mask-Prompted Full-Scene Encoding
- Dense DINOv2 features receive learned region prompts without removing tokens; a transformer contextualizes prompted and unprompted tokens before full-scene pooling.Overlapping masks add their prompt embeddings at shared locations.Source: §2 · p. 3, §2.2 · pp. 4–5
2.1 Problem Formulation
- Evaluation masks come from an external Grounding DINO–SAM pipeline, while ordered mask slots map to learned prompts and object labels or descriptions are not encoder inputs.Training instead uses simulator ground-truth segmentation masks.Source: §2.1 · pp. 3–4
2.3 Policy Conditioning and Optimization
- Language features and recent conditioning features provide global conditioning to Diffusion Policy, and the encoder and policy are jointly optimized using only the policy training objective.The language instruction is encoded with a frozen CLIP text encoder.Source: §2.3 · p. 5
2.4 Masked-Region Color Randomization
- Masked-region color randomization replaces selected regions with a sampled RGB color during training while preserving their spatial support, scene geometry, and unmasked context.The original RGB observations are used at evaluation, and MCR does not change the inference architecture.Source: §2.4 · pp. 5–6
3 Experiments
Experiments assess seven simulated manipulation settings under spatial, clutter, and appearance shifts, comparing full-scene representations and mask-based alternatives. The evaluations include system-level success, controlled architecture comparisons, and representation-change analyses.
3.1 Experimental Setup
- The seven settings cover placement, obstacle avoidance, insertion, tool use, long-horizon interaction, and bimanual coordination; shared masks and policy-training inputs support representation comparisons.Policies train on clean scenes and are evaluated on held-out positions, added clutter, or unseen task-object colors.Source: §3.1 · pp. 6–8
3.2.1 System-Level Evaluation; 3.3 Appearance Generalization with Masked-Region Color Randomization
- Across seven settings, RoboMP-DINOv2 scores 60.7% vs 50.7% under spatial shifts and 59.7% vs 41.0% under clutter; RoboMP-DINOv2-MCR reaches 72.5% vs RoboGround–DINOv2's 35.1% under appearance shifts.The spatial and clutter comparisons are against DP–DINOv2; the appearance comparison uses color-randomized methods.Source: §3.2.1 · pp. 8–9, Table 1 · p. 9, §3.3 · p. 11
3.2.1 System-Level Evaluation
- Imperfect evaluation-time masks change RoboMP-DINOv2's spatial prompts without deleting full-scene features, unlike hard object-centric filtering; the examples do not establish robustness to controlled mask corruption.The displayed examples use predicted masks for Put Can with an obstacle, Long Horizon, and Two-Arm stacking.Source: §3.2.1 · p. 9, Figure 4 · p. 13
3.2.2 Matched Architecture Comparisons
- RoboMP-DINOv2 has the highest observed mean success across three cluttered-scene settings among the matched variants, which remove spatial localization, prompting, or contextualized global readout.The comparison uses the same downstream policy and 256-dimensional visual representations; differences are evidence about complete variants, not causal estimates of individual components.Source: Figure 2 · p. 10, §3.2.2 · p. 10
3.2.3 Feature Consistency under Clutter
- Under clutter, RoboMP-DINOv2 has lower feature distances than DP–DINOv2 on four of five settings, while DP–DINOv2 is slightly lower on Two-Arm stacking.The distances compare paired clean and perturbed observations with the simulator state fixed; they characterize perturbation sensitivity, not whether all control-relevant information is preserved.Source: §3.2.3 · pp. 10–11, Table 2 · p. 11
3.3 Appearance Generalization with Masked-Region Color Randomization
- With task-object recoloring, RoboMP-DINOv2-MCR reduces relative ℓ2 feature distance versus unaugmented DP–DINOv2 by 92.9× on Out-of-bin, 8.1× on Pull Tool, 5.5× on Peg Insertion, 9.8× on Long Horizon, and 31.8× on Two-Arm stacking.The comparison changes both encoder architecture and training augmentation, so it does not attribute the difference to either component alone.Source: §3.3 · p. 11
- Under simultaneous appearance and clutter shifts, RoboMP-DINOv2-MCR scores 0.20, above full-image-randomized DP–DINOv2 at 0.01 and the other evaluated combinations.Restricting randomization to masks without mask-conditioned encoding raises DP–DINOv2's score to 0.11.Source: Table 4 · p. 12, §3.3 · p. 12
- With a behaviorally relevant stove-light cue, RoboMP-DINOv2-MCR achieves 88% and 71% success for green and red lights, versus 25% and 19% for full-image-randomized DP–DINOv2.MCR randomizes selected object regions while leaving scene context, including the stove-light signal, available to the encoder.Source: §3.3 · p. 12, Figure 3 · p. 13