PaperlayerView exact revision on arXiv

Source-linked AI summary

A Taxonomy for Evaluating Generalist Robot Manipulation Policies

Jensen Gao, Suneel Belkhale, Sudeep Dasari, Ashwin Balakrishna, Dhruv Shah, Dorsa Sadigh

arXiv:2503.01238v3cs.ROcs.AIcs.LG

TL;DR

Robot-manipulation policies are difficult to compare because generalization evaluations vary across studies and can be hard to reproduce. STAR-Gen organizes generalization into visual, semantic, and behavioral perturbations and is instantiated in two real-world case studies. The case studies find weak semantic generalization among existing policies and show that GRoD outperforms other models in the ALOHA 2 study.

  • Problem

    Robot-manipulation generalization evaluations vary across studies, making progress difficult to measure and compare.

    Source: §I · p. 1

  • Method

    STAR-Gen organizes generalization around visual, semantic, and behavioral perturbations and is instantiated in Bridge V2 and ALOHA 2 real-world benchmarks.

    Source: §I · p. 1

  • Results

    Existing generalist policies struggle on most BridgeV2-‹ axes, especially semantic generalization, while GRoD consistently outperforms other models in the ALOHA 2 case study.

    Source: B. Main Results · p. 5, B. Results · p. 7

  • Takeaways & Limitations

    STAR-Gen offers a framework for more comprehensive generalization benchmarks and for studying policies across a wider set of tasks and settings.

    Source: §VII · p. 7

    Real-world evaluation constraints limited the study to a subset of the taxonomy’s axes and factors.

    Source: §VII · p. 7

Abstract

from arXiv · show

Machine learning for robot manipulation promises to unlock generalization to novel tasks and environments. But how should we measure the progress of these policies towards generalization? Evaluating and quantifying generalization is the Wild West of modern robotics, with each work proposing and measuring different types of generalization in their own, often difficult to reproduce settings. In this work, our goal is (1) to outline the forms of generalization we believe are important for robot manipulation in a comprehensive and fine-grained manner, and (2) to provide reproducible guidelines for measuring these notions of generalization. We first propose STAR-Gen, a taxonomy of generalization for robot manipulation structured around visual, semantic, and behavioral generalization. Next, we instantiate STAR-Gen with two case studies on real-world benchmarking: one based on open-source models and the Bridge V2 dataset, and another based on the bimanual ALOHA 2 platform that covers more dexterous and longer horizon tasks. Our case studies reveal many interesting insights: for example, we observe that open-source vision-language-action models often struggle with semantic generalization, despite pre-training on internet-scale language datasets. We provide videos and other supplementary material at stargen-taxonomy.github.io.

I. INTRODUCTION

STAR-Gen organizes robot-manipulation generalization around visual, semantic, and behavioral perturbations, addressing inconsistent and often narrow evaluations. The paper applies the taxonomy in two real-world benchmarking case studies and discusses how it can guide more comprehensive benchmark design.

I. INTRODUCTION

  • STAR-Gen groups perturbations by their effects on vision, language, and actions, then defines finer-grained axes for each combination.For example, Object Properties covers changing “put carrot on plate” to “put the orange object on the plate”.Source: §I · p. 1
  • The Bridge V2 case study evaluates open-source generalist policies using a benchmark intended as a reproducible, open-source design blueprint.A second case study uses ALOHA 2 for more dexterous, varied, and longer-horizon tasks.Source: §I · p. 1
  • The case studies comprise 1600+ robot trials across 14 generalization axes and generate findings about policy capabilities and model design choices.The authors aim for the taxonomy to guide training and evaluation efforts.Source: §I · p. 1

II. RELATED WORK

  • Prior dataset diversity efforts leave unclear which forms of diversity matter, while policy evaluations often use narrow, inconsistent axes.The paper presents STAR-Gen as a superset of prior axes intended to structure dataset, model, and benchmark development.Source: §II · pp. 1–2

III. WHAT IS GENERALIZATION?

The paper defines robot-manipulation tasks and policies, then frames generalization as evaluating a policy on perturbations of a base task. Perturbations are categorized by how they affect policy inputs and required behavior.

A. Preliminaries

  • An environment specifies state, observation, action, and transition spaces; a task specifies an initial-state distribution, language instruction, and success function.A policy maps observations and language to action distributions, while an expert policy produces successful episodes for a task.Source: A. Preliminaries · p. 2

B. Defining Generalization for Visuo-Lingual Policies

  • The framework structures generalization around perturbations to a base task and their effects on a policy’s inputs and outputs.This is proposed to address the challenge of characterizing generalization across tasks and large datasets.Source: B. Defining Generalization for… · p. 2
  • A base task is the desired application task, and a perturbation transforms it into a new task by applying a task delta.Source: B. Defining Generalization for… · p. 3
  • Visual perturbations change the initial image-observation distribution, semantic perturbations change the language instruction, and behavioral perturbations change the expert policy’s action distribution.These categories can overlap; the framework can also extend to modalities such as touch or sound.Source: B. Defining Generalization for… · p. 3

C. Additional Assumptions

STAR-Gen organizes manipulation generalization through a hierarchy of perturbation groupings and modality-based categories, while the paper notes scope limits and differences from prior evaluations.

C. Additional Assumptions

  • The evaluation focuses on atomic perturbations involving a single change, and does not consider long-horizon-specific perturbations such as reordering subtasks.The ALOHA 2 case study nevertheless applies STAR-Gen to longer-horizon tasks.Source: C. Additional Assumptions · p. 3

A. ‹-Gen: A Taxonomy of Generalization

  • STAR-Gen groups perturbations hierarchically as factors, axes, and categories, organized by which policy modalities they affect.Factors group perturbations with a common effect; axes group similar factors affecting a common set of modalities; categories group axes affecting the same modality combination.Source: A. ‹-Gen: A Taxonomy of Generalization · p. 4
  • The axes aim to cover salient challenges identified in prior work, though some axes may lack meaningful instantiations for particular tasks.The seven modality-based categories are intended to provide a complete framing of robot-manipulation generalization conditions.Source: A. ‹-Gen: A Taxonomy of Generalization · p. 4

B. Prior Notions of Generalization

  • STAR-Gen aims to be a comprehensive superset of prior generalization evaluations, whose groupings can be less fine-grained or task-specific.For example, RT-2 groups multiple visual and behavioral axes together, while Colosseum includes receiver-object perturbations that apply only to some tasks.Source: B. Prior Notions of Generalization · p. 4
  • BridgeV2-‹ defines perturbations relative to in-domain evaluation-scene data to make them easier to control.The paper notes that reproducing outside-source scenes can be difficult and may introduce unintended perturbations.Source: B. Prior Notions of Generalization · p. 4

V. CASE STUDY 1: BRIDGE V2

BridgeV2-‹ demonstrates how STAR-Gen can guide a reproducible, open-source generalization benchmark using Bridge V2. The case study evaluates open-source policies on base tasks and perturbations spanning multiple taxonomy axes.

V. CASE STUDY 1: BRIDGE V2

  • The benchmark is designed as a reproducible, open-source blueprint for evaluating generalization using the Bridge V2 dataset and platform.Bridge V2 is used for pre-training and has prior work reproducing its training environments for evaluation.Source: §V · p. 5, A. Instantiating ‹-Gen on Bridge V2 · p. 5

A. Instantiating ‹-Gen on Bridge V2

  • The four base tasks are put carrot on plate, put knife on plate, flip pot upright, and put plate in sink, chosen based on pre-training support.They cover different levels of alignment with Bridge V2 tasks in the replicated sink environment.Source: A. Instantiating ‹-Gen on Bridge V2 · p. 5
  • BridgeV2-‹ evaluates four in-distribution base tasks and 55 perturbations covering 13 of 22 STAR-Gen axes.Some axes are omitted because they are incompatible with the selected base tasks.Source: A. Instantiating ‹-Gen on Bridge V2 · p. 5
  • The evaluated policies are OpenVLA, MiniVLA, and a third-party reimplementation of π0.These vision-language-action models fine-tune foundation models on robot data and represent varied design choices.Source: A. Instantiating ‹-Gen on Bridge V2 · p. 5
  • Models are pre-trained on Bridge V2, then co-fine-tuned on evaluation-scene demonstrations together with Bridge V2.The study collects 10 demonstrations each for the carrot and knife tasks, and 50 each for the pot and plate tasks.Source: A. Instantiating ‹-Gen on Bridge V2 · p. 5
  • Each evaluation stops at success, a dangerous or irrecoverable state, or 100 timesteps, with five trials per condition for each model.Fig. 3 shows sink-color, plate-color, and camera-angle perturbations; Fig. 4 shows object-pose examples.Source: A. Instantiating ‹-Gen on Bridge V2 · p. 5, Figure 3 · p. 5, Figure 4 · p. 5

B. Main Results

The BridgeV2-STAR-Gen results show that generalist policies struggle on most axes, especially semantic generalization, with distinct strengths and weaknesses across models. Targeted evaluations reveal that design choices affect different axes unevenly, while semantic axes vary in importance with deployment conditions.

B. Main Results

  • Semantic generalization is mostly weak across 885 BridgeV2-STAR-Gen trials, despite language-model backbones; the authors suggest improving robot language annotations as one possible response.The paper cautions against relying only on language-model initialization to improve semantic generalization.Source: B. Main Results · pp. 5–6

C. Investigating VLA Design Decisions

B. Main Results

  • OpenVLA is weaker on visual generalization, MiniVLA on visual + behavioral axes, and π0 generally performs best, possibly due to its VLM backbone or flow-based action chunking.OpenVLA is best at object properties but still struggles with other semantic axes.Source: B. Main Results · p. 6
  • Performance correlations are higher within generalization categories than across them, except for semantic axes, which may require different kinds of reasoning.The authors contrast object-property understanding with language rephrasing as distinct semantic reasoning demands.Source: B. Main Results · p. 6, Table III · p. 6
  • Visual-only axes generally show stronger generalization than behavioral axes, except Viewpoint; the authors recommend prioritizing behavioral axes over visual-only ones other than Viewpoint.They hypothesize that vision-language pretraining is more likely to convey visual robustness than generalization to new behavior.Source: B. Main Results · p. 6
  • Semantic axes should be prioritized for open-ended language use, but can be de-prioritized when a separate system supplies a limited instruction set.The authors make prioritization contingent on the policy’s deployment setting.Source: B. Main Results · p. 6

C. Investigating VLA Design Decisions

  • Larger, more diverse robot datasets significantly improved visual + behavioral axes (M = 0.22 vs. M = 0.48), but not the weakest Viewpoint, Morphed Objects, or Multi-Object Referencing axes.The comparison uses Bridge-only OpenVLA versus a model trained on a larger cross-embodiment OXE mixture.Source: C. Investigating VLA Design Decisions · p. 6
  • A larger LLM modestly improved semantic axes without significance (M = 0.18 vs. M = 0.35), while VQA co-training improved visual axes (M = 0.30 vs. M = 0.45) but had mixed semantic effects.VQA co-training improved three semantic axes but hurt Object Properties; the authors note that absolute performance remained low for the LLM comparison.Source: C. Investigating VLA Design Decisions · p. 6

VI. CASE STUDY 2: BIMANUAL MANIPULATION

The ALOHA 2 case study applies STAR-Gen to more varied, dexterous, and longer-horizon manipulation tasks. Across 390 trials, GRoD consistently outperforms the other evaluated models, while the discussion notes limits in evaluation coverage and in the taxonomy’s fine-grained axes.

VI. CASE STUDY 2: BIMANUAL MANIPULATION

  • The study develops an additional case study based on the bimanual ALOHA 2 platform and evaluates three models: a multi-task diffusion policy, a reimplementation of π0, and proprietary Gemini Robotics On-Device (GRoD).Source: §VI · p. 7, A. Experimental Setup · p. 7

A. Experimental Setup

  • The case study uses a proprietary dataset of thousands of teleoperated demonstration hours to evaluate 17 base tasks and 68 perturbations across seven STAR-Gen axes.The tasks include tightening a water bottle and folding a dress; evaluation conditions match those used for Gemini Robotics and are recategorized by STAR-Gen axes.Source: A. Experimental Setup · p. 7

B. Results

  • GRoD consistently outperforms the diffusion policy and reimplemented π0 across 390 trials, and STAR-Gen identifies language translation as a perturbation where GRoD is robust while the other models fail almost entirely.The authors speculate that GRoD’s strong VLM backbone and other architectural enhancements may explain its performance.Source: B. Results · p. 7, A. Experimental Setup · p. 7

VII. DISCUSSION

  • The discussion presents STAR-Gen as a real-world benchmark-design framework and a means to study a wider set of tasks and policies on ALOHA 2.The authors hope it will help improve benchmark comprehensiveness and generate better insights.Source: §VII · p. 7
  • Real-world evaluation time constrained the study to a subset of axes and factors; the authors suggest simulation could support broader, more efficient measurement.Their evaluations comprised 1600+ trials.Source: §VII · p. 7
  • The fine-grained axes are empirically derived and may need expansion as practitioners identify new generalization challenges.The authors describe STAR-Gen as a starting point that future work can revise to meet robotics practitioners’ needs.Source: §VII · p. 7

APPENDIX A TAXONOMY CATEGORIZATION

STAR-Gen categorizes perturbations by which policy modalities they affect, while treating its axes as a human-specified, revisable starting point for more systematic discussion of generalization.

APPENDIX A TAXONOMY CATEGORIZATION

  • Visual, semantic, and behavioral perturbations affect image input, language instruction, and required behavior, respectively; combinations span the other modality pairings and all three.Isolated behavioral factors, such as object mass or friction, are unobserved from a single observation; changing an instruction-specified object color can affect both vision and language without changing behavior.Source: Appendix A · p. 10

APPENDIX B FREQUENTLY ASKED QUESTIONS

  • The appendix also introduces FAQs on taxonomy assumptions and scope, but the supplied answer on subjectivity frames categorization as inherently subjective.The taxonomy seeks greater structure and comprehensiveness using human-interpretable concepts such as task-relevant objects and verbs.Source: Appendix B · p. 10, Q: Are the axes and factors in ‹-Gen… · p. 10

Q: Is ‹-Gen meant to be comprehensive?

Q: Are the axes and factors in ‹-Gen subjective?

  • The taxonomy is intended to give practitioners a shared vocabulary for finer-grained model insights, while remaining open to revision as evaluation understanding grows.Which axes matter depends on downstream applications.Source: Q: Are the axes and factors in ‹-Gen… · p. 10

Q: Is each axis equally important for generalization?

The authors leave axis importance to application needs and do not quantify perturbation magnitude; their BridgeV2-‹ setup uses four base tasks and seeks to isolate specified shifts.

Q: Can you quantify how much different perturbations affect a base task?

A. Base Tasks

  • The four BridgeV2-‹ base tasks are put carrot on plate, put knife on plate, flip pot upright, and put plate in sink, selected for varying alignment with demonstrations.Carrot and knife share an initial configuration, as do pot and plate, requiring the policy to use language to distinguish tasks; the environment is tuned for reasonable zero-shot performance on carrot and knife.Source: A. Base Tasks · pp. 10–11
  • OpenVLA performs the carrot and knife tasks somewhat reliably zero-shot, rarely flips the pot, and fails the plate-in-sink task with largely meaningless behavior.The plate-in-sink instruction is not demonstrated in this sink, and its plate differs substantially in initial pose, appearance, and geometry from demonstrated tasks.Source: A. Base Tasks · p. 11

B. Evaluation Procedure Details

  • Evaluations stop at human-judged success, dangerous or irrecoverable state, or 100 timesteps, with task-specific stable-placement and flipping criteria.The setup minimizes other image differences from in-domain demonstrations so results reflect the specified perturbation; real-world initial-state variation is minimized but cannot be eliminated.Source: B. Evaluation Procedure Details · p. 11, A. Base Tasks · p. 11

C. Co-fine-tuning vs. Fine-tuning

The evaluation co-fine-tunes models with pre-training and in-domain data, using task-specific demonstrations and model training variants; the supplied figure reports improved in-distribution and multi-axis performance after co-fine-tuning OpenVLA.

C. Co-fine-tuning vs. Fine-tuning

  • The evaluation co-fine-tunes each model with its pre-training data and in-domain data to preserve original capabilities and support generalization and fair comparison.This may become impractical as pre-training datasets grow or are unavailable; fine-tuning only on in-domain data risks catastrophic forgetting, and best practices are left to future work.Source: C. Co-fine-tuning vs. Fine-tuning · p. 11

D. In-Domain Data

  • In-domain demonstrations come from one experienced teleoperator; carrot and knife receive 10 demonstrations each, while pot and plate receive 50 each for satisfactory in-distribution performance.Source: D. In-Domain Data · pp. 11–12

E. Model Details

  • Model details include distinct co-fine-tuning arrangements and training variants, including VQA data, alternate action tokenization, and a randomly initialized OpenVLA model.The randomly initialized model converged on training metrics but failed all in-distribution base tasks and was excluded; co-fine-tuning generally used 5K gradient steps, while some variants differed substantially.Source: E. Model Details · p. 12
  • Co-fine-tuning OpenVLA on in-domain data significantly improves in-distribution performance and many generalization axes.The figure compares zero-shot OpenVLA with the same model co-fine-tuned for the put-carrot and put-knife tasks.Source: Figure 11 · p. 12

F. Impact of In-Domain Data

Co-fine-tuning OpenVLA on in-domain data improves its base-task and generalization performance; a randomly initialized variant fails the in-distribution tasks, and the evaluation specifies its conditions across task groups.

F. Impact of In-Domain Data

  • Co-fine-tuning OpenVLA on in-domain data significantly improves in-distribution base-task performance and multiple generalization axes.The comparison is between zero-shot OpenVLA and the same model co-fine-tuned for put-carrot and put-knife tasks.Source: F. Impact of In-Domain Data · pp. 12–13

G. Random Initialization

  • OpenVLA trained from randomly initialized weights failed completely on in-distribution base tasks despite converged training metrics, so it was excluded from evaluations.Source: G. Random Initialization · p. 13

H. Evaluation Conditions

  • The evaluation conditions are documented for carrot and knife, pot and plate, and compositional experiments, with implementation adjustments for recoloring sinks and changing relative table height.Sink recoloring is applied to policy images using SAM 2.1 Large [54]; relative table shortening is achieved by raising the robot's mounting table.Source: H. Evaluation Conditions · p. 13, Table IV · p. 14, Table V · p. 17

I. More Evaluation Results

Compositional evaluations show robustness varies by axis combination and model. Additional tables report detailed success rates for the main and compositional evaluations.

1) Composition:

  • The larger OXE mixture seems to improve S-PROP + S-LANG, while π0 performs best on VB-POSE + VB-ISC.The authors suggest greater language-instruction variety as a possible explanation for the semantic-axis result.Source: 1) Composition · p. 20, Table VII · p. 21
  • MiniVLA and π0 are the most performant on V-SC + V-OBJ, with some models described as fairly robust to this combination.Table VII presents compositional results for two axes from semantic, visual, and visual-plus-behavioral categories.Source: 1) Composition · p. 20, Table VII · p. 21

2) Detailed Results:

  • Tables VIII–X provide detailed success rates for the main evaluations and compositional evaluations.Tables VIII and IX cover the “put carrot,” “put knife,” “flip pot,” and “put plate” tasks; Table X covers compositional evaluation.Source: 2) Detailed Results · p. 20, Table VIII · p. 21, Table IX · p. 22
  • Figure 12 shows a demonstration example used to generate New Object perturbations for “pick up the AAA battery.”Source: Figure 12 · p. 23

APPENDIX D AUTOMATIC BENCHMARK DESIGN

The appendix presents a foundation-model approach for proposing benchmark evaluation conditions across STAR-Gen axes. It also notes that human oversight in BridgeV2-STAR-Gen evaluations remains a limitation and may introduce effort and bias.

APPENDIX D AUTOMATIC BENCHMARK DESIGN

  • Human oversight still informed BridgeV2-STAR-Gen evaluation conditions, requiring effort and potentially introducing unintended bias.The authors see reducing human involvement in future evaluation construction as beneficial.Source: Appendix D · p. 23
  • A vision-language model receives a base task and proposes perturbations corresponding to a STAR-Gen generalization axis.The prompt combines an initial scene image, the task instruction, and directions for modifying the task; visual axes produce image-edit instructions, semantic axes produce new language instructions, and combined axes produce both.Source: Appendix D · p. 23
  • The interactive system uses Gemini 2.0 Flash and JSON schemas to generate perturbed tasks across five STAR-Gen axes.Users can provide base-task images and language instructions; the described axes include Visual Task Object, Object Properties, Object Poses, Action Verbs, and New Object.Source: Appendix D · p. 23
Loading 2503.01238v3…