PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation

Hao Li, Ziqin Wang, Zi-han Ding, Shuai Yang, Yilun Chen, Yang Tian, Xiaolin Hu, Tai Wang, Dahua Lin, Feng Zhao, Si Liu, Jiangmiao Pang

arXiv:2602.09973v1cs.RO

TL;DR

RoboInter addresses the shortage of scalable, diverse intermediate supervision for generalizable robotic manipulation. It introduces a unified suite of annotated data, embodied VQA, and plan-then-execute VLA models, with experiments showing improved planner reasoning, grounding, and VLA performance. The suite remains bounded by fixed-base, single-arm data and deployment challenges from heavy inference.

  • Problem

    Existing manipulation datasets are costly, embodiment-specific, and rarely provide the fine-grained intermediate representations required by plan-then-execute VLA systems.

  • Method

    RoboInter combines semi-automatic annotation, RoboInter-Data, RoboInter-VQA, and modular or end-to-end RoboInter-VLA models linked by intermediate representations.

  • Results

    RoboInter improves VLM reasoning and grounding, while planner pretraining and intermediate-representation data provide performance and generalization gains for VLAs.

  • Takeaways & Limitations

    The suite establishes a unified foundation for studying and using diverse intermediate representations in robotic manipulation learning.

  • Takeaways & Limitations

    Annotations primarily cover fixed-base, single-arm robots, while planner robustness on unseen scenarios and executor inference efficiency remain open challenges.

Abstract

from arXiv · show

Advances in large vision-language models (VLMs) have stimulated growing interest in vision-language-action (VLA) systems for robot manipulation. However, existing manipulation datasets remain costly to curate, highly embodiment-specific, and insufficient in coverage and diversity, thereby hindering the generalization of VLA models. Recent approaches attempt to mitigate these limitations via a plan-then-execute paradigm, where high-level plans (e.g., subtasks, trace) are first generated and subsequently translated into low-level actions, but they critically rely on extra intermediate supervision, which is largely absent from existing datasets. To bridge this gap, we introduce the RoboInter Manipulation Suite, a unified resource including data, benchmarks, and models of intermediate representations for manipulation. It comprises RoboInter-Tool, a lightweight GUI that enables semi-automatic annotation of diverse representations, and RoboInter-Data, a large-scale dataset containing over 230k episodes across 571 diverse scenes, which provides dense per-frame annotations over more than 10 categories of intermediate representations, substantially exceeding prior work in scale and annotation quality. Building upon this foundation, RoboInter-VQA introduces 9 spatial and 20 temporal embodied VQA categories to systematically benchmark and enhance the embodied reasoning capabilities of VLMs. Meanwhile, RoboInter-VLA offers an integrated plan-then-execute framework, supporting modular and end-to-end VLA variants that bridge high-level planning with low-level execution via intermediate supervision. In total, RoboInter establishes a practical foundation for advancing robust and generalizable robotic learning via fine-grained and diverse intermediate representations.

1. Introduction

RoboInter addresses the lack of scalable, fine-grained intermediate supervision for generalizable VLA manipulation. It combines annotated data, embodied VQA, and plan-then-execute models into one suite.

  • Existing robot datasets remain costly, embodiment-specific, and limited in coverage, leaving a gap in generalization and robustness.
  • Plan-then-execute methods generate high-level structures before translating them into low-level actions, but their effectiveness depends on unavailable fine-grained intermediate representations.
  • RoboInter-Data provides over 230k episodes across 571 scenes with dense per-frame alignment across more than ten intermediate-representation categories.
  • RoboInter-Data combines large-scale coverage with human-in-the-loop verification rather than relying solely on automatic annotation.
  • RoboInter-VQA contains 9 spatial and 20 temporal embodied VQA categories, while RoboInter-VLA supports modular and end-to-end plan-then-execute variants.

2. Related Works

Prior work explores individual intermediate representations, annotated robot datasets, and planning-based execution, but existing resources generally lack native, diverse intermediate labels at scale.

  • Embodied manipulation research has used traces, optical flow, subtasks, key points, future images, and grounding boxes to guide action generation.
  • Some efforts collect large-scale data for a single intermediate representation, rather than covering diverse representations jointly.
  • Existing robot datasets span varied sources, scenarios, and skills but generally lack native intermediate-representation labels for real-world manipulation.
  • Subsequent annotation efforts add primitive labels, grasp descriptions, trajectories, or VQA, but each addresses only selected forms of intermediate supervision.
  • Plan-then-execute systems are categorized as implicit black-box methods or explicit methods that generate interpretable intermediate representations for action generation.

3. Dataset

RoboInter-Data integrates diverse manipulation sources with semi-automatic annotation and verification, then derives spatial and temporal VQA tasks from the resulting representations.

  • Data collection: RoboInter-Data combines In-the-Wild scenes and instructions with TableTop interaction data emphasizing quality and skill diversity.
  • Data collection: The integrated raw teleoperated recordings yield 230k manipulation episodes after screening and preprocessing.
  • Annotations and checking: RoboInter-Tool supports video segmentation, primitive-skill assignment, language annotation, contact-frame recording, and object segmentation with human inspection.
  • Post-processed annotations: Post-processing derives grasp affordance boxes, contact points, grasp poses, and placement annotations from recorded contacts, locations, and robot states.
  • VQA construction: RoboInter-VQA organizes tasks by spatial or temporal representation and by understanding or generation capability.
  • Dataset statistics: The dataset includes 6 robot-arm types, 571 scene types, 15 primitive skills, nearly 61M-frame object-grounding annotations, and about 70M-frame gripper-trace annotations.
  • VQA statistics: The VQA data contains approximately 1M spatial-generation, 172k spatial-understanding, 131k temporal-generation, and 935k temporal-understanding entries.

4. RoboInter-VLA

RoboInter-VLA separates high-level planning from low-level execution while allowing intermediate representations to condition the Executor through multiple architectural variants. Flexible Chain-of-Thought provides action-aligned intermediate supervision across these designs.

  • Architecture: RoboInter-VLA uses a Planner to produce intermediate representations and an Executor to translate observations, instructions, and those representations into low-level actions.
  • Planner: The Planner gains embodied capabilities through spatial and temporal VQA training on VLM architectures supporting single- and multi-image inputs.
  • Executor: The Executor uses a Qwen2.5-VL backbone with a diffusion action head and conditions on multi-view observations, language, and intermediate representations.
  • Plan-then-execute paradigms: The framework supports implicit end-to-end, explicit end-to-end, and modular Planner-to-Executor variants.
  • Flexible Chain-of-Thought: Flexible Chain-of-Thought combines selectable textual or visual representations, serving as Planner VQA supervision and Executor action guidance.
  • Evaluation: Third-party benchmark comparisons evaluate embodied, grounding, and general performance, while RoboInter-VQA measures spatial and temporal generation and understanding.

5. Benchmarking and Experiments

The experiments benchmark RoboInter planners and executors across embodied reasoning, open-loop action generation, and real-world ID/OOD manipulation. Results show gains from intermediate representations, with explicit and implicit variants exhibiting different strengths across evaluation settings.

  • Planner Benchmarking: Across spatial and temporal benchmarks, RoboInter models substantially outperform base VLMs in embodied grounding and reasoning.The benchmark covers spatial point, grounding, and temporal task-planning capabilities.
  • Planner Benchmarking: 49.1% and 12.7% improvements at 3B, and 76.8% and 42.8% at 7B, are reported over RoboBrain2.0 on RoboRefIt and RoboVQA.At 7B, RoboInter-Qwen-7B also achieves a 27.4% relative improvement over RoboBrain2.0-7B on Refcoco.
  • Planner Benchmarking: General and closed-source VLMs typically struggle with embodied intermediate generation, especially spatial grounding and future-trace prediction.Planning-choice tasks transfer better because they align more closely with general language-model abilities.
  • Executor Evaluation: 0.3543 mOLS for Te-Modular exceeds 0.3340 for EC-E2E and 0.3218 for IC-E2E, indicating benefits from decoupled planning and execution.Oracle-plus-Executor obtains the highest scores, while finer-grained intermediate representations produce larger gains in ablation results.
  • Real-World Evaluation: 77.3% versus 65.0% ID success and 58.3% versus 38.3% OOD success show IC-E2E outperforming Vanilla in closed-loop evaluation.EC-E2E is weaker on ID performance but can be stronger under OOD conditions, where its ID-to-OOD degradation is smaller.

6. Conclusion and Further Applications

RoboInter unifies annotated data, embodied VQA benchmarks, and plan-then-execute VLA models around intermediate representations. The suite is positioned as an open foundation for studying how such representations connect vision-language reasoning with robotic action.

  • Conclusion: RoboInter-Data contains over 230k episodes with dense per-frame annotations spanning diverse intermediate representations.The dataset is presented as a large-scale foundation for real-world manipulation research.
  • Conclusion: RoboInter-VQA benchmarks and improves embodied generation and understanding across rich spatial and temporal reasoning tasks.The suite evaluates embodied capabilities through systematic VQA tasks.
  • Conclusion: RoboInter-VLA integrates intermediate representations into modular and end-to-end frameworks to study their influence on execution performance.The resulting platform supports plan-then-execute research across multiple model designs.
  • Further Applications: RoboInter-Data also supports specialized generative models for individual representations and broader human–robot research directions.Its diverse scenes and large-scale annotations extend potential use beyond the paper’s VLM and VLA experiments.

A.1.1. Experimental Setting

The experimental setting evaluates RoboInter-VLA variants on real-world manipulation with a Franka arm, comparing ID/OOD success rates and open-loop performance. The results show strong IC-E2E ID performance, while EC-E2E provides better OOD robustness in selected comparisons.

  • Experimental Setting: Experiments use a Franka Research 3 arm with third-person and wrist cameras, delta end-effector control, and no proprioceptive input.The evaluation includes few-shot TableTop tasks and reports closed-loop success under ID and OOD conditions.
  • Tasks: The evaluation covers sequential pen collection, cup stacking, and additional manipulation tasks designed to test precision and generalization.The tasks vary object identity, spatial layout, containers, and instructions under OOD testing.
  • Closed-Loop Results: EC-E2E reaches 60.0% on Object Collecting and 73.3% on Cup Stacking, versus 53.3% and 60.0% for IC-E2E.EC-E2E also achieves a higher average success rate than IC-E2E in the reported comparisons.
  • Closed-Loop Results: EC-E2E’s ID-to-OOD drop is 8.3%, compared with 19% for IC-E2E.The modular variant shows competitive OOD generalization but a somewhat larger distribution-shift drop than EC-E2E.

A.1.2. Additional Real-world ID and OOD Validation

Additional validation separates precision-focused ID manipulation from language-guided OOD generalization. EC-E2E is stronger on OOD performance, whereas IC-E2E is more robust in-distribution, and both outperform the Vanilla and π0 baselines.

  • Additional Tasks: Tool Inserting tests precise contact handling and slot alignment in an ID-oriented manipulation task.The robot removes and reinserts a tool into 1.5 cm × 1.5 cm slots under varied initial positions and demonstrations.
  • Additional Tasks: Object Sorting tests language-guided OOD generalization using novel objects and containers beyond the training distribution.The OOD setting changes object and cup colors, shapes, or types.
  • Results: EC-E2E achieves stronger OOD performance, while IC-E2E exhibits superior ID robustness.The two variants therefore show complementary strengths across distribution conditions.
  • Results: Both variants outperform Vanilla and π0, indicating benefits from intermediate representations for ID precision and OOD generalization.The additional experiments reinforce the distinct ID and OOD strengths observed in the main evaluation.

A.1.3. More results and Visualization

Figures 7, 8, and 9 visualize RoboInter’s real-world experimental process, including precise manipulation and long-horizon cleaning behavior.

  • Figures 7, 8, and 9 provide visualizations of the real-world experimental process.
  • RoboInter-VLA generates precise actions such as grasping a pen while avoiding collisions.
  • RoboInter-VLA demonstrates long-horizon behavior by continuously cleaning a board.

A.2.1. Inference Time Analysis.

Inference analysis shows that explicit reasoning improves action-prediction accuracy but increases latency, motivating acceleration strategies for deployment.

  • Explicit reasoning significantly improves open-loop accuracy for VLA-OS and RoboInter-VLA at comparable model sizes.
  • Textual caching, chunked execution, and asynchronous dual-frequency execution accelerate real-world RoboInter-VLA deployment.
  • Autoregressive chain-of-thought generation increases inference latency.
  • Real-world inference tables report that explicit reasoning improves robustness while incurring higher latency.

A.2.2. Experiment Results for Data Scaling Law

Scaling experiments show that larger RoboInter-VQA and F-CoT datasets improve performance, although executor models benefit more strongly from additional F-CoT data than the VLM Planner.

  • RoboInter-VQA experiments vary annotated-data proportions from 10% to 100% using QwenVL2.5-3B.
  • Visual understanding and planning tasks exhibit scaling-law trends as dataset size increases.
  • F-CoT experiments compare Oracle+Executor and RoboInter-Te-Modular across varying data sampling rates.
  • 0.2347 → 0.2705 is the larger gain for Oracle+Executor, compared with 0.2109 → 0.2332 for RoboInter-Te-Modular.With more data, the executor conditioned on oracle intermediate signals benefits more readily, while the VLM Planner improves more slowly.

A.2.3. Ablation for Designs and Intermediate Representations Types of F-CoT

The ablations show that textual F-CoT is the strongest learning-based design, while spatially precise intermediate representations provide greater gains than coarse-grained cues. Cross-platform and closed-loop evaluations further report consistent advantages over vanilla baselines, alongside identified scope and deployment limitations.

  • Ablation for Designs: Textual F-CoT consistently performs best among learning-based models, indicating a stable and expressive interface for composing heterogeneous intermediate representations.It serializes intermediate representations as autoregressive text tokens.
  • Intermediate Representation Types: Spatially precise cues such as Object Box and Affordance provide much greater improvements than coarse-grained Subtask and Primitive skill representations.The comparison uses Oracle+Executor ablations and individual representation tests.
  • Cross-platform Evaluation: All RoboInterVLA variants consistently outperform the vanilla baseline across platforms, with Modular achieving the best overall accuracy among learned models.The evaluation spans five distinct embodiments, while Oracle+Executor serves as the upper bound.
  • Closed-loop Evaluation: 65.7 versus 63.8: RoboInter-IC-E2E with stronger embodied VLM initialization surpasses CogACT on SimplerEnv’s cross-embodiment evaluation.The evaluation covers 12 tasks across WidowX and Google Robot settings.
  • Limitations: The suite remains limited by annotations concentrated on fixed-base, single-arm robots, planner robustness on unseen scenarios, and relatively heavy executor inference.Future work is suggested for mobile or dual-arm systems and more efficient architectures.
Loading 2602.09973v1…