PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulation

Hao Li, Ziqin Wang, Zi-han Ding, Shuai Yang, Yilun Chen, Yang Tian, Xiaolin Hu, Tai Wang, Dahua Lin, Feng Zhao, Si Liu, Jiangmiao Pang

arXiv:2602.09973v1cs.RO

TL;DR

Existing manipulation datasets rarely provide the fine-grained intermediate representations needed for plan-then-execute VLA systems. RoboInter addresses this gap with a unified suite of annotated data, VQA benchmarks, and VLA models, and experiments show intermediate representations improve executor action generation.

  • Problem

    Existing manipulation datasets rarely provide the fine-grained intermediate representations required for plan-then-execute VLA systems, while collecting such annotations remains costly and infrastructure-intensive.

    Source: §1 · pp. 1–2

  • Method

    RoboInter combines semi-automatic annotation, over 230k episodes across 571 scenes, spatial-temporal VQA benchmarks, and modular or end-to-end VLA frameworks using intermediate supervision.

    Source: §1 · pp. 2–3

  • Results

    Te-Modular exceeded EC-E2E in mOLS (0.3543 v.s. 0.3340), while Oracle + Executor achieved the highest scores, showing that intermediate representations improve action generation.

    Source: §5.2 · p. 10

  • Takeaways & Limitations

    RoboInter establishes a unified foundation for studying and advancing intermediate representations in plan-then-execute robotic manipulation.

    Source: §6 · p. 12

    The annotations primarily cover fixed-base, single-arm robots, limiting embodiment diversity and cross-embodiment generalization.

    Source: §A.8 · p. 69

Abstract

from arXiv · show

Advances in large vision-language models (VLMs) have stimulated growing interest in vision-language-action (VLA) systems for robot manipulation. However, existing manipulation datasets remain costly to curate, highly embodiment-specific, and insufficient in coverage and diversity, thereby hindering the generalization of VLA models. Recent approaches attempt to mitigate these limitations via a plan-then-execute paradigm, where high-level plans (e.g., subtasks, trace) are first generated and subsequently translated into low-level actions, but they critically rely on extra intermediate supervision, which is largely absent from existing datasets. To bridge this gap, we introduce the RoboInter Manipulation Suite, a unified resource including data, benchmarks, and models of intermediate representations for manipulation. It comprises RoboInter-Tool, a lightweight GUI that enables semi-automatic annotation of diverse representations, and RoboInter-Data, a large-scale dataset containing over 230k episodes across 571 diverse scenes, which provides dense per-frame annotations over more than 10 categories of intermediate representations, substantially exceeding prior work in scale and annotation quality. Building upon this foundation, RoboInter-VQA introduces 9 spatial and 20 temporal embodied VQA categories to systematically benchmark and enhance the embodied reasoning capabilities of VLMs. Meanwhile, RoboInter-VLA offers an integrated plan-then-execute framework, supporting modular and end-to-end VLA variants that bridge high-level planning with low-level execution via intermediate supervision. In total, RoboInter establishes a practical foundation for advancing robust and generalizable robotic learning via fine-grained and diverse intermediate representations.

1. Introduction

RoboInter addresses the lack of scalable, high-quality intermediate supervision for generalizable robotic manipulation by providing annotated data, embodied VQA benchmarks, and plan-then-execute VLA models.

Motivation

  • Existing robot datasets rarely provide fine-grained intermediate representations, while collecting and annotating such data remains costly, infrastructure-intensive, and difficult to scale.Prior automated or semi-automated efforts face distribution shift, small scale, or misalignment with step-wise actions.Source: §1 · pp. 1–2

RoboInter-Data

  • RoboInter-Data contains over 230k episodes across 571 scenes with dense, temporally aligned annotations spanning more than ten intermediate-representation categories.Built with RoboInter-Tool’s semi-automatic GUI and human-in-the-loop verification, it aligns annotations with actions, robot states, and third-person and wrist-view observations.Source: §1 · p. 2

RoboInter-VQA

  • RoboInter-VQA benchmarks embodied VLM reasoning through 9 spatial and 20 temporal categories covering embodied-scene understanding and generation.The benchmark is built from curated VQA data used to train high-level VLM planners.Source: Figure 1 · p. 2, §1 · p. 3

RoboInter-VLA

  • RoboInter-VLA integrates high-level planning with low-level execution in modular and end-to-end VLA variants using intermediate supervision.The framework supports systematic analysis of how intermediate representations affect VLA generalization and controllability.Source: §1 · pp. 1, 3

Empirical findings

  • Experiments show that RoboInter-Data improves VLM planners’ embodied reasoning and grounding, while pretrained planners and intermediate representations improve VLA performance and generalization in open- and closed-loop manipulation.The suite also analyzes trade-offs among VLA variants and is intended as an open-source foundation for future intermediate-representation research.Source: §1 · p. 3

2. Related Works

Prior work develops diverse embodied intermediate representations, manipulation datasets, and plan-then-execute VLA systems, but coverage, reliability, and native dense supervision remain limited. RoboInter addresses these gaps with dense multi-representation annotations and pretrained planners supporting implicit and explicit reasoning.

Embodied intermediate representation

  • Embodied manipulation research uses traces, optical flow, subtasks, key points, future images, and grounding boxes, but representation reliability varies across scenes.Vision foundation models can estimate some representations, yet results are not always reliable; task-specific resources such as RoboBrain (Team et al., 2025) and VeBrain (Luo et al., 2025) train VLMs to generate them.Source: §2 · p. 3
  • Large-scale datasets have also focused on collecting a single intermediate representation for action generation.This complementary direction is represented by Li et al. (2025b) and Yuan et al. (2024).Source: §2 · p. 3

Robotic manipulation datasets

  • Existing robotic datasets span diverse sources, scenarios, and skills, but generally lack native intermediate-representation labels for real-world manipulation.Later efforts add primitive subtasks, grasp and motion descriptions, gripper-derived 2D trajectories, or sampled-frame VQA, whereas RoboInter provides dense per-frame annotations across varied representations.Source: §2 · pp. 3–4

Embodied planning for execution

  • Plan-then-execute VLA systems range from implicit black-box reasoning to explicit interpretable representations, including π0.5 (Intelligence et al., 2025), Rekep (Huang et al., 2024b), ECoT (Zawalski et al., 2024), and VLA-OS (Gao et al., 2025a).RoboInter supplies large-scale intermediate-representation data and pretrained VLM Planners to support both implicit and explicit reasoning in VLA variants through efficient information aggregation.Source: §2 · p. 4

3. Dataset

RoboInter-Data combines diverse raw manipulation recordings with semi-automatic, quality-controlled annotation of fine-grained intermediate representations. RoboInter-VQA reorganizes these annotations into spatial and temporal understanding and generation tasks, supported by large-scale dataset statistics.

3.1. RoboInter-Data

  • RoboInter-Data integrates in-the-wild and tabletop recordings from Droid (Khazatsky et al., 2024), OXE (et al., 2023), and RH20T (Fang et al., 2023) into 230k screened manipulation episodes.The dataset covers diverse scenes, instructions, skills, and video recordings, with in-the-wild data emphasizing scene diversity and tabletop data emphasizing quality and skill diversity.Source: §3.1 · p. 4
  • RoboInter-Tool annotates primitive-skill clips, language, contact frames, manipulated objects, end-effector traces, and derived grasp, placement, and gripper representations with automated assistance and review.The tool uses ChatGPT for preliminary language references, SAM2 (Ravi et al., 2024) for object segmentation and tracking, and calibration, detection, and point tracking for reliable 2D end-effector reconstruction.Source: §3.1 · pp. 4–5

3.2. RoboInter-VQA

  • RoboInter-VQA organizes spatial and temporal tasks by understanding versus generation, covering selection, judgment, representation prediction, trace generation, and multi-step planning under varied context.Temporal generation varies contextual completeness and evaluates trace generation with or without initial waypoints, while spatial generation targets boxes, poses, placements, key points, and gripper boxes.Source: §3.2 · pp. 5–6

3.3. Data Statistics

  • The dataset spans 230k episodes across 6 robot-arm types, 571 scenes, and 15 primitive skills, with nearly 61M object-grounding and about 70M gripper-trace frames.It also contains 190k affordance-box and placement annotations, nearly 760k language clips, and VQA subsets totaling approximately 1M spatial-generation, 172k spatial-understanding, 131k temporal-generation, and 935k temporal-understanding entries.Source: §3.3 · p. 6

4. RoboInter-VLA

RoboInter-VLA is a plan-then-execute family that combines a VLM-based Planner for intermediate representations with an Executor for low-level actions. It supports multiple execution paradigms and flexible chain-of-thought conditioning across embodied tasks.

4. RoboInter-VLA

  • RoboInter-VLA combines a Planner for high-level embodied reasoning and intermediate representations with an Executor that translates observations and instructions into low-level actions.The Planner generates intermediate representations, while the Executor produces multi-step action chunks from multimodal inputs.Source: §4 · p. 6

4.1. Model Architecture

  • The Planner uses VLM-based VQA co-training, while the Executor combines a Qwen2.5-VL backbone, DiT action head, and controllable information aggregation.The Planner supports single- and multi-image inputs; the Executor conditions on visual observations, language, and intermediate representations.Source: §4.1 · pp. 6–7

4.2. Plan-Then-Execute Paradigms

  • RoboInter-VLA provides implicit end-to-end, explicit end-to-end, and modular Planner-to-Executor paradigms with distinct uses of pretrained features and intermediate representations.The modular variant trains with ground-truth representations but uses Planner-predicted representations during inference.Source: §4.2 · p. 7
  • Flexible Chain-of-Thought combines selectable textual or visual representations as Planner supervision and action-aligned Executor guidance across architectures.Supported representations include subtasks, skills, object bounding boxes, affordance boxes, and motion traces.Source: §4.2 · pp. 7–8

5. Benchmarking and Experiments

Experiments show that RoboInter improves embodied planner reasoning and executor performance across external benchmarks, open-loop evaluations, and real-world closed-loop manipulation. Intermediate representations and pretrained planning provide measurable gains, while representation granularity and evaluation distribution shape outcomes.

5.1. Benchmarking the Planner

  • RoboInter models substantially outperform base VLMs across Where2Place, RoboRefIt, and RoboVQA, with RoboInter-Qwen-7B achieving a 27.4% relative improvement over Robobrain2.0-7B on grounding.At 3B, RoboInter-Qwen-3B improves over RoboBrain2.0 by 49.1% on RoboRefIt and 12.7% on RoboVQA; at 7B, gains reach 76.8% and 42.8%.Source: §5.1 · pp. 8–9

5.2. Open-Loop Evaluation of the Executor

  • 0.3543 mOLS: Te-Modular surpasses EC-E2E’s 0.3340 and IC-E2E’s 0.3218, while Vanilla reaches 0.3086, showing benefits from explicit representations and decoupled planning.Oracle + Executor performs substantially best; in Table-Top evaluation, Te-Modular and Oracle + Executor converge faster and higher, while IC-E2E consistently beats Vanilla after 20k steps.Source: Table 4 · p. 9, §5.2 · p. 10

5.1. Benchmarking the Planner

  • RoboInter-VQA reveals that general and closed-source VLMs usually lack embodied intermediate-representation abilities, while curated annotations improve spatial and temporal reasoning.Most spatial-generation accuracies remain below 40%; RoboBrain-2.0 attains a better Trace Generation DTW than Qwen-VL-2.5 (541 v.s. 1702).Source: §5.1 · p. 9

5.3. Closed-Loop Real-World Evaluation of the Executor

  • 60.0% OOD success: EC-E2E exceeds IC-E2E’s 58.3% average and drops only 8.3% from ID to OOD, versus IC-E2E’s 19% decline.Conversely, IC-E2E leads ID success at 77.3% versus EC-E2E’s 68.3%, while EC-E2E is stronger on Object Collecting and Cup Stacking under OOD conditions.Source: Figure 5 · p. 11, §5.3 · pp. 11–12

6. Conclusion and Further Applications

RoboInter establishes a unified platform for intermediate representations in plan-then-execute VLA systems, combining large-scale annotated data, embodied VQA, and modular or end-to-end VLA frameworks. Its temporally aligned annotations also support specialized generative models, human–robot interaction, embodied world models, and video action models.

Conclusion

  • RoboInter unifies RoboInter-Data, RoboInter-VQA, and RoboInter-VLA to advance intermediate representations as a bridge between vision-language reasoning and robotic action.RoboInter-Data contains over 230k episodes with dense per-frame annotations; RoboInter-VQA benchmarks spatial and temporal reasoning; RoboInter-VLA studies intermediate signals in modular and end-to-end execution frameworks.Source: §6 · p. 12

Further applications

  • RoboInter-Data supports specialized generative models for individual intermediate representations through diverse scenes and large-scale, high-quality annotations.Its annotations span multiple representation types rather than a single annotation category, enabling representation-specific pre-training.Source: §6 · p. 12
  • RoboInter-Data enables shared-autonomy human–robot interaction by connecting high-level instructions with low-level control through traces, bounding boxes, and subtask sequences.These intermediate representations are intended to support intuitive and precise interaction systems.Source: §6 · p. 12
  • RoboInter-Data supports embodied world model learning because its annotations are temporally aligned with 640×360 manipulation videos.The representations provide structured control signals for controllable video generation, while robot, object, and environment information supervises structured embodied world models.Source: §6 · p. 12
  • RoboInter-Data supports video action model learning by combining action annotations, videos, and diverse intermediate representations for rich, multi-level supervision.The dataset’s multimodal and multilevel supervision makes it suitable for training video action models.Source: §6 · p. 12

A. Appendix · A.1. Real-World Experiments for Executor · A.1.1. Experimental Setting

Real-world closed-loop experiments evaluate RoboInter executor variants on a Franka arm across four long-horizon manipulation tasks under in-distribution and out-of-distribution settings. EC-E2E achieves the strongest reported average and robustness, while Modular remains competitive but is more sensitive to distribution shift.

A.1.1. Experimental Setting

  • Experiments use a Franka Research 3 with a Robotiq-2f-85 gripper, delta end-effector controls, two camera views, no proprioception, and a control loop below 10 Hz.Inference runs on an online A800 cluster communicating with a local control PC, while demonstrations use a SpaceMouse.Source: §A.1.1 · p. 19
  • The comparison includes OpenVLA, π0, Vanilla, RoboInter-IC-E2E, RoboInter-EC-E2E, and RoboInter-Modular, spanning pretrained, from-scratch, end-to-end, and modular executor designs.EC-E2E jointly optimizes action and chain-of-thought generation with a 1:1 mixture of annotated and collected data; Modular separately guides Planner and Executor learning.Source: §A.1.1 · p. 19
  • EC-E2E uses a shortened inference chain of thought and caches slowly varying components, refreshing them every 30 control steps to accommodate real-world execution.The cached components include the subtask and affordance box.Source: §A.1.1 · p. 19
  • The benchmark covers Object Collecting, Cup Stacking, Towel Folding, and Clutter Cleaning, testing precise placement, spatial grounding, deformable-object manipulation, sustained operation, and spatial perception.Demonstration counts are 100, 150, 64, and 150 respectively, with OOD changes to objects, layouts, materials, backgrounds, disturbances, or continuous streams.Source: §A.1.1 · p. 19
  • Additional validation examines Tool Inserting as a precision in-distribution task and Object Sorting as an out-of-distribution test of language-guided generalization.The figure specifically highlights accurate contact handling, slot alignment, and novel objects and containers.Source: Figure 6 · p. 20
  • EC-E2E achieves higher success than IC-E2E on Object Collecting (60.0% vs. 53.3%) and Cup Stacking (73.3% vs. 60.0%), with a higher average (60.0% vs. 58.3%).Its ID-to-OOD drop is 8.3%, compared with 19% for IC-E2E.Source: §A.1.1 · pp. 19–20, Table 6 · p. 19
  • The Modular variant delivers strong real-world performance and competitive OOD generalization, but its asynchronous two-module design has a slightly larger ID→OOD drop than EC-E2E.It nevertheless substantially benefits from the intermediate representation.Source: §A.1.1 · p. 20

A.1.2. Additional Real-world ID and OOD Validation

Additional real-world validation separates ID precision from OOD language-guided generalization using tool insertion and object sorting tasks. EC-E2E is stronger OOD, IC-E2E is more robust ID, and both outperform Vanilla and 𝜋0.

Tool Inserting

  • Tool Inserting tests ID-oriented precision by requiring accurate extraction and reinsertion of a metal tool across two slot configurations with different starting positions and demonstrations.Each slot measures 1.5 cm × 1.5 cm, emphasizing fine-grained contact handling.Source: §A.1.2 · p. 20

Object Sorting

  • Object Sorting tests OOD generalization by requiring language-guided placement of novel objects and cups with unseen colors, shapes, or types into corresponding containers.Training uses only red, green, and blue objects and cups with simple pick-and-place motions.Source: §A.1.2 · p. 20

Results

  • Both RoboInter-VLA variants outperform Vanilla and 𝜋0, with EC-E2E stronger on OOD generalization and IC-E2E more robust on ID precision.Figure 6 shows that explicit reasoning benefits EC-E2E on OOD performance, while IC-E2E better handles fine-grained in-distribution actions.Source: §A.1.2 · p. 20

A.1.3. More results and Visualization

This section visualizes RoboInter-VLA’s real-world experimental process and highlights precise action generation alongside long-horizon manipulation capabilities.

Real-world evaluation

  • Real-world evaluation demonstrates precise action generation and long-horizon behavior, including collision-avoiding pen grasping and continuous board cleaning.The accompanying visualizations cover the real-world experimental process in Figures 7, 8, and 9.Source: §A.1.3 · p. 21, Figure 9 · p. 22

A.2. Additional Ablation Experiments for RoboInter-VLA · A.2.1. Inference Time Analysis.

RoboInter-VLA’s explicit intermediate reasoning improves open-loop accuracy and robustness, but autoregressive chain-of-thought generation increases inference latency. Practical acceleration strategies reduce deployment latency while preserving strong in-distribution and out-of-distribution performance.

A.2.1. Inference Time Analysis.

  • Explicit reasoning improves open-loop accuracy for both VLA-OS (Gao et al., 2025a) and RoboInter-VLA, while increasing latency from autoregressive CoT generation.This comparison is made under comparable model sizes.Source: §A.2.1 · p. 23, Table 7 · p. 23
  • Real-world acceleration strategies substantially reduce inference latency while preserving strong ID and OOD performance.The strategies include textual caching and chunked execution for EC-E2E, plus asynchronous dual-frequency execution for the Modular variant.Source: §A.2.1 · p. 23, Table 8 · p. 23
  • Textual caching and chunked execution are applied to the EC-E2E variant for practical deployment acceleration.These strategies target the latency introduced by explicit reasoning.Source: §A.2.1 · p. 23
  • Asynchronous dual-frequency execution is applied to the Modular variant to accelerate real-world inference.The strategy preserves strong ID and OOD performance according to the real-world evaluation.Source: §A.2.1 · p. 23, Table 8 · p. 23
  • The results show a trade-off in which explicit intermediate reasoning enhances robustness at the cost of higher inference time.This trade-off motivates future work on more efficient execution.Source: §A.2.1 · p. 23, Table 7 · p. 23, Table 8 · p. 23

A.2.2. Experiment Results for Data Scaling Law

RoboInter-VQA performance follows a scaling-law trend as annotated-data volume increases. In F-CoT experiments, both evaluated models improve with more data, but Oracle+Executor benefits more than RoboInter-Te-Modular.

RoboInter-VQA data scaling law

  • RoboInter-VQA models follow a scaling-law trend as the annotated-data proportion increases from 10% to 100%.Experiments use QwenVL2.5-3B with general VQA data combined with corresponding subsets of annotated data across visual generation, understanding, and planning tasks.Source: §A.2.2 · p. 23, Table 9 · p. 23, Table 10 · p. 24

F-CoT data scaling law

  • Oracle+Executor improves from 0.2347 to 0.2705, exceeding RoboInter-Te-Modular’s gain from 0.2109 to 0.2332 as F-CoT data increases.Both models improve with higher sampling ratios, but the executor conditioned directly on oracle intermediate signals benefits more readily, while the VLM Planner improves more slowly.Source: §A.2.2 · pp. 23–24, Table 12 · p. 24

A.2.3. Ablation for Designs and Intermediate Representations Types of F-CoT … A.8. Limitations and LLM Usage Statement

This section details RoboInter’s annotation pipelines, dataset and VQA construction, benchmark protocols, and evaluation procedures for intermediate-representation-based manipulation learning. It also presents task formats and F-CoT serialization alongside ablations, case studies, and documented limitations.

A.2.3. Ablation for Designs and Intermediate Representations Types of F-CoT

  • Textual F-CoT consistently performs best among learning-based designs, while spatially precise Object Box and Affordance cues improve performance more than coarse Subtask and Primitive skill representations.The design comparison uses Textual F-CoT in RoboInter-Te-Modular versus Visual F-CoT in RoboInter-Im-Modular; representation ablations use the Oracle+Executor setting.Source: §A.2.3 · p. 24

A.3.1. Open-loop Cross-platform Evaluation

  • All RoboInterVLA variants outperform the vanilla baseline across five robot embodiments, with Modular achieving the best overall accuracy among learned models and Oracle+Executor providing an upper bound.The open-loop evaluation exploits diverse platforms in both training and validation splits as a cross-platform test.Source: §A.3.1 · p. 27, Table 18 · p. 27

A.3.2. Close-loop Evaluation on SimplerEnv

  • On SimplerEnv, RoboInter-IC-E2E achieves 65.7 success rate, surpassing CogACT’s 63.8, while the Vanilla design scores 61.8 versus CogACT’s 63.8, in cross-embodiment evaluation.RoboInter-Data lacks action annotations for WidowX or Google robots, making this a strictly cross-embodiment evaluation.Source: §A.3.2 · p. 28

A.3.3. Close-loop Evaluation on Real-world WidowX Robot

  • RoboInter-IC-E2E demonstrates the strongest real-world performance across four repeated kitchen manipulation tasks on the WidowX-250, including out-of-distribution Pikachu-object tests.All models were pretrained on BridgeV2 without post-training or fine-tuning before deployment, using the same real-world setup.Source: §A.3.3 · p. 28

A.4.3. Metric Details.

  • The evaluation protocol uses Open-Loop Score over independently assessed continuous action chunks, with a validation set exceeding 100K transitions, while keypoint alignment reconstructs 3D gripper geometry and projects it into images.Keypoint poses are propagated through updated URDFs, camera intrinsics and extrinsics, and end-effector offset calibration to reduce projection deviation.Source: §A.4.3 · p. 30, §A.6.4 · p. 38

A.5. Case Study

  • RoboInterVLM consistently outperforms RoboBrain and Qwen2.5VL across case-study settings and maintains strong zero-shot generalization on novel objects, supporting intermediate-level reasoning and data selection.The qualitative evaluations include RoboInter-VQA examples and a self-collected zero-shot dataset for object grounding and grasp affordance tasks.Source: §A.5 · p. 31, Figure 12 · p. 31, Figure 13 · p. 32

A.6.1. RoboInter-Tool

  • RoboInter-Tool provides a PyQt5 multimodal platform for video-, clip-, and frame-level annotation, integrating segmentation, language labeling, visualization, editing, and asynchronous quality verification.Its interfaces support gripper keypoints, SAM2-assisted object masks, contact frames, keyframe selection, structured language descriptions, and pre-annotation refinement.Source: §A.6.1 · pp. 33–34

A.6.2. Principle of Semantic Segmentation Annotation

  • The annotation principles combine pixel-level object masks, clip-level contact frames, geometric gripper keypoints, and exhaustive language segmentation into subtasks and 15 parameterized primitive skills.SAM2 generates masks from annotator prompts, while keypoint annotations capture visible gripper geometry and language annotations span global descriptions, clips, and canonical behaviors such as Pick and Place.Source: §A.6.2 · pp. 34–35, §A.6.3 · p. 35, Table 20 · p. 36

A.6.4. Keypoint Alignment and Annotation Pipeline

  • Reprojection error decreases from 46.25 to 5.31 on cfg.1 and from 108.21 to 11.61 on cfg.4 after offset optimization, improving 3D–2D consistency.The method manually annotates 2D keypoints, optimizes a 3D EE-gripper translation by minimizing projected-point error, and integrates the calibrated offset into the URDF.Source: §A.6.4 · p. 39, Table 21 · p. 39

A.6.5. Quality Control Pipeline

  • The quality-control pipeline combines SAM-assisted mask review, cross-validation, and subset sampling, while language annotations add ChatGPT drafts and repeat repair cycles until subsets exceed 90% acceptance.Semantic-mask subsets below 90% were re-annotated once, whereas language subsets required four repair-and-verification iterations.Source: §A.6.5 · p. 40

A.6.6. Postprocessing

  • DROID postprocessing corrects spatial offsets using object centroids and TCP projections, then synchronizes trajectories with video by matching gripper-motion onset frames using CoTracker3 (Karaev et al., 2024).The corrected trajectories are uniformly shifted when eligible, then padded or truncated to ensure frame-level correspondence.Source: §A.6.6 · p. 40, Figure 19 · p. 41

A.7.1. Dataset Statistics

  • The dataset spans video-, frame-, and local-region annotations across diverse embodied scenarios, supporting both generation and understanding in spatial and temporal domains.Table 22 summarizes the raw dataset used for annotation, including global descriptions, subtasks, keyframe counts, object masks, and gripper keypoints.Source: §A.7.1 · p. 42, Table 22 · p. 42

A.7.2. Visualization for Intermediate Representations

  • Figures 20–23 visualize contact points and boxes, gripper boxes, object masks and boxes, and temporally colored traces, demonstrating annotation diversity and labeling fidelity.The visualizations cover multi-task episodes and annotate corresponding subtasks beneath the representations.Source: §A.7.2 · p. 42, Figure 20 · p. 43, Figure 21 · p. 44

A.7.3. Details of VQA Dataset

  • The VQA dataset analyzes language, object distribution, subtask lengths, task complexity, skills, embodiments, and spatial tasks, and details 18 temporal VQA types for sequential understanding and decision-making.Spatial tasks include grasp pose choice, grounding choice, contact decisions, and scene understanding; statistics are reported in Tables 23–24.Source: §A.7.3 · pp. 47, 49, 53, Table 23 · p. 48, Table 24 · p. 48
  • The VQA suite tests trajectory selection, direction prediction, and trajectory-to-subtask matching using gradient-colored paths and candidate choices.Green marks trajectory starts and red marks endpoints; direction tasks instead present multiple colored arrows around the gripper.Source: §A.7.3 · p. 54
  • It evaluates affordance reasoning by asking models to confirm feasible next tasks, reject infeasible alternatives, or generate the most practical immediate action.Discriminative affordance prompts require strict Yes/No answers, while generative affordance prompts request a short phrase.Source: §A.7.3 · pp. 55–56
  • Temporal tasks probe past-action recall, success verification, future-event prediction, scene-state grounding, and selection of completed or upcoming subtasks and primitive skills.They include identifying the most recent task or primitive, confirming success or failure with Yes/No, choosing execution-state images, and selecting past or future actions from options.Source: §A.7.3 · pp. 54, 56
  • Planning tasks assess immediate next-step reasoning, context-aware planning, and generation of the next five sequential actions toward a long-horizon goal.Prompts either request a short phrase or sequential phrases after specifying completed subtasks and the overarching objective.Source: §A.7.3 · p. 55
  • A.7.3. Details of VQA Dataset: The VQA dataset covers discriminative and generative tasks for primitive, multi-task, planning, scene-understanding, and temporal reasoning.Examples include selecting future or past tasks, predicting affordances, describing completed actions, and planning remaining steps.Source: Figure 31 · p. 60, Figure 32 · p. 61, Figure 33 · p. 62

A.7.4. CoT Data for RoboInter-VLA

  • A.7.4. CoT Data for RoboInter-VLA: Flexible Chain-of-Thought data trains Explicitly-Conditioned E2E and Modular Planner-to-Executor settings and is serialized as textual sequences.The data has a representative visualization and corresponding textual chain-of-thought representations.Source: §A.7.4 · p. 68, Figure 39 · p. 68, Figure 40 · p. 68

A.8. Limitations and LLM Usage Statement

  • A.8. Limitations and LLM Usage Statement: Annotations primarily cover fixed-base, single-arm robots, while planner robustness, executor inference efficiency, and cross-embodiment generalization remain limitations; LLMs supported only linguistic polishing.The authors state that ideas, experiments, analysis, and conclusions were developed without LLM assistance, and that all edits were reviewed for fidelity.Source: §A.8 · p. 69

Related papers

Loading 2602.09973v1…