Source-linked AI summary
RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
Pierre Sermanet, Tianli Ding, Jeffrey Zhao, Fei Xia, Debidatta Dwibedi, Keerthana Gopalakrishnan, Christine Chan, Gabriel Dulac-Arnold, Sharath Maddineni, Nikhil J Joshi, Pete Florence, Wei Han, Robert Baruch, Yao Lu, Suvir Mirchandani, Peng Xu, Pannag Sanketi, Karol Hausman, Izhak Shafran, Brian Ichter, Yuan Cao
TL;DR
Robotics VLMs still lack sufficient grounded multimodal data for accurate real-world high-level reasoning. RoboVQA addresses this gap with scalable, diverse cross-embodiment collection and a video-conditioned model, achieving stronger grounded performance and supporting intervention-based long-horizon robot control. The authors also identify a scope limitation in their comparisons with human-only datasets.
Problem
State-of-the-art VLMs remain insufficiently accurate for grounded high-level reasoning in realistic robotic settings, motivating more grounded data collection.
Method
RoboVQA uses bottom-up collection of user-requested long- and medium-horizon tasks across robot and human embodiments, then trains RoboVQA-VideoCoCa and evaluates intervention-based planning.
Results
RoboVQA-VideoCoCa outperforms the reported zero-shot state-of-the-art baseline by 46% in intervention rate, while mixed human-and-robot data performs comparably to robot-only data on robot tests.
Takeaways & Limitations
Scalable human and robot collection can expand grounded robotics data, and intervention provides a single metric while enabling tasks to complete under human oversight.
Takeaways & Limitations
The study does not compare its human-and-robot dataset or benchmarks with human-only datasets such as Ego4D or EpicKitchens.
Abstract
from arXiv · showhide
We present a scalable, bottom-up and intrinsically diverse data collection scheme that can be used for high-level reasoning with long and medium horizons and that has 2.2x higher throughput compared to traditional narrow top-down step-by-step collection. We collect realistic data by performing any user requests within the entirety of 3 office buildings and using multiple robot and human embodiments. With this data, we show that models trained on all embodiments perform better than ones trained on the robot data only, even when evaluated solely on robot episodes. We find that for a fixed collection budget it is beneficial to take advantage of cheaper human collection along with robot collection. We release a large and highly diverse (29,520 unique instructions) dataset dubbed RoboVQA containing 829,502 (video, text) pairs for robotics-focused visual question answering. We also demonstrate how evaluating real robot experiments with an intervention mechanism enables performing tasks to completion, making it deployable with human oversight even if imperfect while also providing a single performance metric. We demonstrate a single video-conditioned model named RoboVQA-VideoCoCa trained on our dataset that is capable of performing a variety of grounded high-level reasoning tasks in broad realistic settings with a cognitive intervention rate 46% lower than the zero-shot state of the art visual language model (VLM) baseline and is able to guide real robots through long-horizon tasks. The performance gap with zero-shot state-of-the-art models indicates that a lot of grounded data remains to be collected for real-world deployment, emphasizing the critical need for scalable data collection approaches. Finally, we show that video VLMs significantly outperform single-image VLMs with an average error rate reduction of 19% across all VQA tasks. Data and videos available at https://robovqa.github.io
I. INTRODUCTION
RoboVQA introduces a scalable bottom-up collection process for diverse long- and medium-horizon grounded reasoning data. The approach combines continuous robot and human-embodiment collection with hindsight labeling and automatic VQA generation.
- Data collection: 2.2x higher throughput than traditional top-down collection supports scalable long- and medium-horizon reasoning data.The gains arise from reducing resets and scene preparation while collecting continuous episodes.
- Data collection: User-requested long-horizon tasks are teleoperated, segmented and labeled in hindsight, then converted into 10 types of question/answer pairs.Crowd-sourced temporal labeling produces medium-horizon instructions from coherent task sequences.
- Dataset diversity: The dataset spans three office buildings and three embodiments: robot, human single arm, and human grasping tool.The collection includes realistic user requests and multiple embodiment types.
- Dataset diversity: Bottom-up crowd-sourced tasks provide broader, more user-aligned diversity than fixed researcher-defined task lists.The process yields naturally diverse medium-horizon steps and coherent long-horizon sequences for planning.
- Reasoning formulation: The dataset represents high-level planning as sequences of natural-language VQA steps resembling chain-of-thought.The formulation connects medium-horizon task decomposition with long-horizon planning.
- Evaluation motivation: Video conditioning substantially improves VQA performance over image conditioning, while zero-shot VLMs remain weak in realistic settings.These observations motivate further scalable collection of grounded multimodal data.
III. MODELS
RoboVQA-VideoCoCa is a video-language model fine-tuned on RoboVQA to study video-based visual question answering. Its architecture combines contrastive and generative video-text pretraining.
- RoboVQA-VideoCoCa: RoboVQA-VideoCoCa is derived from VideoCoCa and fine-tuned on RoboVQA video-text datasets.The base model has 383M parameters and starts from an image-captioning checkpoint.
- RoboVQA-VideoCoCa: The model combines contrastive pretraining between video and text with generative pretraining.This encoder-decoder design extends CoCa across video and text modalities.
B. Baselines
The evaluation compares RoboVQA-VideoCoCa with zero-shot and planning baselines using human-rated VQA and intervention-based long-horizon planning. Results show a substantial gap between fine-tuned and zero-shot models in realistic settings.
- VQA baselines: PaLM-E-562B is evaluated zero-shot without training on RoboVQA, while RoboVQA-VideoCoCa is fine-tuned on the dataset.The comparison assesses how prior-dataset state-of-the-art models perform in the real world.
- Planning baselines: SayCan and Grounded Decoding use a text-only LLM guided by visual affordances, alongside end-to-end RoboVQA-VideoCoCa and PaLM-E planners.The adapted baselines decode at phrase or token level using RoboVQA-VideoCoCa as a video value function.
- Evaluation: Human raters evaluate model answers against stored exact-match results, with unmatched answers sent for additional rating.This procedure supports error-rate evaluation on individual video-question tasks.
- Planning results: Human intervention keeps all long-horizon tasks completed even when the model makes mistakes.The intervention benchmark evaluates planning steps in dataset episodes, live teleoperation, and end-to-end robot control.
- VQA results: Zero-shot state-of-the-art models retain a substantial error-rate gap relative to the fine-tuned model in realistic settings.The result motivates additional scalable data collection for real-world deployment.
B. Planning Benchmark with Intervention
The benchmark evaluates long-horizon planning through human-graded medium-horizon plans and intervention rates, combining offline episodes with live robot settings. Results show that broad task training improves planning and that the model outperforms the state-of-the-art baseline despite being much smaller.
- Intervention evaluation: Intervention rate averages cognitive and physical intervention rates into one metric for long-horizon planning evaluation.The decomposition separates high-level text reasoning from low-level motor-command intervention.
- Live evaluation: Live evaluation supplies real-time medium-horizon plans to a real robot teleoperated by a human, while another evaluation deploys a policy in a narrower domain.The narrower policy setting obtains lower intervention rates but does not cover the broad domain of the teleoperated evaluation.
- Grounded chat: Grounded chat lets an operator validate or correct plans while the user can ask questions during execution.In the illustrated episode, the system reports a 12.5% cognitive intervention rate.
- Task augmentation: 70.9% error for all-task training improves on 77.2% error for planning-only training on the robot-plus-human dataset.The all-task model uses roughly one-eighth as many planning examples as the planning-only model.
B. Tasks Transfer via Cross-Embodiment Data
Cross-embodiment training transfers useful grounded understanding across human and robot data, while video conditioning improves temporal reasoning and confidence-filtered value functions support success detection. Training on both embodiments is strongest across the reported evaluation settings.
- Cross-embodiment transfer: Training on both embodiments performs better than robot-only training even when the test set contains only robot data.Human-only training also yields errors below 100% across all tasks, indicating that human data can support grounded understanding of robot videos.
- Video conditioning: More frames substantially reduce error between one and sixteen frames by capturing longer temporal dynamics for visual grounding.The model is trained with 1, 2, 4, 8, and 16 frames.
- Video value-functions: Confidence filtering improves the accuracy of RoboVQA-VideoCoCa's stable binary success detections.These video value functions can support closed-loop planning by indicating when a step has been performed.
- Video value-functions: Figure 8 displays raw answers in blue, confidence in red, and confidence-filtered answers in green for video success detection.The example asks whether placing a purple marker on the table is satisfied.
- Video value-functions: Video-conditioned value functions provide richer understanding than traditional image-based success or affordance detectors.The passage attributes this to dataset breadth and video conditioning.
VI. RELATED WORK
RoboVQA builds on video-language modeling and video captioning while distinguishing its egocentric, segmented, question-augmented data. It also frames intervention rate as both an evaluation metric and a deployment mechanism, while noting an unresolved comparison with human-only benchmarks.
- Vision-Language Models: VideoCoCa is fine-tuned on conversations grounded in long-horizon egocentric videos, whose full-video input supports fine-grained temporal reasoning.RoboVQA further adds segment labels and question-answer pairs for long-horizon tasks executed by robots or humans.
- Video Captioning: RoboVQA differs from video captioning by using egocentric videos, temporal segment labels, and diverse question-answer supervision for task execution.
- Intervention Rate: Intervention rate measures policy performance and enables task completion, which the paper identifies as necessary for real-world deployment.
- Limitations: The authors filtered episodes with more than 5 identical medium-horizon steps and did not compare their human-and-robot benchmark with human-only datasets.They identify comparisons with datasets such as Ego4D and EpicKitchens as future work.
- Conclusion: The paper releases the dataset and establishes real-robot planning benchmarks using intervention as both a metric and a deployment mechanism.
IX. APPENDIX
Figure 9 provides randomly selected visual examples from the training set. Its purpose is to illustrate the dataset’s training imagery rather than report a quantitative result.
- Training-set examples: Figure 9 presents random frames from the training set.
- Training-set examples: The figure samples visual observations from the dataset’s training portion.
- Training-set examples: The figure serves as a visual illustration of the training data.
B. Dataset Statistics
The dataset contains 5,246 long-horizon episodes, with 5,046 for training and 100 for validation, and substantial variation in instruction repetition. Nearly half of test episodes lack a matching long-horizon instruction in training.
- Dataset composition: 5,246 long-horizon episodes comprise the dataset, including 5,046 training and 100 validation episodes.
- Instruction repetition: Each long-horizon instruction has an average of 2.01 episodes, a median of 1, and a maximum of 90.
- Instruction overlap: 46% of test episodes have no matching long-horizon instruction in the training set.
- Episode structure: Each episode contains one long-horizon instruction and a varying number of temporally segmented medium-horizon instructions.
C. Comparing Embodiment Mixtures
The embodiment-mixture analysis asks which robot-human data mixture minimizes error under a fixed collection budget. Across tested cost settings, adding human data preserves comparable robot performance while substantially improving human-test performance.
- Fixed-budget comparison: 500,000 VQA conversations provide the fixed-budget basis for comparing robot-human data mixtures.
- Equal-cost mixture: 62.4% robot-test error for equal robot250k-human250k training is comparable to 62.7% for the full 500k robot dataset.
- Equal-cost mixture: 53.9% human-test error for the equal robot250k-human250k mixture is lower than 67.0% for the full 500k robot dataset.
- Cost-ratio comparison: At a 4.0 robot-human cost ratio, robot-62k plus human-250k yields 65.3% robot-test error versus 63.5% for robot-only 125k.
- Cost-ratio comparison: At the same 4.0 cost ratio, the mixed dataset achieves 51.1% human-test error versus 68.7% for robot-only 125k.
- Implication: The analysis concludes that human collection efficiently scales robot data despite embodiment differences.
D. Instructions Samples
The instructions span short, medium, and long horizons, combining simple object placements with multistep retrieval, sorting, cleaning, and relocation tasks. The accompanying figures compare embodiment mixtures and video-model frame counts, including robot-only evaluation and 16-frame inputs.
- Instruction examples: Long-horizon requests include chained actions such as retrieving an object, transporting it, and completing a final placement or disposal step.Examples include retrieving a mouse box from a filing cabinet, moving it to a table, and taking an item to a specified room or locker.
- Evaluation figures: Cross-embodiment training improves performance on the robot-only test set, while training on both embodiments provides the broadest capabilities across robot and human tests.The figures compare models trained on robot-only, human-only, and combined data on robot-only and combined test sets.
- Instruction examples: Medium-horizon samples include atomic movements and placements alongside short action sequences involving objects, tables, trays, bowls, and containers.The examples include touching or approaching objects, placing items into containers, and moving a bin, pens, and other objects through several steps.
- Evaluation figures: 16 input frames produce the best results for the video model, which is trained on 875k robot-and-human samples and evaluated on the combined test set.A separate figure compares error reductions when using 16 frames rather than 1.
E. Dataset Language Statistics Analysis by LLM
An LLM-based analysis organizes the short-horizon instruction language into object, action, location, and attribute categories. The extracted lists are broad and intentionally retain redundant descriptions rather than representing unique instances.
- Extracted categories: The analysis identifies 1795 objects, including descriptions such as “tissue box” and “purple color plate.”
- Extracted categories: The analysis identifies 494 actions, including “add something into something” and “go out of a room.”
- Extracted categories: The analysis identifies 2064 locations, including “in the green plate” and “left trash can.”
- Extracted categories: The analysis identifies 462 attributes, including shapes and colors, while retaining redundant descriptions without clustering them into unique instances.