PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboReason: Boosting Robotics-Relevant VLM Reasoning Through Scalable Data Generation

Sagnik Anupam, Utkarsh Lal, Aneesh Edara, Botong Zhang, Yao Tang, Jiatao Gu, Dinesh Jayaraman, Osbert Bastani

arXiv:2610.33778v1cs.ROcs.AIcs.CV

TL;DR

VLMs are difficult to evaluate at scale on real-world robotics tasks, where spatiotemporal reasoning must handle cluttered, robot-centric views and relationships across images. RoboReason evaluates ordering tasks from robot trajectories and generates training data for finetuning smaller models. Leading VLMs struggle on these tasks, while generated-data training improves ordering and downstream reasoning performance, with gains varying by task.

  • Problem

    Realistic robotics evaluation of VLM reasoning is difficult to scale, and general spatiotemporal settings do not capture robot-view clutter, cross-frame relationships, and multiple viewpoints.

    Source: §1 · p. 1

  • Method

    RoboReason evaluates spatiotemporal ordering on robot-manipulation trajectories and uses a scalable pipeline to generate data for supervised finetuning and reinforcement learning.

    Source: §1 · p. 1, §2.1 · p. 3

  • Results

    Leading VLMs struggle as ordering sets grow, while generated-data training improves smaller models on ordering and downstream tasks; GPT-5 and Gemini-ER-1.5 achieve no more than 80% accuracy on two-object ordering.

    Source: §1 · pp. 1–3

  • Takeaways & Limitations

    Generated-data training can improve smaller VLMs on spatiotemporal ordering and downstream spatial and robotics reasoning tasks, but gains vary across tasks and approaches.

    Source: §1 · pp. 2–3

    Depth and left-to-right task labels can be erroneous because of module failures, and object selection can leave overlaps or near-equal camera distances ambiguous.

    Source: §4 · p. 9

Abstract

from arXiv · show

Vision-Language Models (VLMs) have demonstrated impressive performance when integrated into robot manipulation pipelines, but real-world evaluation of VLM capabilities on downstream robotics tasks is both difficult and expensive to scale. We propose that spatial and temporal ordering tasks using images from stereo and wrist cameras attached to real-world robots are particularly well-suited for evaluating visual reasoning capabilities, as they assess skills directly applicable to robots. We evaluate VLM reasoning abilities on three questions: ordering shuffled frames conditioned on a language description of the task, ordering objects depthwise based on distance from the camera, and ordering objects from left-to-right in pixel coordinates. The complexity of these tasks can also be modulated by changing the size of the set to sort. Our experiments show that several state-of-the-art vision-language models struggle on ordering tasks. To fix this, we propose a scalable autonomous training data generation pipeline, and show that both supervised finetuning and reinforcement-learning on the generated data can significantly improve VLM reasoning performance both on spatiotemporal ordering tasks as well as downstream robotics and visual reasoning tasks.

1 Introduction

Robot-centric spatiotemporal reasoning remains challenging because robotics images include cluttered scenes, cross-frame relationships, and multiple viewpoints. RoboReason evaluates ordering on robot-manipulation images and uses generated training data to improve VLM performance, including on downstream tasks.

1 Introduction

  • Robot manipulation requires reasoning about object relationships within and across images, while general spatiotemporal settings may miss robot-view clutter, cross-frame relations, and multiple viewpoints.RoboReason evaluates realistic ordering tasks and generates training data from robot manipulation trajectories.Source: §1 · p. 1
  • Training smaller VLMs on pipeline-generated data improves ordering and downstream reasoning, although no single approach improves every downstream task.The pipeline is scaled to generate 8,857 samples for each ordering task.Source: §1 · pp. 1–2
  • GPT-5 and Gemini-ER-1.5 achieve no more than 80% accuracy on two-object ordering, while leading VLM performance falls as the number of items increases.These models struggle on real-world robot manipulation images despite stronger reported results on other spatial reasoning benchmarks.Source: §1 · p. 2
  • Providing segmentation maps or bounding boxes from specialized vision models mostly improves left-to-right and depth-wise spatial ordering accuracy.The results suggest VLMs struggle to identify objects in cluttered environments compared with specialized vision models.Source: §1 · p. 2
  • Merge sort over VLM-generated pairwise comparisons improves spatial ordering, but performs worse or comparably to one-shot input on temporal ordering.The authors interpret the temporal results as evidence that VLMs can reason over more than two frames.Source: §1 · p. 2
  • GRPO yields large gains on spatial ordering but limited gains on temporal ordering; Qwen2-VL surpasses larger VLMs on some spatial tasks.Performance also increases with the size of the generated training dataset.Source: §1 · pp. 2–3

2 RoboReason

RoboReason formulates temporal, depthwise, and left-to-right ordering as tasks for VLMs, then generates robot-image training data and uses it to train models with reinforcement learning.

2 RoboReason

  • The three tasks ask a VLM to restore a shuffled ordering, using timestamps for frames, camera distance for depth, and horizontal image position for left-to-right order.Temporal frames combine left, right, and wrist camera images in the canonical setting.Source: §2 · p. 3

2.1 Data Generation Pipeline

  • The pipeline samples frames from DROID trajectories for temporal ordering, using timestamp order to measure task progression.Five trajectory segments supply frames; the three-image setting uses frames 1, 3, and 5.Source: §2.1 · p. 3
  • Depth ordering uses selected left-camera frames, identified objects, and estimated object depths to establish the ground-truth order.Scenes with fewer than five sufficiently distinct objects are filtered out; object depths are averaged over segmentation-mask pixels.Source: §2.1 · pp. 3–4
  • Left-to-right ordering filters object detections by size, alignment, and overlap, then orders valid objects by bounding-box centroid x-coordinate.Trajectories with fewer than five valid objects are discarded.Source: §2.1 · p. 4

2.2 Reinforcement Learning Finetuning

  • For reinforcement-learning finetuning, GRPO trains two small VLMs with Kendall rank correlation between predicted and ground-truth orderings as the reward.The experiments use pipeline-generated data across all three ordering tasks.Source: §2.2 · p. 4

3 Experiments

The experiments assess VLM ordering performance and RoboReason-generated training data. Finetuning improves ordering and downstream task performance, with gains varying by task and training-data amount.

3.1 Experimental Setup

  • The experiments compare VLM reasoning levels and ordering formats, using token limits calibrated to GPT-5 and human-validated test sets for ordering evaluations.Accuracy requires an exact match to the ground-truth order; each small-data task uses 300 test samples.Source: §3.1 · pp. 4–5

3.2 Zero-Shot VLM Performance

  • Bounding-box and segmentation overlays give comparable or better depth-ordering performance, while bounding boxes outperform segmentation maps in left-to-right ordering.Overlays also increase mean reasoning differences for nearly all models, with standard deviations slightly lower for all but Qwen3-VL-235B.Source: Table 2 · p. 5, Table 3 · p. 6, §3.2 · p. 7
  • Merge-sort comparisons usually match or improve spatial ordering against presenting all objects at once, while temporal ordering performs better with all frames together.The reported comparison is for the high-reasoning setting.Source: Table 6 · p. 5, §3.2 · p. 6
  • Ordering accuracy declines sharply as set cardinality grows, and models’ relative performance can change across cardinalities and settings.In temporal high reasoning, the relative order remains consistent; in other Figure 2 settings it is inconsistent.Source: Table 4 · p. 6, §3.2 · p. 6, Figure 2 · p. 8

3.3 RoboReason Evaluations

  • Finetuning on pipeline-generated data improves small-model performance on ordering and downstream robotics and visual-reasoning tasks, despite substantial disagreement between generated orders and human assessments.Human review found at least one incorrectly positioned object in 15.67% of left-to-right samples and 26.67% of depth samples.Source: §3.3 · p. 7
  • Small-data finetuning improves base-setting ordering, with GRPO generally outperforming SFT and Qwen2-VL-2B surpassing larger VLMs on three-object depth and left-to-right tasks.Performance also scales with training-data size; the table reports GRPO and SFT accuracies.Source: §3.3 · pp. 7–8, Table 8 · p. 8, Table 9 · p. 8

4 Conclusion

RoboReason evaluates robotics-relevant spatiotemporal ordering using real-world robot images, and the authors report that finetuning smaller models can surpass much larger VLMs on spatial ordering while improving downstream reasoning. The related-work discussion distinguishes this setting from prior benchmarks by its robot-manipulation scenes, whole-scene ordering, and retrievable trajectory-based ground truth.

4 Conclusion

  • Finetuning smaller VLMs on RoboReason-generated data can surpass much larger VLMs on spatial ordering tasks and improve downstream reasoning performance.The conclusion describes these as findings from experiments on the generated data.Source: §4 · p. 8, Table 7 · p. 9
  • Depth and left-to-right ground truths can be erroneous because of failures in object detection, bounding-box generation, or depth estimation.Filtering can also select overlapping objects or objects equidistant from the camera, while some temporal descriptions leave motion direction ambiguous.Source: §4 · p. 9

A Related Works

  • Existing DROID-based spatial tasks focus on gripper-object interactions, whereas RoboReason targets spatiotemporal ordering of the whole scene from robot viewpoints.The authors describe existing ground truths as sourced from underlying gripper data and their own as constructed by analyzing trajectory images.Source: A Related Works · p. 12
  • Prior visual-question-answering benchmarks often use pairwise depth comparisons or pointing questions, and many use internet-sourced images unlike real-world robot-manipulation scenes.The authors note that pointing tasks combine reasoning with the ability to point reliably in image space.Source: A Related Works · pp. 12–13
  • Prior temporal-ordering approaches had not evaluated real-world robot-manipulation videos and images, in part because retrieving accurate ground truth from low-resolution robot-camera images is difficult.RoboReason instead uses ordering tasks with ground truth retrievable from trajectory video or traditional computer-vision models.Source: A Related Works · p. 13

B Downstream Task Evaluations

The downstream evaluations compare finetuned Qwen3-VL-8B models trained with SFT and GRPO on 8.8K samples of each task type. Evaluation spans visual reasoning, robotics failure reasoning, and trajectory value estimation.

B Downstream Task Evaluations

  • Qwen3-VL-8B was trained with SFT and GRPO on 8.8K task samples of each type to evaluate finetuning on RoboReason-generated data.The training used task-specific checkpoints and seeds; temporal training used the 3-image prediction task, while depth and left-to-right training used k-object ordering.Source: B Downstream Task Evaluations · p. 13
  • Downstream evaluation covered 150 SAT questions, 1.9K multiple-choice and 5.9K open-ended RoboFAC questions, and 1,000 trajectories each from RT-1 and Dobb-E.Open-ended RoboFAC accuracy used exact string match.Source: B Downstream Task Evaluations · p. 13
  • SAT and RoboFAC performance was measured by question accuracy, while trajectory value estimation used absolute average Value-Order Correlation.VOC is the rank correlation between predicted values and original chronological order; unsuccessful rankings received VOC 0.Source: B Downstream Task Evaluations · p. 13

C Impact of Camera View on Temporal Reasoning

The canonical temporal-ordering setup supplies all three DROID camera views because frame differences can be obscured in particular views. In the three-image task, single-view performance was generally better than or comparable to using a composite of multiple views.

C Impact of Camera View on Temporal Reasoning

  • The canonical temporal task supplies all three camera views because subtle frame differences useful for human evaluation can be occluded in particular views.This differs from the depth and left-to-right tasks, which do not use multiple camera views in the canonical setting.Source: C Impact of Camera View on Temporal… · pp. 13–14
  • For all models and views except GPT-5-Nano and Qwen3-VL-235B on the right view, single-view temporal ordering performed better than or within 2% of the multi-view composite.The comparison used the high-reasoning setting on the three-image base task.Source: Table 10 · p. 13, C Impact of Camera View on Temporal… · p. 14
  • The authors interpret the single-view results as evidence that VLMs struggle to synthesize information across views during temporal reasoning.They note that this pattern is consistent with results on MindCube.Source: C Impact of Camera View on Temporal… · p. 14

D Spatiotemporal Ordering Task Construction and Prompts

This appendix section describes the construction and prompting of robotics-relevant ordering evaluations, including DROID image filtering and pairwise merge-sort comparisons. It also covers varying ordering-set size and points to reinforcement-learning experiment details in the following section.

D Spatiotemporal Ordering Task Construction and Prompts

  • The evaluation covers temporal frame order, closest-to-farthest depth order, and left-to-right pixel order, using base images and, where specified, segmentation-map or bounding-box overlays.Temporal settings include 3- and 5-image tests; depth and left-to-right settings include 3- and 5-object tests, each across low, medium, and high reasoning levels.Source: D Spatiotemporal Ordering Task… · p. 14

3/5-image Prompt for the Merge Sort Setting

D.1.1 Temporal Filtering Algorithm for DROID

  • Temporal examples are sampled from DROID robot-task trajectories, excluding empty, undescribed, or “No action” sequences and saving frame images with sequence metadata.The filtering procedure selects target indices, extracts episode steps, and records each retained trajectory’s task description and length.Source: §D.1 · p. 14, §D.1.1 · pp. 14–15

n-image Prompt for GPT models; n-image Prompt for non-GPT models

n-image Prompt for VLMs in the Segmentation Map and Bounding Box Settings; n-object Prompt for VLMs in the Segmentation Map and Bounding Box Settings

D.2 Depth Ordering Task; D.2.1 Depth Filtering Algorithm for DROID

  • Depth examples retain scenes with enough non-overlapping objects, detect and segment objects, estimate image depth, and sort retained objects from closest to farthest.The filtering procedure uses Grounding DINO for object detection and Marigold for depth estimation, then rejects overlapping masks.Source: §D.2 · p. 15, §D.2.1 · p. 16

General VLM Prompt; n-object Prompt for GPT Models; n-object Prompt for non-GPT models; 3/5-object Prompt for the Merge Sort Setting

3/5-object Prompt for the Merge Sort Setting

  • Depth merge-sort prompts reduce ordering to a pairwise decision, requiring the model to choose only one of the two exact object names.The prompt prohibits name variations or invented labels.Source: 3/5-object Prompt for the Merge Sort… · p. 17
  • The depth-comparison prompt requires a JSON object naming the closer object, with confidence from 0.0 to 1.0 and a brief explanation.The chosen object must exactly match one of the two supplied names.Source: 3/5-object Prompt for the Merge Sort… · p. 17

D.3 Left-to-Right Ordering Task

D.3.1 Left-to-right Filtering Algorithm for DROID

  • The DROID filtering procedure samples sequences and selects a random image from each sampled sequence for left-to-right ordering data.Source: §D.3.1 · p. 17
  • The ordering filter sorts objects by left edge, keeps a vertically aligned cluster, removes highly overlapping boxes and duplicate labels, and rejects insufficient sets.The prompt parameters specify a minimum horizontal gap of 0.05 and maximum vertical distance of 0.5.Source: §D.3.1 · pp. 17–18
  • It detects categories with GPT-5, uses those labels as Grounding DINO text prompts, and rejects samples with fewer than the minimum object count.Source: §D.3.1 · pp. 17–18

n-object Prompt for all VLMs

  • For 2-, 3-, 4-, and 5-object cases, VLMs receive randomly shuffled detected labels and return the ordered list in JSON.The cases may use an image overlaid with segmentation maps or bounding boxes.Source: n-object Prompt for all VLMs · p. 19

3/5-object Prompt for the Merge Sort Setting

E Reinforcement Learning Experiments

E.1 Overview of Experiments

The GRPO evaluation comprises six experiments on 2B models, covering temporal, depth, and left-to-right ordering at two set sizes each.

E.1 Overview of Experiments

  • Six GRPO runs on 2B models cover temporal, depth, and left-to-right ordering in 3- and 5-image/object settings.Each task has one run at each set size.Source: §E.1 · p. 19

E.2 Training Configuration

The configuration uses GRPO with a Kendall τ reward, short English reasoning traces, and task-specific prompts for temporal, depth, and horizontal ordering.

E.2 Training Configuration

  • The final reward uses Kendall τ, scoring the percentage of correctly ordered pairs rather than a sparse exact-order reward or Spearman rank correlation.The authors report trying sparse and Spearman-based alternatives before settling on Kendall τ.Source: §E.2 · p. 19
  • GRPO training uses 700 unique prompts for 1,000 update steps, about 1.43 epochs, with six generations sampled per prompt at each step.The algorithm is Group Relative Policy Optimization.Source: §E.2 · p. 19

E.3 Global Settings

  • Across tasks, prompts request a short English reasoning trace before the final ordering, with a 700-token output cap for 2B RL tasks.The reasoning trace is treated as an optimization trajectory and is intended to help overcome default ordering bias.Source: §E.3 · p. 19

E.4 Temporal Ordering Task (with stamping + randomized example output)

  • Temporal ordering asks the model to chronologically order shuffled robot-task images, using red numeric labels and randomized example outputs during training.The labels identify images after shuffling and are used only for the temporal task; randomized examples are also temporal-only.Source: §E.4 · pp. 19–20

E.5 Depth Ordering Task

  • For depth ordering, the model orders objects in one robot-workspace image from closest to farthest, using object centroids in the image.Source: §E.5 · p. 20

E.6 Left-to-Right Ordering Task

  • For left-to-right ordering, the model uses one image to order objects horizontally from left to right.Source: §E.6 · p. 20

E.7 Training Curves

The appendix presents reward curves for six GRPO runs and examples of temporal-ordering tasks across base and visual-overlay settings.

E.7 Training Curves

  • Figures 3–8 plot Kendall reward for six GRPO runs spanning temporal, depth, and left-to-right ordering at 3- and 5-image/object sizes.The y-axis reward is the fraction of correct pairwise orderings, and each run uses a 700-token generation budget.Source: §E.7 · p. 20

F Task Examples

  • The task-example section covers temporal, depth, and spatial ordering in base, bounding-box, and segmentation-map settings.Source: F Task Examples · p. 22

F.1 Examples of Temporal Ordering Tasks

  • Temporal-ordering examples span the base, bounding-box, and segmentation-map settings, shown in Figures 9–13.Source: §F.1 · p. 22

Base Setting Tasks

The examples illustrate task descriptions and correct frame orders for base temporal tasks, plus towel-hanging examples with bounding-box and segmentation overlays; depth examples cover the same three settings.

Base Setting Tasks

Bounding Box Setting Task

  • The bounding-box temporal example asks the model to order descriptions of hanging a towel on a black stand or rod.Its correct order is 3, 2, 1.Source: Figure 12 · p. 25

Segmentation Map Setting Task

  • The segmentation-map temporal example uses the same towel-hanging descriptions and correct order as the bounding-box example.Source: Figure 13 · p. 25

F.2 Examples of Depth Ordering Tasks

  • Depth-ordering examples are provided for base, bounding-box, and segmentation-map settings in Figures 14–18.Source: §F.2 · p. 26

F.2.1 Base Setting Task

The appendix illustrates depthwise and left-to-right object-ordering examples across task settings, and reports that Qwen3-VL-8B-Instruct outperforms the Thinking model on downstream reasoning tasks at a similar token budget.

F.2.1 Base Setting Task

  • Depth-ordering examples give closest-to-farthest sequences for five objects in three scenes, including a robot arm–bottle–bowl cup–bowl–bin sequence.The other examples order container, cup, tray, robot arm, and bottle; and cardboard box packet, trash bin, coffee maker, sink, and faucet.Source: Figure 14 · p. 26, Figure 15 · p. 26, Figure 16 · p. 26

Bounding Box Setting Task

  • The bounding-box depth-ordering examples place the paper towel roll before the napkin dispenser and coffee machine.Figures 17 and 18 show the same correct order.Source: Figure 17 · p. 26, Figure 18 · p. 26

F.3 Examples of Left-to-Right Tasks

  • Left-to-right examples show five-object sequences such as chair, table, plant, door, and rug, as well as shorter three-object sequences.The examples include base, bounding-box, and segmentation-map settings.Source: Figure 19 · p. 27, Figure 20 · p. 27, Figure 21 · p. 27

H Token Limits for Reasoning Levels

  • Table 11 reports maximum permitted output tokens across reasoning levels and numbers of ordering elements.The supplied table caption gives the scope but not the token limits.Source: Table 11 · p. 28

I Downstream Reasoning Performance on Qwen3-VL-8B-Thinking

  • Qwen3-VL-8B-Instruct outperforms Qwen3-VL-8B-Thinking on downstream reasoning tasks at a similar token budget; the authors attribute the gap to Thinking exhausting its token budget before producing a valid response.Table 12 provides the Thinking model's accuracies on spatiotemporal ordering and downstream reasoning tasks.Source: I Downstream Reasoning Performance on… · p. 28, Table 12 · p. 28

Related papers

Loading 2610.33778v1…