Source-linked AI summary
OpenGVL -- Benchmarking Visual Temporal Progress for Data Curation
Paweł Budzianowski, Emilia Wiśnios, Michał Tyrolski, Gracjan Góral, Igor Kulakov, Viktor Petrenko, Krzysztof Walas
TL;DR
Robotics data is growing, but well-curated datasets remain scarce, making efficient assessment of shared data a relevant need. OpenGVL builds a benchmark on GVL and evaluates open-source models; these models reach approximately 60–70% of proprietary models’ upper-bound scores. The paper also demonstrates using OpenGVL to identify data issues for curation.
Problem
Robotics data is increasingly shared, but few large-scale datasets are available and existing ones remain much smaller than vision or language datasets.
Source: §1 · p. 1Method
Results
Open-source counterparts reach only approximately 60–70% of proprietary models’ upper-bound scores.
Source: §3.2 · p. 5Takeaways & Limitations
OpenGVL can identify issues within episodes and across datasets, including task-definition problems, labeling ambiguity, and failed or out-of-distribution examples.
Source: §4 · p. 6, §5 · p. 8The evaluation used a single temperature, one system-prompt template, and uniformly sampled expert trajectories; alternative settings remain to be examined.
Source: §5.1 · p. 8
Abstract
from arXiv · showhide
Data scarcity remains one of the most limiting factors in driving progress in robotics. However, the amount of available robotics data in the wild is growing exponentially, creating new opportunities for large-scale data utilization. Reliable temporal task completion prediction could help automatically annotate and curate this data at scale. The Generative Value Learning (GVL) approach was recently proposed, leveraging the knowledge embedded in vision-language models (VLMs) to predict task progress from visual observations. Building upon GVL, we propose OpenGVL, a comprehensive benchmark for estimating task progress across diverse challenging manipulation tasks involving both robotic and human embodiments. We evaluate the capabilities of publicly available open-source foundation models, showing that open-source model families significantly underperform closed-source counterparts, achieving only approximately $70\%$ of their performance on temporal progress prediction tasks. Furthermore, we demonstrate how OpenGVL can serve as a practical tool for automated data curation and filtering, enabling efficient quality assessment of large-scale robotics datasets. We release the benchmark along with the complete codebase at \href{github.com/budzianowski/opengvl}{OpenGVL}.
1 Introduction
OpenGVL builds on GVL to benchmark temporal task-progress prediction with open-source models and demonstrate data curation applications. The work addresses a growing supply of shared robotics data and a shortage of well-curated datasets.
1 Introduction
- Only a few large-scale robotics datasets are available, and they remain an order of magnitude smaller than datasets in vision or language.The passage names Agibot-World, OXE, and Droid as examples.Source: §1 · p. 1
- More than 2.6 million episodes were publicly shared on Hugging Face’s Dataset Hub as of August 2025, motivating efficient, cost-effective filtering.Some shared datasets are copies of previously available datasets.Source: §1 · p. 1
- Temporal progress prediction can be repurposed to curate and filter already collected datasets.Prior work proposed GVL, which uses VLM knowledge to predict universal value functions and estimate task progress; its VOC metric was described as useful for data curation.Source: §1 · p. 2
- OpenGVL replicates and extends GVL for open-source models, creating a benchmark and demonstrating practical data-curation methods.The authors present it as an open-source temporal progress prediction system for data management at scale.Source: §1 · p. 2
- OpenGVL reveals significant performance gaps between open-source and proprietary models.The authors also report that it can highlight patterns within episodes and across full datasets.Source: §1 · p. 2
2 Related work
Prior work frames task progress as a trajectory-level value and explores both task-specific training and in-context prediction with vision-language models. GVL uses shuffled observations and rank correlation to assess temporal progress and data quality.
2 Related work
- A value function assigns a scalar score to a trajectory, reflecting how well it achieves its goal.The trajectory is defined by observations from the first to the final time step.Source: §2 · p. 2
- Earlier approaches used sparse task-success signals and often trained or fine-tuned models for specific tasks or domains.Alakuijala et al. [25] and Zhang et al. [26] fine-tuned VLMs with sequential ranking objectives.Source: §2 · p. 2
- Ma et al. used in-context learning for fine-grained temporal success prediction, and found that shuffling input observations improved prediction quality.The VLM receives example trajectories with temporal value progress and a trajectory to evaluate.Source: §2 · p. 2
- GVL’s Value-Order Correlation compares predicted values for shuffled trajectory frames with their temporal order using rank correlation.VOC ranges from −1 for perfect inverse correlation to 1 for perfect alignment, and was shown to provide a useful, though not sufficient, signal of data quality.Source: §2 · pp. 2–3
3 OpenGVL Benchmark
OpenGVL extends GVL into a benchmark for temporal progress prediction with open-source VLMs and data-curation applications. The benchmark finds a substantial performance gap between open-source and proprietary models.
3.1 Experimental Setup
- On the hidden human task, Gemini-2.5-Pro predictions show an upward trend, whereas Gemma-3-27B-it shows minimal alignment with task completion.Source: Figure 2 · p. 3
- The initial validation samples 50 episodes from each of four manipulation datasets and evaluates zero-shot and two-shot conditioning using 15 shuffled frames per episode.Two hidden datasets target long-horizon planning and dexterous manipulation.Source: §3.1 · pp. 3–4
- The evaluation spans Gemma-3 and Qwen2.5-VL-Instruct model sizes, plus four reasoning-capable models and three proprietary models.The proprietary models are evaluated on shuffled frames after unshuffled testing showed reliance on temporal ordering cues.Source: §3.1 · p. 4
3.2 Benchmarking Open Source VLMs for GVL
- VOC scores improve with model size in both the Gemma and Qwen families, while MiMo-VL-7B-RL-2508 performs strongly among reasoning models.The largest Qwen and Gemma variants achieve similar scores and significantly outperform their smaller counterparts.Source: Table 1 · p. 4, §3.2 · p. 4
- Open-source counterparts reach approximately 60–70% of proprietary models’ upper-bound performance on temporal progress prediction.Source: §3.2 · p. 5
- On hidden tasks 1 and 2, zero-shot VOC clusters are at or below chance; two-shot prompting generally helps, but often remains insufficient for robust performance.Many two-shot scores remain around 0.1–0.3, with few reaching 0.7 or higher.Source: Figure 3 · p. 5
- Two additional last-mile assembly datasets test the same task with human execution and two 7-DOF robotic arms, with task data withheld to prevent contamination.Source: §3.2 · p. 5
3.3 OpenGVL Benchmark Space
- The OpenGVL Space invites community submissions of models and datasets, alongside a public evaluation interface and complete codebase.Source: §3.3 · p. 5
4 Data curation in the wild
OpenGVL applies temporal progress scoring to public datasets to surface curation issues, including unclear task definitions, ambiguous labels, and atypical or failed trajectories.
4 Data curation in the wild
- With more than 13,000 datasets published, the authors use OpenGVL to identify curation problems that can disrupt training, while treating upper-bound scores only as proxies for unobservable ground truth.They identify task-definition problems, labeling ambiguity, and failed or out-of-distribution examples.Source: §4 · pp. 5–6
4.1 Task definition
- Ambiguous “dig grass and dump” instructions make progress hard to define, so completion scores may decrease when they should consistently increase.The ambiguity concerns what counts as a dump and how much material must be excavated; VOC accuracy can reveal the issue.Source: §4.1 · p. 7
- In dopaul/1500_chess_moves, unclear task instructions and an often-obscured camera leave the VLM without useful visual information, challenging performant model training.The authors suggest clearer instructions or improved visual markers.Source: Figure 5 · p. 6, §4.1 · p. 7
4.2 Labeling ambiguity
- In willx0909/pickplace_joint, unclear instructions allow multiple pocket movements, leaving the VLM unable to establish consistent progress patterns across episodes.The authors report a very low VOC score and warn that this data can deteriorate VLA training.Source: §4.2 · p. 7
4.3 OOD/Failed examples
- Episode 93 in Rorschach4153/so101_60_new differs from the other 149 episodes, with rising and falling completion patterns that can flag failures or out-of-distribution behavior.The paper links such outliers to execution failures, sensor malfunctions, or different task interpretations.Source: Figure 6 · p. 7, §4.3 · p. 7
5 Conclusions
OpenGVL is an open-source benchmark for evaluating VLMs on robotic temporal task progress and a tool for finding issues in collected datasets. The evaluation used fixed temperature and prompt settings and uniform sampling, which the authors identify as areas for future study.
5 Conclusions
- OpenGVL enables rapid validation of VLMs on temporal task progress prediction and comparisons across models.The authors also demonstrate its use for identifying issues at macro and micro levels in collected datasets.Source: §5 · p. 8
- Qualitative examples show that OpenGVL can detect issues in open-source datasets, supporting work toward large-scale robotics datasets in the wild.The authors propose future investigation of visual goal or failure conditioning and enhancements to rank-correlation metrics.Source: §5 · p. 8
5.1 Limitations
- The evaluation fixed temperature at 1.0, used one system prompt, and sampled trajectories uniformly from expert demonstrations.The authors suggest testing temperature and prompt sensitivity and alternative sampling strategies.Source: §5.1 · p. 8
A Prompt
The prompt asks a VLM to predict task completion percentages from frames presented in random order. It provides labeled context examples and requests a description and percentage for each evaluation frame.
A Prompt
- The prompt frames the model as an expert roboticist predicting completion from 0 to 100, with 100 meaning full completion.It specifies that the evaluation frames are presented in random order.Source: A Prompt · p. 12
- A labeled initial scene anchors task completion at 0, and context frames provide example images with corresponding completion percentages.The prompt repeats this labeled format for the context frames.Source: A Prompt · p. 12
- For each randomly ordered evaluation frame, the model must output a description and task completion percentage in the specified format.The requested format includes the frame number, description, and completion percentage.Source: A Prompt · p. 12