PaperlayerView exact revision on arXiv

Source-linked AI summary

VisualTrans: A Benchmark for Real-World Visual Transformation Reasoning

Yuheng Ji, Yipu Wang, Yuyang Liu, Xiaoshuai Hao, Yue Liu, Yuting Zhao, Huaihai Lyu, Xiaolong Zheng

arXiv:2508.04043v1cs.CV

TL;DR

Existing VTR benchmarks have a sim-to-real gap, limited transformation complexity, and incomplete reasoning coverage. VisualTrans is a real-world manipulation benchmark that evaluates multiple reasoning dimensions using paired visual inputs. Evaluations find persistent VLM limitations in multi-step and temporal reasoning, including intermediate-state recognition and action-sequence planning.

  • Problem

    Existing VTR benchmarks often rely on synthetic scenes, cover simple transformations, and lack systematic evaluation across spatial, procedural, and quantitative reasoning.

    Source: Introduction · p. 2

  • Method

    VisualTrans constructs a benchmark from real-world human manipulation scenarios and evaluates transformations between paired scene images using task-specific questions.

    Source: Introduction · p. 2, Task Definition · p. 3

  • Results

    Evaluations reveal VLM limitations in multi-step reasoning, temporal modeling, and object relationship understanding, including intermediate-state recognition and action-sequence planning.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The findings identify intermediate-state recognition and action-sequence planning as areas for improvement in future intelligent systems.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Visual transformation reasoning (VTR) is a vital cognitive capability that empowers intelligent agents to understand dynamic scenes, model causal relationships, and predict future states, and thereby guiding actions and laying the foundation for advanced intelligent systems. However, existing benchmarks suffer from a sim-to-real gap, limited task complexity, and incomplete reasoning coverage, limiting their practical use in real-world scenarios. To address these limitations, we introduce VisualTrans, the first comprehensive benchmark specifically designed for VTR in real-world human-object interaction scenarios. VisualTrans encompasses 12 semantically diverse manipulation tasks and systematically evaluates three essential reasoning dimensions - spatial, procedural, and quantitative - through 6 well-defined subtask types. The benchmark features 472 high-quality question-answer pairs in various formats, including multiple-choice, open-ended counting, and target enumeration. We introduce a scalable data construction pipeline built upon first-person manipulation videos, which integrates task selection, image pair extraction, automated metadata annotation with large multimodal models, and structured question generation. Human verification ensures the final benchmark is both high-quality and interpretable. Evaluations of various state-of-the-art vision-language models show strong performance in static spatial tasks. However, they reveal notable shortcomings in dynamic, multi-step reasoning scenarios, particularly in areas like intermediate state recognition and transformation sequence planning. These findings highlight fundamental weaknesses in temporal modeling and causal reasoning, providing clear directions for future research aimed at developing more capable and generalizable VTR systems. The dataset and code are available at https://github.com/WangYipu2002/VisualTrans.

Introduction

VisualTrans targets gaps in real-world VTR benchmarks with a benchmark built from human manipulation scenarios and designed to assess spatial, procedural, and quantitative reasoning. It formalizes transformation questions over paired images and uses first-person videos to construct its samples.

Introduction

  • Evaluations reveal that current VLMs remain limited in multi-step reasoning, temporal modeling, and object relationship understanding, especially intermediate-state recognition and action-sequence planning.The authors identify these as areas for improvement in future intelligent systems.Source: Introduction · p. 2
  • VisualTrans comprises 12 real-world manipulation task types, three reasoning dimensions, six subtask categories, and three QA formats, with 472 samples.Its construction uses multimodal expert models for object extraction, metadata generation, and QA construction, followed by human verification.Source: Introduction · pp. 2–3

Related Work

  • Earlier benchmarks often use synthetic scenes, simple transformations, or lack systematic coverage of spatial, procedural, and quantitative reasoning.The authors position VisualTrans as a real-world human-interaction benchmark for complex object dynamics and multiple reasoning dimensions.Source: Introduction · p. 2, Related Work · p. 3

Task Definition

  • Each VisualTrans sample pairs initial and transformed scene images; a task-specific question guides inference of a transformation-aware answer.Questions specify a reasoning requirement, and answers use one of the defined output formats.Source: Task Definition · p. 3

Task Categorization

  • The task taxonomy covers spatial changes, procedural inference, and quantity changes, including location and layout recognition, intermediate states, action causality, and sequence planning.Quantitative tasks ask models to identify added or removed objects and estimate their quantities.Source: Task Categorization · p. 4

VisualTrans Construction Pipeline

  • VisualTrans is built from EgoDex first-person operation videos, and the authors manually selected 12 representative real-world task scenarios.The construction process includes task selection, image cleaning, sample filtering, metadata annotation, QA generation, and verification.Source: VisualTrans Construction Pipeline · p. 4

d. QA Generation and Verification

The section presents VisualTrans’s construction and evaluation, finding that current VLMs remain limited on real-world visual transformation reasoning, especially dynamic and fine-grained tasks.

d. QA Generation and Verification

  • From EgoDex, the pipeline selects manipulation clips, filters poor image pairs, annotates localized objects and scene metadata, then template-generates QA pairs that humans review.Source: d. QA Generation and Verification · pp. 4–5

Benchmark Statistics

  • VisualTrans contains 472 QA samples from 12 real-world manipulation scenarios, spanning three reasoning dimensions, six subtask types, and three QA formats.Source: Benchmark Statistics · p. 5

Experimental Setup

  • The evaluation covers 17 open- and closed-source VLMs in a zero-shot setting and scores final answers using scalar- or set-level exact match.Source: Experimental Setup · p. 5

Quantitative Results

  • Quantitative tasks are most tractable, with o3 at 79.37%, while no model exceeds 55% on intermediate-state recognition.For transformation sequence planning, o3 and Gemini-2.5-Pro reach 78.57%.Source: Quantitative Results · p. 5
  • Most VLMs reach only 30%–50% accuracy, while closed-source o3 (59.96%) and Gemini-2.5-Pro (54.93%) lead VisualTrans.Source: Quantitative Results · p. 5, Table 2 · p. 6
  • Fine-grained spatial changes remain difficult: Qwen2.5-VL-7B reaches 17.86% and InternVL3-14B 18.45%, while global spatial performance improves.Source: Quantitative Results · p. 6

Error Analyses

  • Five recurring failures span visual-state misalignment, task-intent misunderstanding, perception errors, insufficient causal reasoning, and object-tracking failure.Source: Error Analyses · pp. 6–7

Conclusion

  • The authors conclude that VLMs handle static spatial tasks better than dynamic and causal subtasks, exposing limitations in temporal and transformation modeling.Source: Conclusion · p. 7

Appendix

The appendix supplements the main paper with construction, annotation, and evaluation details. Figure 5 illustrates how benchmark image pairs capture manipulation transformations from video endpoints.

Appendix

  • The supplementary material provides implementation details and resources on VisualTrans construction, annotation, and evaluation to support reproducibility and further research.It complements the main paper.Source: Appendix · p. 11
  • The appendix covers the raw video source, data curation, evaluation setup, prompts and case studies, and future directions.Its sections address task diversity, multi-agent scenarios, and data transparency.Source: Appendix · p. 11
  • Each benchmark image pair corresponds to a video's first and last frame, capturing a manipulation transformation.Figure 5 depicts Egodex as a collection of diverse manipulation scenarios.Source: Figure 5 · p. 11

A Details of Data Source

VisualTrans draws representative image pairs from EgoDex, a large egocentric dataset of dexterous manipulation. EgoDex contains extensive video, pose, camera, and language annotations across diverse tabletop tasks.

A Details of Data Source

  • VisualTrans extracts representative image pairs from EgoDex manipulation episodes, using a dataset released under CC-BY-NC-ND with approximately 2.0 TB of content.EgoDex is intended to support research including dexterous manipulation trajectory prediction and human-object interaction modeling.Source: A Details of Data Source · p. 11
  • EgoDex contains 829 hours of egocentric video across 338,000 task episodes, covering 194 tabletop manipulation tasks and around 500 household objects.Videos are recorded at 1920×1080 resolution and 30 FPS, comprising approximately 90 million frames.Source: A Details of Data Source · p. 11
  • EgoDex provides synchronized 3D hand and upper-body pose annotations, camera parameters, pose-confidence scores, and natural-language episode descriptions.Its tasks are categorized as reversible, reset-free, or reset-required.Source: A Details of Data Source · p. 11

B Details of Data Curation

The data-curation pipeline comprises task selection, image filtering, metadata annotation, and question-answer generation.

B Details of Data Curation

Task Selection

The authors manually select manipulation scenarios with substantial object interactions and transformations, then use endpoint and intermediate frames to build benchmark questions.

Task Selection

  • The authors exclude tasks with minimal object-state changes or low-dynamic, localized actions because they lack meaningful spatial reconfiguration or multi-object interaction.Source: Task Selection · p. 11
  • Selected tasks involve at least two identifiable objects, noticeable spatial displacement, and at least two sequential manipulation steps.Source: Task Selection · p. 11
  • The authors manually curate 12 representative scenarios for salient visual transitions and spatial-relationship changes.The listed scenarios include stacking bowls, sorting beads, making a sandwich, and playing Connect Four.Source: Task Selection · p. 11
  • Initial and completion frames form the core data unit, while one or two intermediate frames per video support QA construction.Intermediate frames provide context for transformation progress and help generate semantically plausible distractors, increasing QA diversity and difficulty.Source: Task Selection · p. 12

Image Cleaning and Filtering

Direct frame extraction creates quality challenges from visual artifacts and samples with too little manipulation complexity or transformation saliency. The filtering process excludes samples that fail visual-quality or interaction-diversity thresholds.

Image Cleaning and Filtering

  • Blur, occlusion, and minimal spatial reconfiguration hinder scene understanding, object-state recognition, and modeling of multi-step or intermediate-state reasoning.The examples include hands or arms obscuring objects, camera motion or limited depth of field causing blur, and simple actions involving only one food item or one or two cups.Source: Image Cleaning and Filtering · p. 12
  • The o3 model assesses image clarity, key-object visibility and distinctness, severe occlusions, and the number of actively manipulated objects.Samples that fail the quality or interaction-diversity thresholds are excluded.Source: Image Cleaning and Filtering · p. 12
  • Approximately 85% of samples are retained as high-quality after filtering.Directly extracting initial and final video frames enables visual state contrast samples but also introduces data-quality challenges.Source: Image Cleaning and Filtering · p. 12

Automatic Metadata Annotation

VisualTrans automatically annotates sample metadata with types and prompts tailored to manipulation scenarios. Structural annotations receive additional visual assistance, while one scenario is manually annotated because its structural parsing remains unreliable.

Automatic Metadata Annotation

  • Metadata types and prompt templates are tailored to different manipulation scenarios.Source: Automatic Metadata Annotation · p. 12
  • Metadata covers manipulation direction, background, manipulated objects, initial absolute and relative positions, final layouts, quantitative states, and board states.These fields support spatial, procedural, quantitative, and implicit-manipulation reasoning tasks.Source: Automatic Metadata Annotation · p. 12
  • Gemini 2.5 Pro labels the metadata; Grounding DINO detects task-relevant objects so cropped regions can aid final-layout descriptions.Crops are used because full-frame extraction of final structure layouts can be unreliable amid clutter and visual noise.Source: Automatic Metadata Annotation · p. 12
  • All metadata for the build unstack lego scene is manually annotated because the model's structural parsing remains unreliable.Source: Automatic Metadata Annotation · p. 12

QA Generation and Verification

The QA pipeline uses task-specific templates and metadata, then human review to refine question quality and alignment. The resulting benchmark contains 472 samples distributed across procedural, spatial, and quantitative transformation categories.

QA Generation and Verification

  • Question templates are designed for representative scenes in each fine-grained subtask, with masked placeholders filled automatically from structured metadata.Table 3 is identified as showing examples of templates aligned with different VisualTrans subtasks.Source: QA Generation and Verification · p. 12, Table 3 · p. 13
  • Most ground-truth answers come from metadata, distractors are generated for multiple-choice questions, and correct answers are manually annotated when metadata coverage is incomplete.Source: QA Generation and Verification · pp. 12–13
  • Human reviewers answer from visual inputs alone and flag missing correct options, incorrect answers, ambiguity, or insufficient visual evidence; problematic items are corrected or discarded.The review process follows a random selection of 500 pairs for downstream evaluation, with an estimated automatic-labeling error rate of approximately 10 percent.Source: QA Generation and Verification · p. 13
  • Iterative refinement of templates and QA synthesis procedures improves the quality and clarity of the resulting pairs.Source: QA Generation and Verification · p. 13
  • 472 QA samples comprise 45.8% procedural, 40.9% spatial, and 13.3% quantitative transformation; intermediate state recognition, action causality, and sequence planning account for 18.6%, 18.2%, and 8.9%.Fine-grained and global change recognition account for 35.6% and 5.3%, respectively.Source: QA Generation and Verification · p. 13

C Details of Evaluation

The section describes evaluation procedures and computational resources used for VisualTrans experiments. Benchmark construction and evaluation use a workstation with four NVIDIA Tesla V100-SXM2 GPUs.

C Details of Evaluation

  • The evaluation-details section outlines model inference configurations and computational resources used in VisualTrans experiments.Source: C Details of Evaluation · p. 13

C.1 Computational Resources

  • The workstation uses Intel Xeon E5-2698 v4 CPUs and four 32GB NVIDIA Tesla V100-SXM2 GPUs for data processing, benchmark construction, and evaluation.Source: §C.1 · p. 13

C.2 General Evaluation Setup

This material describes the model evaluation setup and supplementary visualizations, including cleaning and annotation prompts and representative benchmark case studies.

C.2 General Evaluation Setup

  • Models were evaluated through official APIs with default decoding settings and a unified prompt format followed by task-specific questions.The prompts correspond to each reasoning type.Source: §C.2 · p. 14

D Additional Visualizations

  • The supplementary visualizations include prompts for data cleaning and metadata annotation, plus selected case studies of benchmark performance.The case studies show two questions and corresponding model responses for representative subtasks.Source: D Additional Visualizations · p. 14, §D.3 · p. 14

D.1 Cleaning and Filtering Prompts

  • Image-pair filtering prompts support automated screening for image clarity, object visibility, and transformation complexity.The prompts use the o3 model and are illustrated in Figures 7–15.Source: §D.1 · p. 14

D.2 Metadata Annotation Prompts

  • The full prompt collection for automatic metadata annotation uses the Gemini 2.5 model.The prompts are shown in Figures 16a–29.Source: §D.2 · p. 14

D.3 Case Studies

  • Representative subtask case studies each present two questions with corresponding model responses.Examples appear in Figures 30–41.Source: §D.3 · p. 14

E Future Work

This section outlines future expansion of VisualTrans beyond tabletop, single-agent manipulation and presents figures illustrating action-causality reasoning, sequential planning, and quantitative transformation.

E Future Work

SYSTEM_PROMPT_CONNECT_FOUR

  • One figure illustrates action-causality reasoning through an example of unstacking Lego.It is labeled Sample 165.Source: Figure 37 · p. 52
  • Two figures illustrate sequential planning for unscrewing a finger fixture and setting up or cleaning up a table.They are labeled Samples 143 and 148, respectively.Source: Figure 38 · p. 53, Figure 39 · p. 54
  • Two figures illustrate quantitative transformation in picking and placing food and sorting beads.They are labeled Samples 021 and 046, respectively.Source: Figure 40 · p. 55, Figure 41 · p. 56
Loading 2508.04043v1…