Source-linked AI summary
CoVA-SFT: A Large-Scale Dataset for Chain of Visual Abstractions
Tsung-Han Wu, Heekyung Lee, Anya Ji, Haoming Chen, Trevor Darrell, Joseph E. Gonzalez, David M. Chan
TL;DR
Text-only chain-of-thought poorly represents non-textual structured states, while the field lacks a large, self-corrected dataset for learning internal visual workspaces. CoVA-SFT supplies such data through rationale formulation, rendering, and verification, and its trained models learn to leverage visual abstractions on complex domains, though the dataset remains bounded by generator reliability and static schematic formats.
Problem
Text-only chain-of-thought inefficiently serializes visual, spatial, structural, and relational states, while comprehensive training and evaluation data for long-horizon visual workspace maintenance is lacking.
Method
CoVA-SFT constructs 51,904 trajectories with 222,046 visual abstraction steps across five layout families and 17 tasks using rationale formulation, agentic rendering, and verification loops.
Results
CoVA-SFT-trained models learn to leverage visual abstractions, including reported results of 44.5% on Graph reasoning and 19.3% on Game simulation.
Takeaways & Limitations
The dataset provides a resource for teaching multimodal models to build and maintain internal visual workspaces during complex textual reasoning.
Takeaways & Limitations
Subtle hallucinations in generated traces or rendering code can enter the training distribution, while the taxonomy is limited to static structured formats rather than continuous or deeply perceptual environments.
Abstract
from arXiv · showhide
Chain-of-thought (CoT) reasoning has dramatically improved large language models (LLMs) by allowing them to decompose problems into intermediate steps. While CoT is widely effective for linguistic tasks, text-only CoT forces models to serialize visual problems into awkward prose. Although architectural solutions exist to process visual inputs, the community lacks a massive, multi-step, self-corrected dataset to teach models how to build and maintain internal visual workspaces when solving purely textual reasoning problems. To address this limitation, we introduce CoVA-SFT, a highly structured corpus of 51.9K samples containing over 222K multimodal reasoning steps across 5 distinct layout families and 17 complex tasks, and CoVA-Bench, a companion benchmark of 1,700 held-out test samples spanning the same tasks for reproducible evaluation. By providing explicit rationale formulations, agentic renderings, and verification loops, CoVA-SFT teaches multimodal language models to interleave text and visual abstractions. We validate the dataset by demonstrating that models fine-tuned on CoVA-SFT outperform all interleaved CoT baselines by more than 2x on average on CoVA-Bench, though they still fall short of strong text-only CoT baselines, highlighting open challenges for future work.
1 Introduction & Background
Text-only chain-of-thought is effective for linguistic intermediate states but poorly represents visual, spatial, structural, and relational states. CoVA-SFT addresses this gap with a large interleaved dataset and benchmark for learning visual workspaces.
- Motivation: Text-only chain-of-thought forces models to serialize non-textual structured states into inefficient prose.The paper contrasts this with reasoning over naturally visual states such as chessboards.
- Motivation: The community lacks a massive, multi-step, self-corrected dataset for building internal visual workspaces during purely textual reasoning.Existing resources are described as limited in scale, domain diversity, or sequential reasoning depth.
- Contributions: CoVA-SFT uses rationale formulation, agentic rendering, and verification to construct visual workspaces from text-based problems.The pipeline explicitly asks models to explain the visual abstraction, render it, and verify structural consistency.
- Evaluation: Models trained on CoVA-SFT learn to leverage visual abstractions, with reported results including 44.5% on Graph reasoning and 19.3% on Game simulation.These figures are presented as examples of performance on complex domains.
- Contributions: CoVA-SFT contains 51,904 trajectories with 222,046 visual abstraction steps across five layout families and 17 complex tasks.The dataset is designed to teach multimodal models to interleave text and visual abstractions.
- Evaluation: CoVA-Bench provides 1,700 held-out test samples spanning all 17 tasks for evaluating logical interleaved-reasoning skills.The benchmark accompanies the training corpus and supports reproducible validation.
2 Interleaved Reasoning Dataset: CoVA-SFT
CoVA-SFT is constructed as a multisource interleaved-reasoning corpus whose trajectories teach models to maintain visual state across complex textual problems. Its synthesis pipeline formulates rationales, renders visual workspaces step by step, and verifies them through self-correction.
- Dataset Scope: CoVA-SFT contains 51,904 examples and 222,046 multimodal reasoning steps across 17 diverse tasks.The corpus is intended to teach models to build internal workspaces for complex textual reasoning.
- Construction Pipeline: Rationale and QA formulation specifies why a visual workspace is useful, what to render, and the corresponding question–answer pair.This is the first stage of the three-stage construction process.
- Construction Pipeline: Agentic rendering generates the interleaved trace step by step, programmatically renders each workspace with Matplotlib, and feeds the image back as context.The rendered workspace becomes part of the continuing reasoning process.
- Construction Pipeline: Verification and self-correction check each rendered image against the problem statement and redraw it when structural errors are detected.The model re-enters the rendering loop after detecting an inconsistency.
- Dataset Scope: The dataset covers Table, Graph, Layout, Game, and Math abstraction families across 17 tasks.The same five families and tasks define the held-out CoVA-Bench evaluation scope.
- Evaluation: CoVA-Bench contains 1,700 withheld samples, with 100 samples per task, verified to require multi-step logical reasoning and visual-state maintenance.The benchmark is drawn from the same task and abstraction-family taxonomy as CoVA-SFT.
3 Experiments
The experiments evaluate CoVA-SFT on a held-out benchmark against text-only and interleaved visual-reasoning baselines. CoVA-SFT leads interleaved baselines overall, but its advantage over text-only reasoning depends on the task.
- Evaluation setup: CoVA-SFT is evaluated on CoVA-Bench, a 1,700-sample held-out benchmark spanning all 17 tasks.The comparison includes text-only and interleaved CoT baselines under zero-shot conditions.
- Results: 38.2% average score places CoVA-SFT above the next-best interleaved baseline, MathCanvas, at 16.8%.MathCanvas performs better than CoVA-SFT on Math.
- Analysis: External visual-reasoning baselines remain weaker overall because rendering errors can propagate into later reasoning with limited recovery.CoVA-SFT instead internalizes the visual workspace within the reasoning process.
- Results: CoVA-SFT trails text-only baselines on Table at 47.2%, Layout at 53.4%, and Math at 8.5%.The Math comparison is 8.5% for CoVA-SFT versus 27.3% for the text-only baseline.
- Results: 62.0% on Graph lets CoVA-SFT surpass text-only models at 57.0%.The paper connects this result to graphs’ relational structure and the difficulty of maintaining adjacency in prose.
4 Conclusion
CoVA-SFT is introduced as a large-scale interleaved reasoning dataset for teaching multimodal models to build and maintain internal visual workspaces on spatial, structural, and relational tasks.
- Conclusion: CoVA-SFT contains 51,904 trajectories and 222,046 visual abstraction steps across 17 tasks.The dataset is intended to teach multimodal models to build and maintain internal visual workspaces.
Limitations
The paper identifies limitations in data provenance, visual coverage, and computational cost. These constraints affect the reliability, scope, and efficiency of CoVA-SFT training and inference.
- Data quality: Generated text traces or rendering code may contain subtle hallucinations that propagate into the distilled training data.The resulting logic flaws may bound downstream fine-tuned-model accuracy.
- Visual scope: The taxonomy excludes highly complex open-world spatial simulations and continuous robotic environments.Its abstractions are constrained to static structured formats such as tables, graphs, layouts, and coordinate plots.
- Visual scope: The learned visual abstractions remain symbolic and schematic rather than continuous or deeply perceptual.This follows from the dataset’s static structured visual formats.
- Computation: Joint optimization of textual and latent visual tokens adds computational overhead over standard text-only SFT.The dual objective requires hyperparameter tuning and increased memory bandwidth, while long trajectories increase inference context and compute requirements.
A Performance and Evaluation Details
The appendix reports detailed per-task evaluation results and explains that the paper’s main performance numbers use an LLM-as-a-judge setting.
- Evaluation details: Detailed per-task results appear in Table B.2, with test answers spanning letters, words, names, tables, and symbolic expressions.The main paper reports numbers judged by Gemini-2.5-flash as a practical reasoning-performance estimate.
B Dataset Construction Details
CoVA-SFT combines generated and open-source question-answer data across several structured reasoning tasks. Its construction uses constraint-based generation, procedural geometric rendering, and automatic or solver-based answer computation.
- CoVA-SFT question-answer pairs come from programmatic generation or existing open-source datasets.
- Generated tasks: Seating Arrangement uses constraint-satisfaction generation over straight-line and circular layouts with 18 seating constraint templates.Constraints are added iteratively until one valid arrangement remains; conflicting constraints are discarded.
- Generated tasks: Logic Puzzles sample worlds with n houses and m attributes, add weighted clues, and retain instances whose CSP solution is unique.The resulting puzzles are rendered in natural language with JSON answer keys.
- Generated tasks: 3D Cube tasks procedurally render connected voxel structures and ask about exposed area, painted-face counts, or removal-induced surface-area changes.Ground-truth answers are computed automatically from voxel geometry.
- Open-source tasks: Remaining data is sourced from existing open-source datasets, with source details provided in Table B.1.
C Training Data Examples
Figure C.1 presents representative interleaved reasoning trajectories from CoVA-SFT. The examples cover all 17 tasks across five visual abstraction families.
- Figure C.1 shows representative interleaved reasoning trajectories from CoVA-SFT.
- The examples cover all 17 tasks across five visual abstraction families.
D Ethics, Risks, and Artifact Documentation
The documentation includes dataset risks and representative examples of visualized reasoning tasks. It warns that generated traces and renderings may retain errors, requiring additional validation for high-stakes use.
- Ethics and risks: CoVA-SFT may inherit hallucinations, incorrect reasoning steps, or malformed renderings from its upstream vision-language generation process.Verification and self-correction are intended to reduce structural inconsistencies, but subtle errors may remain.
- Ethics and risks: The authors recommend using CoVA-SFT for research and evaluation rather than high-stakes decision making without additional validation.
- Training examples: Maze examples encode start, end, and open cells while requiring paths to remain within traversable cells.The visual reasoning trace represents walls, corridors, endpoints, and path extensions through grid cells.
- Training examples: Graph examples visualize nodes, directed capacities, source-to-sink routes, and an 8-unit maximum-flow result.The reasoning identifies node 5 as the bottleneck predecessor of sink 6 and decomposes the flow across two paths.
- Training examples: 3D cube examples reason about painted faces by modeling a one-layer solid and counting each cube’s horizontal neighbors.A cube has exactly four painted faces when it has exactly two horizontal neighbors; the enumeration identifies the relevant cubes.
- Training examples: Visitall examples test action applicability using robot location, cell availability, connectivity, and visitation state.Only action B is applicable because its source matches the robot location, the cells are connected, and the destination is available.
E Experimental Hyperparameters
The appendix documents CoVA’s training hardware and systems configuration, including distributed SFT across NVIDIA H200 GPUs with performance-oriented kernels.
- SFT training used 8 NVIDIA H200 (80GB) GPUs with Fully Sharded Data Parallel (FSDP2) in mixed-precision bfloat16.
- Training used FlashAttention v2 and Liger Kernel fused operations to improve throughput and memory efficiency.
F AI Use Disclosure
The authors disclose using AI-based tools during manuscript preparation and research-code development, with author review and verification of assisted content.
- AI tools assisted with code generation, editing, and writing during the paper’s preparation.The assistance included drafting and revising manuscript text and supporting code development, debugging, and refinement.
- The authors reviewed, verified, and modified AI-generated or AI-assisted content, code, analyses, and interpretations when necessary.
- Appendix materials include per-category reasoning performance and SFT training hyperparameters.