Source-linked AI summary
Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation
Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, Christopher Clark
TL;DR
Text-rich image understanding is hindered by scarce diverse training data. CoSyn uses text-only LLMs and executable code to generate images and instruction data, producing strong benchmark results and enabling adaptation and pointing tasks.
Problem
High-quality, realistic, and diverse vision-language datasets for text-rich images are limited, although such understanding supports scientific analysis, accessibility, and agentic workflows.
Method
CoSyn generates code-rendered text-rich images and uses the underlying code as context for text-only LLMs to create instruction-tuning data, including chain-of-thought and pointing data.
Results
Across seven text-rich VQA benchmarks, CoSyn-trained models achieve state-of-the-art performance among competitive open-source models and surpass GPT-4V and Gemini 1.5.
Takeaways & Limitations
CoSyn supports data-efficient adaptation to novel domains and multimodal grounding for agentic tasks such as screenshot element pointing.
Takeaways & Limitations
Synthetic data quality and coverage depend on prompt and rendering-pipeline diversity, may require domain expertise, may miss real-world complexity, and are currently limited to English.
Abstract
from arXiv · showhide
Reasoning about images with rich text, such as charts and documents, is a critical application of vision-language models (VLMs). However, VLMs often struggle in these domains due to the scarcity of diverse text-rich vision-language data. To address this challenge, we present CoSyn, a framework that leverages the coding capabilities of text-only large language models (LLMs) to automatically create synthetic text-rich multimodal data. Given input text describing a target domain (e.g., "nutrition fact labels"), CoSyn prompts an LLM to generate code (Python, HTML, LaTeX, etc.) for rendering synthetic images. With the underlying code as textual representations of the synthetic images, CoSyn can generate high-quality instruction-tuning data, again relying on a text-only LLM. Using CoSyn, we constructed a dataset comprising 400K images and 2.7M rows of vision-language instruction-tuning data. Comprehensive experiments on seven benchmarks demonstrate that models trained on our synthetic data achieve state-of-the-art performance among competitive open-source models, including Llama 3.2, and surpass proprietary models such as GPT-4V and Gemini 1.5 Flash. Furthermore, CoSyn can produce synthetic pointing data, enabling VLMs to ground information within input images, showcasing its potential for developing multimodal agents capable of acting in real-world environments.
1 Introduction
Text-rich image understanding is important but limited by scarce, diverse vision-language data. CoSyn addresses this gap with code-guided synthetic generation and improves performance across benchmarks, novel domains, and pointing tasks.
- Text-rich images require textual comprehension and spatial reasoning, yet high-quality, realistic, and diverse training datasets remain limited.These capabilities support scientific analysis, accessibility, and real-world agentic workflows.
- CoSyn uses text-only LLMs to generate code that renders diverse text-rich images, then uses that code to create instruction-tuning data.The framework supports rendering tools including Python, HTML, and LaTeX.
- CoSyn-400K enables evaluation across seven text-rich VQA benchmarks, where trained models achieve state-of-the-art performance among competitive open-source models and surpass GPT-4V and Gemini 1.5.The evaluation also reports sample-efficient learning, chain-of-thought benefits, and stronger generalization to human-written questions.
- With 7K in-domain nutrition-label examples, CoSyn-trained models surpass most open VLMs trained on millions of images in a novel zero-shot domain.This demonstrates targeted synthetic data generation for adapting VLMs to new domains.
- Synthetic pointing data enables models to retrieve coordinates for queried screenshot elements and achieves state-of-the-art performance on ScreenSpot.The result extends CoSyn beyond standard VQA toward agentic tasks.
2 Related Work
Prior work addresses text-rich image understanding and synthetic annotation, but existing approaches often cover limited formats or rely on handcrafted templates. CoSyn broadens this direction through code-guided generation across diverse image types.
- VLM research has advanced multimodal processing, but text-rich image understanding remains a distinct challenge because naturally occurring training data are scarce.
- Synthetic vision-language data have been used where paired data are limited, including chart and plot VQA.
- Earlier chart and plot approaches typically use a small set of chart types and handcrafted question templates.
- Recent methods use text-only LLMs or generated code to create annotations and render synthetic charts, while PixMo-docs is smaller in scale and diversity.
3 Problem Formulation
CoSyn formulates synthetic multimodal data generation around a text query describing an image type. Code serves as the intermediate representation connecting rendered images with textual instruction-tuning data.
- Given a text query q about an image type, CoSyn aims to create a dataset Dq containing images I and textual instruction-tuning data T.
- The generated dataset is used to train a VLM to improve understanding of images related to the queried domain.
- Code C bridges image and text by rendering the image and providing textual context for instruction-data generation.
4 CoSyn System
CoSyn combines configurable rendering pipelines, persona-conditioned diversity, and code-conditioned instruction generation to produce a large text-rich image dataset. Its system supports nine categories and 400K synthetic images.
- CoSyn System: CoSyn selects among 20 pipelines built on 11 rendering tools based on a language query.
- CoSyn System: CoSyn uses tools such as Matplotlib, Plotly, Vega-Lite, LaTeX, HTML, Mermaid, and Graphviz for varied text-rich image formats.
- CoSyn-400K: The dataset includes 2.7M instruction-tuning examples associated with its text-rich image collection.
- CoSyn System: Each pipeline performs topic generation, detailed data generation, executable code generation, and code-conditioned instruction generation.Instructions can include questions, answers, and chain-of-thought explanations.
- CoSyn System: Persona-conditioned topic generation is used to increase diversity beyond what sampling parameters alone typically provide.
- CoSyn-400K: CoSyn-400K contains 400K images across nine categories, with fine-grained image types controlled through language queries.Document generation uses over 100 queries covering examples such as receipts, resumes, and meal plans.
5 Experimental Setup
The experiments evaluate CoSyn-based VLMs using established benchmarks, auxiliary datasets, and comparisons with similarly scaled open-source and proprietary models. Training follows a two-stage setup, with a best-performing model using all dataset categories and a separate zero-shot configuration.
- Model and training: The model follows Molmo’s two-stage training strategy: pre-training on dense PixMo-Cap captions, then supervised fine-tuning on evaluation, synthetic, and auxiliary datasets.The architecture connects a CLIP ViT-L/14 336px vision encoder to Mistral-7B through an MLP layer.
- Datasets: The evaluation covers seven text-rich benchmarks, while auxiliary fine-tuning data comprises around 1M training images across multiple VQA, chart, table, and science datasets.TableVQA is evaluation-only, and the ScreenQA training split is not used.
- Training configurations: The best-performing model uses all three dataset categories, while a separate zero-shot model uses only auxiliary and synthetic data without evaluation-dataset examples.The zero-shot model’s benchmark performance is reported as competitive in Table 1.
- Baselines: The comparison includes similarly scaled open-source VLMs alongside GPT-4V, Gemini-1.5-Flash, and Claude-3 Opus.Open-source baselines range from 3B to 12B parameters, including Llama 3.2 V.
- Implementation: Training uses TPU v3-128 with batch size 32 for 60K steps, and checkpoints with the highest validation performance are retained for testing.Training the best-performing model takes about 30 hours.
6 Results
Across seven text-rich benchmarks, CoSyn-trained models achieve leading performance, while analyses show gains from synthetic data in efficiency, generalization, diversity, reasoning, and pointing tasks.
- 6.1 Main Results: The 7B model achieves the highest average performance across seven benchmarks, surpassing Llama 3.2 11B by 3.9%.It ranks first on four datasets and second on the remaining three.
- 6.2 Analysis: Synthetic data produces a significant 3.6% boost over auxiliary-data training in the reported ablation.
- 6.2 Analysis: 400K synthetic samples make the model match GPT-4V on NutritionQA, while 7K domain-specific samples outperform most open-source VLMs.The NutritionQA evaluation contains 100 examples, and the x-axis measures instruction-tuning examples.
- 6.2 Analysis: Chain-of-Thought reasoning improves ChartQA, TableVQA, and NutritionQA but reduces DocVQA and InfoVQA performance because benchmark answer biases favor short responses.Strict string matching can assign zero to a correct but more detailed response.
- 6.2 Analysis: Using multiple rendering tools increases image diversity and improves ChartQA performance, while scaling synthetic chart images further improves performance.The final dataset contains 400K images and cost about $8,000 to produce.
- 6.3 Synthetic Pointing Data: Synthetic pointing data reaches performance comparable to 155K human-annotated samples with 65K samples and, when combined with human data, achieves state-of-the-art ScreenSpot performance.The comparison includes UGround, which was trained on 1.3M screenshots.
7 Conclusion
CoSyn uses synthetic data to improve VLM performance on text-rich image understanding, highlighting the utility of text-only LLM coding capabilities for multimodal learning and real-world applications.
- CoSyn significantly enhances VLM performance on text-rich image understanding.
Limitation
Synthetic-data effectiveness depends on prompt and rendering-pipeline quality, while specialized domains may require customization and domain expertise to achieve diverse, relevant coverage.
- Synthetic-data effectiveness depends heavily on prompt and rendering-pipeline quality and diversity.
- Highly specialized domains may require prompt engineering, customized rendering tools, and domain-specific expertise to ensure adequate coverage.
- Synthetic data may not fully capture real-world complexity, especially in highly variable or evolving domains.
Ethical Statement
The paper reports no significant ethical concerns, while noting that synthetic data can either propagate generator biases or mitigate bias by expanding training-data coverage.
- The authors report no significant ethical concerns for this work.
- Synthetic data can propagate biases from the generation model but may also mitigate bias by expanding vision-language training-data coverage.
A.4 Academic Datasets
During supervised fine-tuning, the authors combine academic and synthetic datasets and report dataset sizes and official evaluation metrics for seven text-rich benchmarks.
- Academic datasets are included alongside synthetic datasets during supervised fine-tuning.
- The training mixture includes datasets ranging from 6.2K ScienceQA images to 200.0K DVQA images.
- Evaluation uses each benchmark’s official metrics, including relaxed correctness for ChartQA and ANLS for DocQA and InfoQA.
A.5 Training Details
Training uses overlapping image crops and a fixed optimization setup, while ablations compare chain-of-thought prompting and supervised-data selection. Testing permits more crops than training to accommodate high-resolution text-rich images.
- Image Preprocessing: Training limits inputs to 12 overlapping crops, increasing the testing limit to 25 for high-resolution text-rich images.The preprocessing follows Molmo and encodes crops with CLIP.
- Hyper Parameters: The best-performing model uses 2304-token sequences, learning rate 1e-6, batch size 32, 60K steps, and 200 warm-up steps.Training uses a cosine scheduler with end factor 0.1 on a single TPU v3-128.
- Ablations: The analyses compare chain-of-thought with short-answer prompts and evaluate supervised fine-tuning data combinations, including auxiliary, synthetic, and evaluation datasets.Table 7 defines the prompting comparison, while Table 8 distinguishes zero-shot and test-only evaluation settings.
B Additional Analysis
Additional analyses find that synthetic data consistently improves benchmark performance, while chain-of-thought helps tasks requiring multi-hop or mathematical reasoning. The study also evaluates synthetic pointing data and documents examples spanning multiple text-rich image categories.
- Synthetic Data: Synthetic data consistently boosts performance across benchmarks in both zero-shot and supervised settings.The analysis attributes this comparison to different combinations of supervised fine-tuning data.
- Chain-of-Thought Reasoning: Chain-of-thought improves ChartQA, TableVQA, and NutritionQA, while short-answer prompts perform better on the other five datasets.The reported explanation is that concise-answer annotation biases penalize verbose chain-of-thought responses.
- Document Pointing Task: DocPointQA contains 300 question-point pairs annotated from the DocVQA validation set and evaluates precision, recall, F1, and L2 distance.The task requires multiple-point answers and maps predicted points to ground truth following Molmo's setup.
- Document Pointing Task: Pointing models are compared using 155K human-annotated PixMo-point examples, 65K synthetic examples, or their combination.Table 9 reports zero-shot DocPointQA performance for these training-data configurations.