PaperlayerView exact revision on arXiv

Source-linked AI summary

VisionFoundry: Teaching VLMs Visual Perception with Synthetic Images

Guanyu Zhou, Yida Yin, Wenhao Chai, Shengbang Tong, Xingyu Fu, Zhuang Liu

arXiv:2604.09531v1cs.CVcs.AIcs.CL

TL;DR

VLMs lack systematic low-level visual supervision, motivating targeted synthetic data without reference images or human annotation. VisionFoundry generates and verifies task-specific VQA data, and VisionFoundry-10K improves visual perception benchmarks while preserving broader capabilities. The study leaves benefits for more complex visual reasoning as an open question.

  • Problem

    Natural image datasets may provide limited systematic supervision for low-level visual skills, motivating targeted synthetic supervision without reference images or expensive human annotation.

  • Method

    VisionFoundry uses task names to generate questions, answers, and T2I prompts, synthesize images, and automatically verify alignment; VisionFoundry-10K contains 10k triples across 10 tasks.

  • Results

    Training across multiple VLMs consistently improves visual perception benchmarks, including +7% on MMVP and +10% on CV-Bench-3D, while preserving broader capabilities and scaling favorably with data size.

  • Takeaways & Limitations

    The results suggest that limited task-targeted supervision contributes to the perception bottleneck and that targeted synthetic data curation can meaningfully alleviate part of it.

  • Takeaways & Limitations

    VisionFoundry-10K focuses on visual perception, leaving its effectiveness for complex visual reasoning requiring longer inference chains and stronger compositional reasoning open.

Abstract

from arXiv · show

Vision-language models (VLMs) still struggle with visual perception tasks such as spatial understanding and viewpoint recognition. One plausible contributing factor is that natural image datasets provide limited supervision for low-level visual skills. This motivates a practical question: can targeted synthetic supervision, generated from only a task keyword such as Depth Order, address these weaknesses? To investigate this question, we introduce VisionFoundry, a task-aware synthetic data generation pipeline that takes only the task name as input and uses large language models (LLMs) to generate questions, answers, and text-to-image (T2I) prompts, then synthesizes images with T2I models and verifies consistency with a proprietary VLM, requiring no reference images or human annotation. Using VisionFoundry, we construct VisionFoundry-10K, a synthetic visual question answering (VQA) dataset containing 10k image-question-answer triples spanning 10 tasks. Models trained on VisionFoundry-10K achieve substantial improvements on visual perception benchmarks: +7% on MMVP and +10% on CV-Bench-3D, while preserving broader capabilities and showing favorable scaling behavior as data size increases. Our results suggest that limited task-targeted supervision is an important contributor to this bottleneck and that synthetic supervision is a promising path toward more systematic training for VLMs.

1 Introduction

VisionFoundry addresses limited low-level visual supervision by generating task-targeted synthetic VQA data without reference images or human annotation. Training on this data improves visual perception while preserving broader capabilities.

  • Natural image datasets may not systematically cover the combinatorial variations required for robust visual perception.
  • Synthetic data offers on-demand, controllable coverage of entity, attribute, relation, viewpoint, and style combinations that are rare in natural corpora.
  • VisionFoundry generates task-conditioned questions, answers, and T2I prompts, synthesizes images, and filters samples using multimodal verification without reference images or manual labeling.
  • VisionFoundry-10K contains 10k image–question–answer triples spanning 10 low-level visual perception tasks.The dataset uses a controlled setting intended to isolate the contribution of synthetic images to visual capability learning.
  • +7% on MMVP and +10% on CV-Bench-3D were reported after finetuning three open-source VLMs on VisionFoundry-10K.Broader capabilities were preserved, and performance increased predictably as synthetic data size grew.
  • The results suggest that limited task-targeted supervision contributes to VLM perception bottlenecks and that targeted synthetic curation can alleviate part of this weakness.

2 VisionFoundry

VisionFoundry is a task-aware pipeline that constructs visually grounded VQA supervision from task specifications, then synthesizes and verifies images automatically. Its design emphasizes controllability, visual determinism, and alignment filtering.

  • VisionFoundry produces VQA supervision using only task specifications, without reference images, human-written QA annotations, or real image–caption pairs.
  • The pipeline is organized around controllability, visual determinism, and verification.These principles target systematic capability coverage, answer-determining visual content, and removal of misaligned samples.
  • VisionFoundry generates questions, answers, and T2I prompts, synthesizes images, and filters them with a multimodal judge.
  • VQA Triplet Generation: Task-aware generation samples entities across objects, attributes, scenes, styles, and task-specific dimensions to support systematic coverage.
  • VQA Triplet Generation: The LLM creates visually answerable questions, deterministic answers, and detailed prompts encoding the answer-determining facts.It is instructed to use visually verifiable properties and avoid hidden or commonsense assumptions.
  • Image Synthesis: Prompt-conditioned image synthesis specifies objects, attributes, spatial arrangements, scene context, and visual style while treating the T2I model as a black box.
  • Image Synthesis: Failed images may receive limited localized refinement without changing their question–answer pairs.
  • Alignment Verification: Verification converts each question and candidate answer into a declarative visual statement and retains only judge-confirmed aligned samples.

3 VisionFoundry-10K Dataset

VisionFoundry-10K is a fully synthetic, automatically verified VQA dataset designed to target low-level visual perception through controlled task-specific supervision. It contains 10k balanced examples across 10 perception tasks rather than maximizing semantic or linguistic coverage.

  • Dataset scope: VisionFoundry-10K contains 10k image–question–answer triples across 10 visual perception tasks, with 1k cases per task.The tasks include spatial understanding, relative depth, viewpoint variation, and other low-level skills.
  • Construction: For each task, sampled entity configurations constrain objects, attributes, and relational predicates before generating deterministic questions, answers, and T2I prompts.The prompt encodes the visual facts that determine the answer.
  • Construction: Exactly 1k verifier-accepted samples are retained per task, producing a balanced dataset of 10k examples with concise questions and answers.Answers can be binary, categorical, or short phrases, supporting reliable finetuning and evaluation.
  • Design goal: The dataset prioritizes precise visual grounding and controlled difficulty over semantic coverage or linguistic diversity.This positioning is intended to diagnose and improve core perceptual weaknesses of VLMs.
  • Validation: Training on VisionFoundry-10K yields consistent improvements on multiple visual perception benchmarks.The experiments are presented as validation of targeted synthetic supervision for low-level visual understanding.

4 Experiments

Experiments evaluate whether VisionFoundry improves visual perception across VLMs, preserves broader capabilities, and scales with synthetic data. Results show consistent perception gains, benchmark-dependent broader-task changes, upward data-size trends, and task-specific transfer effects.

  • Main results: Across three open-source VLMs, VisionFoundry training consistently improves visual perception benchmarks while general-purpose changes remain mixed and benchmark-dependent.The study evaluates Qwen2.5-VL-3B-Instruct, Llama-3.2-11B-Vision-Instruct, and MiMo-VL-7B-SFT across 13 benchmarks.
  • Data-size sensitivity: At 10k samples, MMVP-pair improves from 35.3 to 42.0 (+6.7), while CV-Bench-3D improves from 66.0 to 76.5 (+10.5).The overall trend is upward as the synthetic data budget increases, despite mild non-monotonicity in some curves.
  • Data mixture: Equal-sized mixtures of VisionFoundry-10K and natural data consistently outperform size-matched natural-only subsets on visual perception benchmarks while maintaining comparable general-purpose performance.General-purpose results show slight fluctuations and partial improvements.
  • Epoch trade-off: With a single-task 1k subset, performance typically saturates after around 8 epochs, whereas the full 10-task dataset reaches convergence in fewer epochs.At 16 epochs, some single-task benchmarks show slight regressions.
  • Task-wise analysis: Task transfer is benchmark-dependent: Depth Order helps spatial benchmarks but has limited or negative transfer on ScreenSpot-Pro and MMMU.OCRBench shows small consistent drops because the synthetic task set lacks OCR-focused supervision.
  • Task-wise analysis: Cross-task response divergence is stronger on benchmarks sensitive to the selected synthetic capability, while RealWorldQA shows relatively small divergence because it emphasizes reasoning.The analysis indicates that expanding task diversity may broaden gains across benchmarks.

5 Related Work

Related work uses synthetic multimodal data for captioning, instruction tuning, representation learning, classification, compositional generalization, failure-driven synthesis, and robustness analysis. VisionFoundry is distinguished by generating datasets automatically from task keywords alone.

  • Synthetic data for VLMs: Prior synthetic multimodal pipelines scale captioning and reasoning supervision, including ShareGPT4V’s 1.2M captions and ALLaVA’s 3.4M captioning and reasoning QA pairs.Other work refines captions or enables self-recaptioning for vision–language learning.
  • Synthetic data for VLMs: Synthetic-data research also covers compositional generalization, failure-driven synthesis, and robustness analysis.These efforts extend synthetic supervision beyond caption generation and instruction tuning.
  • VisionFoundry positioning: VisionFoundry differs from prior work by generating datasets from task keywords alone through a fully automated pipeline.Its distinguishing input is a task keyword rather than reference images or manual dataset construction.
  • T2I-generated training data: T2I-generated data has been used for representation learning, VLM training, scaling-law studies, and supervised classification.This work includes synthetic-only versus mixed training and filtered diffusion samples.
  • VLM training and evaluation: Vision-language instruction-tuning systems convert image–text corpora into instruction-following data to align perception and generation.Representative pipelines include LLaVA, InstructBLIP, MiniGPT-4, and Qwen-VL.

6 Conclusions and Discussion

VisionFoundry-10K provides an initial study of targeted synthetic supervision for visual perception, while leaving its extension to more complex visual reasoning for future work.

  • VisionFoundry-10K focuses on visual perception tasks including spatial understanding, attribute recognition, and viewpoint reasoning.
  • Limitations and Future Work: The study leaves open whether VisionFoundry can benefit complex visual reasoning requiring longer inference chains and stronger compositional reasoning.
  • Appendix: The appendix documents implementation details, prompt specifications, verifier diagnostics, task-wise results, and controlled ablations supporting the main paper.
  • Appendix: The appendix also studies verification necessity and compares natural-image with synthetic-image supervision under controlled settings.

A Implementation Details

The implementation uses fixed model-specific training settings and structured prompts to generate, verify, and optionally regenerate synthetic VQA samples under a controlled evaluation protocol.

  • Optimization and Hyperparameters: Main experiments use model-specific hyperparameters, global batch size 128, and one training epoch.ηViT, ηadapter, and ηLLM denote learning rates for the ViT encoder, adapter modules, and LLM backbone.
  • Supplementary Protocols: Supplementary protocols vary data size, compare synthetic with natural data, and alter training epochs or task specialization while holding other settings constant.Data-size subsets contain 0.5k, 1k, 2k, and 5k samples; synthetic and natural subsets use seed 42.
  • Exact Pipeline Prompts: Generation prompts expand task attributes into controlled candidate pools and jointly produce a T2I prompt, question, deterministic answer, and metadata.Conditional instructions adapt generation to spatial, color, object-count, and multi-image requirements.
  • Exact Pipeline Prompts: The pipeline converts each QA pair into a concise declarative statement for visual verification and accepts only exact YES outputs from the binary judge.The verifier ignores minor details and evaluates whether the image matches the statement overall.
  • Exact Pipeline Prompts: Failed verification can trigger minimal edit-based regeneration, with retries and retention requiring approval.The edit instruction changes only what is needed to match the verification statement.
  • Verifier Diagnostics: Manual audits classify samples by generation correctness and verifier judgment, producing true-positive, false-negative, false-positive, and true-negative outcomes.

C.1 True Positive Cases

The case studies cover correct accepted generations across visual attributes, relative depth, and viewpoint recognition, alongside representative verifier errors and correct rejections.

  • True Positive Cases: Correct accepted cases include glossiness judgment, relative depth comparison, and high-viewpoint recognition.Examples ask whether a snowbank appears glossy, which object is closer, and whether a statue is viewed from above.
  • False Negative Cases: A false negative occurs when the forklift correctly faces the camera but the verifier incorrectly rejects the image.
  • False Positive Cases: A false positive occurs when the sailboat’s bow points right instead of left but the verifier accepts the incorrect sample.
  • True Negative Cases: A true negative occurs when the shopping cart faces lower-right instead of the required lower-left and the verifier rejects it.
  • Audit Setup: The diagnostic analysis tracks a full random-sampling generation trajectory rather than only retained outputs.This design evaluates whether high-quality supervision emerges systematically rather than by chance.

D.1 Verification-Accuracy Records

The verification audit compares verifier decisions with manual correctness labels using a confusion matrix and scalar quality metrics from one 140-attempt trajectory.

  • Audit Results: 140 audited attempts produced 100 retained samples: 99 valid samples and 1 false accept, with 40 candidates rejected.The outcome shares were 70.7% correct-and-pass, 7.1% correct-and-fail, 0.7% wrong-and-pass, and 21.4% wrong-and-fail.
  • Audit Results: The confusion matrix places manual correctness labels in rows and verifier outputs in columns.
  • Verifier Metrics: Verifier performance reached 99.0% precision and 94.7% F1, while recall was 90.8%.The audit characterizes the verifier as a high-precision filter rather than a perfect selector.
  • Pipeline Quality: The pipeline retained 71.4% of candidates, with a 70.7% final valid proportion, 3.2% false accepts, and 9.2% false rejects.Observed Cohen’s κ was 0.794, consistent with substantial agreement.
  • Experimental Controls: Task-wise experiments use one epoch with identical training and evaluation settings so differences reflect task specialization rather than protocol changes.

E.2 Complete Per-task Results

The complete per-task results report performance for ten task-specialized models across broad multimodal benchmarks. Scores are presented as percentages, with higher values indicating better performance.

  • Task-wise Results: Ten task-specialized models are evaluated across the reported multimodal benchmarks.The task abbreviations include OaD, VaP, PaRC, SR, SaC, SPC, CaA, DO, RD, and RWSU.
  • Benchmark Coverage: Table 5 reports scores on MVP-S, MVP-P, CV2D, CV3D, RWQA, BLK, MMS, and OCR.All reported values are percentages, and higher scores are better.
  • Benchmark Coverage: Table 6 reports scores on LEGO, MMMU, MMB, MVis, SSP, MMSI, and 3DSR.All reported values are percentages, and higher scores are better.

F Ablation on Verification Necessity

The verification ablation tests whether verification contributes performance under a fixed 1k-sample budget and identical finetuning conditions. The verified pipeline outperforms the non-verified variant, especially on vision-focused evaluations.

  • Protocol: A controlled ablation isolates verification using a fixed training size of 1k samples.The experiment compares a baseline, finetuning with verification, and finetuning without verification.
  • Protocol: Identical finetuning recipes are used for the verified and non-verified models.Both models use the same backbone, trainable components, learning rates, and one-epoch training setup.
  • Results: +2.6 points on MathVista-Mini is the largest reported improvement of verification over non-verification.Verification also improves CV-Bench-2D by +0.5 points, RealWorldQA by +0.7 points, BLINK by +1.2 points, and OCRBench by +0.3 points.
  • Results: The verified run improves over baseline on four of five benchmarks, while the non-verified run falls below baseline on three.The reported below-baseline results for the non-verified run are BLINK, MathVista-Mini, and OCRBench.
  • Interpretation: Verification produces its clearest gains on vision-focused benchmarks, while differences on other benchmarks are comparatively small.The ablation therefore supports verification as a necessary component of VisionFoundry under the tested conditions.

G Ablation on the Synthetic Process

This ablation compares natural images with synthetic QA against VisionFoundry’s synthetic images with synthetic QA, then controls for QA construction by matching captions and questions. Synthetic-image training performs better overall, with especially clear gains on visual perception benchmarks.

  • Practical Comparison: The practical comparison asks whether gains come from synthetic QA alone or from the full synthetic image-and-QA process.The initial comparison uses real images with synthetic QA versus synthetic images with synthetic QA.
  • Practical Comparison: Synthetic images with synthetic QA outperform natural images with synthetic QA on all five reported benchmarks.The improvements are +0.8 points on CV-Bench-2D, +5.3 points on CV-Bench-3D, +0.9 points on RealWorldQA, +0.6 points on BLINK, and +0.8 points on MMMU-Val.
  • Practical Comparison: The synthetic-image setting remains broadly competitive on general-purpose evaluation while providing larger gains on visual perception metrics.CV-Bench-3D shows the especially clear gain in the initial comparison.
  • Strict Control: The strict-control variant matches synthetic QA across branches and changes only the image source.The natural branch uses natural images, while the synthetic branch replaces the image source through caption-conditioned T2I synthesis.
  • Strict Control: +2.7 points on CV-Bench-3D is the largest listed strict-control improvement for synthetic images with synthetic QA over natural images with synthetic QA.The other reported improvements are +0.2 points on CV-Bench-2D, +0.8 points on RealWorldQA, +2.1 points on MMMU-Val, and +0.6 points on ScreenSpot-Pro.
  • Strict Control: The strict-control results show a consistently stronger synthetic-image frontier, especially on visual perception evaluations.The reported findings attribute additional training value to the synthetic process beyond caption-derived synthetic QA alone.
Loading 2604.09531v1…