Source-linked AI summary
FineVision: Open Data Is All You Need
Luis Wiedmann, Orr Zohar, Amir Mahla, Xiaohan Wang, Rui Li, Thibaud Frere, Leandro von Werra, Aritra Roy Gosthipaty, Andrés Marafioti
TL;DR
Public VLM datasets are fragmented, inconsistent, and contaminated, limiting reproducible data-centric research. FineVision builds and releases a 24-million-sample corpus through semi-automated curation with human oversight, and models trained on it consistently outperform existing open-data mixtures across broad evaluations.
Problem
Fragmented, inconsistent, and contaminated public VLM datasets limit reproducibility and robust data-centric research.
Method
FineVision unifies over 200 sources through semi-automated ingestion, schema mapping, human audits, task-aware conversion, duplicate control, and benchmark decontamination.
Results
Models trained on FineVision consistently outperform those trained on existing open datasets across a broad suite of benchmarks, with gains persisting after test-set decontamination.
Takeaways & Limitations
FineVision provides an open, curated foundation for reproducible VLM training and evaluation, including broader GUI and agentic capabilities through a unified action space.
Takeaways & Limitations
Residual overlaps may persist, long-context and multi-document reasoning remain challenging, and GUI benchmarks are not integrated into the standard evaluation stack.
Abstract
from arXiv · showhide
The advancement of vision-language models (VLMs) is hampered by a fragmented landscape of inconsistent and contaminated public datasets. We introduce FineVision, a meticulously collected, curated, and unified corpus of 24 million samples - the largest open resource of its kind. We unify more than 200 sources into 185 subsets via a semi-automated, human-in-the-loop pipeline: automation performs bulk ingestion and schema mapping, while reviewers audit mappings and spot-check outputs to verify faithful consumption of annotations, appropriate formatting and diversity, and safety; issues trigger targeted fixes and re-runs. The workflow further applies rigorous de-duplication within and across sources and decontamination against 66 public benchmarks. FineVision also encompasses agentic/GUI tasks with a unified action space; reviewers validate schemas and inspect a sample of trajectories to confirm executable fidelity. Models trained on FineVision consistently outperform those trained on existing open mixtures across a broad evaluation suite, underscoring the benefits of scale, data hygiene, and balanced automation with human oversight. We release the corpus and curation tools to accelerate data-centric VLM research.
1 Introduction
FineVision addresses fragmented, inconsistent, and contaminated public VLM datasets with a large, curated, openly released corpus. Its human-supervised curation pipeline produces stronger open-data VLM results across 11 benchmarks.
- FineVision responds to fragmented, inconsistent, and contaminated public datasets that hinder reproducibility and robust data-centric VLM research.
- FineVision unifies over 200 sources into a curated corpus with automated ingestion and schema mapping plus reviewer audits and spot-checks.The workflow also cleans data, verifies annotations and formatting, and triggers targeted fixes or re-runs when issues arise.
- 27.7% over The Cauldron, 11.4% over Cambrian-7M, and 38.4% over LLaVA-OneVision on an average of 11 benchmarks.These are reported relative improvements for models trained on FineVision over the respective open-data baselines.
- The release includes FineVision and associated resources intended to broaden public access to high-quality training data for open VLM development.
2 FineVision Curation
FineVision converts heterogeneous public datasets into standardized conversational supervision through semi-automated processing and targeted human review. Its curation includes task-aware conversion, cleaning, duplicate control, and contamination screening.
- FineVision unifies over 200 public datasets into 185 subsets after filtering and deduplication.
- Reviewers inspect mapping plans, dry runs, and random outputs, then request focused fixes or re-runs when annotations, formatting, or diversity are inadequate.
- All datasets converge to a standardized conversational representation, with templates converting nonconversational annotations into QA pairs and preserving task-specific metadata.
- Task-specific strategies preserve supervision across visual QA, captioning, grounding, document understanding, OCR, classification, and detection while varying templates for stylistic diversity.
- GUI data are mapped into a coherent typed action schema with normalized coordinates, enabling cross-domain training across desktop, mobile, and browser environments.
- Cleaning removes corrupted images and malformed text, while SSCD-based processing clusters near-duplicates within FineVision and supports contamination control.
3 Exploring FineVision
FineVision is characterized by turn quality across four axes, dataset category profiles, and visual diversity. Its datasets combine broad conceptual coverage with balanced representation, while preserving breadth supports downstream generalization.
- 3.2 Analysis of Characteristic Axes: 97.2% of turns score 4 or 5 for Formatting, while more than 85% score 4 or 5 for Relevance.These text-based axes indicate well-formed, on-topic question–answer pairs.
- 3.2 Analysis of Characteristic Axes: Chart & Table has high Image–Question Correspondence, whereas Captioning and General VQA combine high Visual Dependency with strong Formatting and Relevance.Naive OCR is highly visually dependent but lower on the other axes, while Mathematics scores lower across all four.
- 3.2 Analysis of Characteristic Axes: Formatting and Relevance trend together, while Visual Dependency and Image–Question Correspondence are inversely correlated across dataset profiles.Grounding scores highly on the text-based axes, contrasting with Naive OCR’s high visual dependency and lower Formatting and Relevance.
- 3.3 Visual Diversity: FineVision releases per-turn scores for analysis and reweighting, and experiments find that preserving breadth yields the best downstream generalization.The corpus is characterized along category composition, turn quality, and visual diversity.
- 3.3 Visual Diversity: FineVision and Cambrian occupy a high-diversity tier with greater effective rank than Cauldron and LLaVA, indicating broader conceptual coverage.The narrower scope of Cauldron and LLaVA may limit the world knowledge of models trained on them.
- 3.3 Visual Diversity: FineVision matches Cambrian’s high effective rank but has a substantially higher participation ratio, making its conceptual coverage more uniform.The comparison indicates that scale alone does not explain diversity; curation strategy is also critical.
4 Experiments and Results
Experiments compare FineVision-trained models with models trained on three open datasets across 11 benchmarks and assess contamination and GUI capabilities. FineVision achieves the strongest average benchmark performance and retains more performance after decontamination.
- 4.1 Experimental Setup: The evaluation uses a 460M-parameter SmolVLM trained for 20,000 steps and assesses performance on 11 diverse benchmarks.Baselines are The Cauldron, LLaVA-OneVision, and Cambrian-7M.
- 4.2 Main Results: FineVision achieves the highest average performance across all 11 benchmarks, improving 10.8 pp over The Cauldron, 5.1 pp over Cambrian-7M, and 13.8 pp over LLaVA-OneVision.The model initially lags during the first few thousand steps but surpasses the other models after approximately one epoch.
- 4.2 Main Results: FineVision’s contamination rate is 1.02%, compared with 2.15–3.05% for the baseline datasets.After decontamination, baseline performance drops by 2.7–3.7 pp, while FineVision drops by only 1.6 pp.
- 4.2 Main Results: FineVision shows the smallest post-decontamination performance drop at 1.6 pp, whereas baselines degrade by roughly 2.7–3.7 pp.This indicates that FineVision’s gains are not explained by contamination.
- 4.3 New GUI Capabilities: FineVision includes GUI and agentic data, enabling comparison on ScreenSpot-V2 and ScreenSpot-Pro before and after fine-tuning on the aguvis-stage-1 subset.Table 2 reports that the FineVision-trained model achieves comparable results to an architecturally equivalent model four times its size.
5 Related Work
Related work progresses from early human-curated and aggregated instruction datasets toward larger meta-datasets, synthetic generation, and GUI or embodied vision resources. FineVision belongs to this broader shift toward scalable multimodal data and action-oriented capabilities.
- Large-scale multimodal data generation pipelines: Synthetic and multi-expert generation pipelines address the scalability limits of human annotation by creating large multimodal datasets.LLaVA-Instruct-150K is presented as an early example of GPT-4-generated multimodal instructions guided by BLIP- or CLIP-style embeddings.
- Meta-datasets for multimodal instruction tuning: Early multimodal instruction efforts established human curation and aggregation, while later collections expanded task, dataset, image, and dialogue-turn coverage.Examples include MultiInstruct, InstructBLIP, Vision-FLAN, Cambrian-10M, The Cauldron, and LLaVA-OneVision.
- GUI and embodied vision datasets: GUI and embodied vision datasets connect visual perception to action through cross-platform screenshots, GUI elements, and sequence-modeling or function-call interfaces.OS-Atlas and ShowUI represent this newer frontier of agentic vision data.
6 Conclusion
FineVision is an open, rigorously curated dataset and reproducible toolset for training vision-language models. Models trained on it outperform existing open datasets across broad benchmarks, while remaining limitations include residual overlap and difficult reasoning settings.
- FineVision unifies over 200 public sources into a standardized conversational schema spanning captions, VQA, document understanding, OCR, grounding, and GUI interaction.It is presented as a large-scale, open, rigorously curated dataset for vision-language models.
- Its semi-automated, human-in-the-loop pipeline combines systematic cleaning, near-duplicate control, and SSCD-based benchmark decontamination.These measures are intended to provide reproducible and hygienic training data.
- Models trained on FineVision consistently outperform those trained on existing open datasets across broad benchmarks, with gains persisting after test-set decontamination.The dataset also broadens capabilities, particularly in GUI and agentic settings through a unified action space.
- The release includes the dataset, conversion recipes, de-duplication tools, and precomputed embeddings to support transparent, repeatable research.
- Residual overlaps may persist, while long-context and multi-document reasoning remain challenging and GUI-control benchmarks are not integrated into the standard evaluation stack.Future audits will strengthen licensing provenance, privacy, and bias checks.
A.1 Duplicate Cluster Visualization
The duplicate-detection visualization illustrates why one similarity threshold must balance false positives against false negatives. Manual tuning selected τ = 0.95.
- τ = 0.95 was selected after manual tuning as a compromise between false-positive and false-negative duplicate matches.
- The visualization contrasts true photographic duplicates, false positives from templated charts, and false negatives involving hand drawings.
- Each panel shows a query image alongside retrieved matches and their similarity scores.
A.2 Quality Ratings
The quality-rating prompts score visual dependence, image–question correspondence, answer quality, and text formatting on five-point scales. Each evaluator is instructed to return only a numeric score.
- Answer ratings range from 1 for inadequate or unintelligible responses to 5 for direct, complete, accurate answers.
- Formatting ratings range from 1 for severe corruption or garbling to 5 for clean, professional, well-structured text.
- Quality ratings assess visual dependence, image–question correspondence, answer quality, and formatting quality using five-point scales.
- Visual-dependence ratings range from 1 for nonvisual questions to 5 for questions requiring specific visual details.
- The prompts require evaluators to respond directly with only the numeric score and provide the question–answer pair as context where specified.
- Image–question correspondence ratings range from 1 for no match to 5 when the image directly contains the needed elements.
A.3 Action Space
FineVision's GUI data uses a unified action-space schema with categories and typed arguments. The schema addresses heterogeneous action representations across source datasets.
- Action-space unification addresses heterogeneous function signatures, parameter naming conventions, and action taxonomies across GUI datasets.
- The unified action-space schema organizes GUI actions into categories with typed arguments.
A.4 Data Quality Filtering
Prompt-based quality scores did not improve model performance when used as data filters, whereas combined filtering performed best when training on the full dataset. FineVision’s category statistics characterize how sequence length, interaction turns, and image resolution vary across supervision types.
- Prompt-based quality scores did not yield an effective filtering scheme because thresholding generally failed to improve performance over unfiltered training.The authors caution that this negative result applies to the specific prompt-based metrics tested, not to all quality estimators or filtering strategies.
- Combined filtering achieved its best performance when models were trained on the full dataset rather than on threshold-selected subsets.
- FineVision’s sequence-length distributions separate perceptual/extractive, descriptive-generation, and transcription task archetypes.Questions are generally short, whereas answers are broader and often heavy-tailed; Naive OCR’s long tail reflects fidelity demands rather than inference.
- Naive OCR and Captioning show the largest answer-minus-question token gaps, while Mathematics and Chart & Table have minimal or negative gaps.Reported median gaps include 104.15 for Naive OCR, 203.47 for Captioning & Knowledge, 1.36 for Mathematics, and −19.26 for Chart & Table.
- Chart & Table and Grounding & Counting support more multi-turn interactions per image, while document-centric OCR QA uses higher image resolutions to preserve legibility.Figure 12 reports category differences in turns per sample, and the dataset statistics report higher post-resizing resolutions for OCR QA.