Source-linked AI summary
Data-Efficient Curation for Multimodal Reasoning under Fixed Training Protocols
Yosub Shin, Michael Buriek, Boris Sobolev, Pavel Bushuyeu, Vikas Kumar, Haoyang Xu, Samuel Watson, Igor Molybog
TL;DR
The paper asks which data-curation choices matter when multimodal fine-tuning uses a fixed model and training protocol. Using DCVLR, it evaluates aligned sources, model-relative difficulty, scale, diversity, and synthetic mixtures, finding that difficulty filtering on an aligned corpus is strongest among tested interventions. The resulting guidance is scoped to this fixed-recipe setting rather than universal data selection.
Problem
It is unclear which data-curation choices drive multimodal reasoning gains when architecture, training recipe, supervision, and data composition change together.
Method
The paper uses the fixed-protocol NeurIPS 2025 DCVLR challenge to compare aligned-source selection, model-relative difficulty filtering, dataset size, diversity heuristics, and rewritten synthetic mixtures.
Results
Difficulty filtering on an aligned source corpus is the strongest tested signal at matched scale, with the 1k submission raising aggregate DCVLR score from 38.4 to 46.0.
Takeaways & Limitations
Under the fixed recipe, select aligned data and a broad challenging-but-learnable region; treat additional size mainly as a stability control and test diversity or synthetic mixtures empirically.
Takeaways & Limitations
The conclusions are scoped to the fixed DCVLR-style recipe, aligned Walton starting corpus, model-relative difficulty signal, and several low-repetition exploratory checks.
Abstract
from arXiv · showhide
We study data curation for multimodal reasoning in a fixed-protocol fine-tuning regime, where the base model, optimizer, training schedule, and evaluation pipeline are held constant and the main degree of freedom is the training data. Using the NeurIPS 2025 Data Curation for Vision--Language Reasoning (DCVLR) challenge as a controlled testbed, we analyze how source-dataset alignment, model-relative difficulty, dataset size, diversity heuristics, and rewritten synthetic mixtures affect downstream reasoning accuracy. Among the tested interventions, difficulty filtering on an aligned source corpus provides the strongest gains at matched scale. The effect is not explained only by LiveXivTQA weighting: a per-benchmark decomposition shows that much of the improvement over random sampling comes from OlympiadBench, the largest non-LiveXivTQA benchmark. Qwen-derived difficulty scores also transfer to some additional model families, though the benefit is architecture-dependent. In contrast, increasing dataset size beyond roughly 1k aligned examples mainly reduces run-to-run variance under the fixed recipe, while the diversity and rewritten CoSyn mixtures we tested do not improve over the difficulty-filtered baseline. These results provide a scoped empirical recipe for data-constrained multimodal reasoning fine-tuning, rather than a universal claim about data selection across all training regimes.
1. Introduction
The paper isolates data curation by fixing the model and fine-tuning protocol, using DCVLR to test which training-data choices matter. A 1k-example aligned submission improved aggregate DCVLR score, with difficulty filtering emerging as the strongest tested signal.
- Motivation: Fixed-protocol DCVLR separates data-curation effects from changes in model architecture, training recipe, supervision, and data composition.The base model and training procedure remain fixed while the curated training set varies.
- Main result: 46.0 versus 38.4: the 1k-example submission improved the aggregate DCVLR score.The submission was derived primarily from the Walton Multimodal Cold Start corpus.
- Main result: Aligned-source model-relative difficulty filtering outperformed random, easy-only, and stricter hard-only selection at matched scale.The paper characterizes this as the strongest signal among the tested interventions.
- Scope: The controlled analysis also tests whether diversity and rewritten synthetic-data mixtures add benefit under the fixed recipe.These representative heuristics did not improve over the difficulty-filtered baseline.
2. Setting and Curation Method
The method starts from a source corpus aligned with the benchmark and base model, then scores examples by the base model’s stochastic-answer consistency. Additional diversity and synthetic-data interventions are evaluated as matched-scale ablations.
- Fixed-protocol benchmark: DCVLR fixes the model, optimizer, learning-rate schedule, training duration, and evaluation pipeline while varying the curated training set.The suite spans academic QA, mathematics, physics, and general visual reasoning.
- Starting from an aligned source: Walton is selected because it is aligned with the benchmark’s dominant academic-QA portion and the Qwen2.5-VL-7B-Instruct base model.Its embeddings lie closer to LiveXivTQA than the other tested organizer-provided sources.
- Model-relative difficulty: For each Walton example, Qwen2.5-VL-7B-Instruct generates 16 stochastic rollouts, and k counts the correct final answers.Correctness uses the staged matcher from the DCVLR evaluation pipeline.
- Model-relative difficulty: High-k examples are treated as easy, whereas low-k examples are treated as difficult because they expose model failures or instability.This defines difficulty relative to the base model rather than human problem difficulty.
- Model-relative difficulty: The main selection effect is coarse: eight rollouts produced the same qualitative filtering behavior, so 16 are not claimed necessary.The primary selected region is a broad thresholded pool rather than a fine ranking.
- Other curation interventions: Embedding balancing, category balancing or exclusions, and controlled CoSyn-400K mixtures are tested as matched-scale ablations.CoSyn examples are rewritten into longer reasoning traces while preserving their original final answers.
3. Experimental Setup
Experiments fine-tune Qwen2.5-VL-7B-Instruct with the official DCVLR recipe and evaluate the full benchmark suite. Matched subsets, repeated runs, and exploratory cross-model and curation checks support the comparisons.
- Main experiments: Qwen2.5-VL-7B-Instruct is fine-tuned with the official DCVLR training recipe and evaluated on all ten DCVLR benchmarks.Aggregate and per-benchmark accuracy are reported where relevant.
- Cross-model checks: Molmo-7B, Phi-4, and Gemma3-4B are trained on matched random and Qwen-score difficulty-filtered subsets for robustness checks.These tests examine whether Qwen-derived scores transfer across model families.
- Repetition and compute: Repeated runs use five repetitions for the random 1k Walton baseline and three repetitions per difficulty threshold.Dataset-size ablations use three to five repetitions depending on size, except the 10k endpoint.
- Curation pipeline: Figure 2 summarizes the pipeline from candidate data sources through difficulty scoring and optional diversity or synthetic-data interventions.The final process includes fixed-size sampling.
- Evaluation outputs: Table 1 reports per-benchmark accuracy for the base model and the curated 1k submission, establishing the competition outcome for later ablations.The table is organized around benchmark-level accuracy comparisons.
4. Results
At matched scale, difficulty filtering on aligned Walton data is the strongest tested intervention, while larger random datasets mainly improve stability and diversity or rewritten mixtures add no benefit over the filtered baseline.
- Competition Outcome: 38.4 to 46.0: the curated 1k submission improved aggregate DCVLR score over the base model.The large LiveXivTQA gain is consistent with Walton’s alignment, while held-out benchmarks show smaller or negative deltas.
- Difficulty Filtering: At fixed 1k scale, the moderate difficulty threshold outperformed random, easy-only, and stricter hard-only Walton subsets.Thresholded pools overlap; the result suggests that challenging-but-learnable examples are more useful than either reliably solved or nearly always failed examples.
- Benchmark Decomposition: +0.011 versus +0.004: OlympiadBench contributed more to the improvement over random sampling than LiveXivTQA.OlympiadBench is the largest non-LiveXivTQA benchmark, so the aggregate difference is not explained by LiveXivTQA weighting alone.
- Cross-Model Transfer: Difficulty-filtered subsets improved Molmo-7B and Phi-4 but not Gemma3-4B, indicating partial, model-dependent transfer of Qwen-derived difficulty scores.The reported Gemma3-4B accuracies were 0.307 for filtering and 0.316 for random sampling; these checks were not exhaustive multiseed studies.
- Dataset Size: 0.475 ± 0.008 at 1k versus 0.472 ± 0.004 at 5k: increasing random Walton subsets beyond roughly 1k did not reliably improve aggregate accuracy.Larger datasets reduced run-to-run variance, while benchmark trends offered no consistent evidence that scale alone expanded transfer; specialization patterns were suggestive rather than definitive because confidence intervals overlapped.
- Diversity and Synthetic Mixtures: 46.4 versus 45.3: difficulty filtering outperformed tested embedding clustering and topic exclusion on Open-R1, while combined diversity constraints and rewritten CoSyn mixtures did not beat the filtered baseline.The CoSyn experiment covered a specific source-pool mixture rewritten into longer reasoning traces, so its negative result is scoped to that configuration.
5. Practical Guidance and Scope of Claims
For fixed-protocol fine-tuning, the paper recommends aligned sources, model-relative difficulty filtering, and matched-scale tests of diversity and synthetic augmentation. Its conclusions are scoped: dataset-size gains saturate in mean accuracy, and the findings do not establish universal rules across training regimes.
- Practical Guidance: Choose the most aligned available source corpus before applying curation signals.Source alignment is presented as a prerequisite for efficient gains from small curated subsets.
- Practical Guidance: Select a broad challenging-but-learnable difficulty region rather than only the hardest failures.The paper recommends model-relative difficulty filtering because easy examples may provide little learning signal and near-always-failed examples may be unstable or poorly matched.
- Practical Guidance: Treat dataset size as a stability knob once mean performance saturates under the fixed recipe.The size ablation caption reports plateauing mean aggregate accuracy and reduced variance with larger datasets.
- Practical Guidance: Evaluate diversity and synthetic augmentation through matched-scale ablations rather than assuming they compose additively with difficulty filtering.The tested diversity heuristics and rewritten CoSyn mixtures did not improve over the difficulty-filtered baseline.
- Scope of Claims: The central claim is limited to difficulty being strongest among tested interventions after choosing an aligned source under the fixed DCVLR-style recipe.The paper explicitly rejects universal claims about difficulty, diversity, synthetic data, or scale across all multimodal training stages.
A. Comparison of DCVLR Submission Strategies
The reported DCVLR curation strategies were reconstructed from public competition write-ups and system cards, with teams anonymized by final ranking and no private communications used.
- Comparison of DCVLR Submission Strategies: Table 7 derives curation-strategy details from publicly available DCVLR competition materials.The sources include participating teams’ leaderboard write-ups and system description cards.
- Comparison of DCVLR Submission Strategies: Teams are anonymized according to final ranking, and the analysis uses no private communications.
B. Alignment and Pipeline Details
The paper evaluates source alignment through neighborhood similarity and finds that Walton-like LiveXivTQA questions are easier for the base model. This supports Walton’s alignment with both the benchmark and the base model.
- Alignment: For each LiveXivTQA example, k = 32 nearest neighbors are computed from Walton and MM-Open-R1 embeddings using Qwen2.5-VL-7B-Instruct.Neighbor-source fractions are used to characterize how Walton-like each question is.
- Alignment: Base-model accuracy on LiveXivTQA is higher for questions with greater Walton coverage among their nearest neighbors.Questions are binned by the fraction of Walton neighbors.
- Alignment: The alignment analysis supports Walton as a source aligned with both LiveXivTQA and the base model.The figure reports absolute LiveXivTQA accuracy against Walton coverage.
C. Implementation and Reproducibility Details
The experiments use stochastic difficulty scoring, staged answer matching, controlled CoSyn rewriting and mixing, standardized preprocessing, and repetition schemes under the official protocol. The paper also reports compute requirements, planned release artifacts, and broader evidence that top DCVLR submissions favor difficulty-aware curation.
- Scoring: Difficulty scores use 16 rollouts at temperature 0.7 and top-p 0.9, while eight-rollout checks show similar coarse filtering behavior.The full scoring pass requires approximately 16× inference over the Walton corpus.
- Evaluation: Correctness is evaluated through exact normalized matching, structured answer extraction, and an LLM judge for unresolved responses.The structured stage includes LaTeX, mathematical, symbolic-equivalence, natural-language, multiple-choice, and boolean handling.
- Synthetic mixtures: CoSyn examples are uniformly sampled from the validation split, rewritten with GPT-4o using the image, question, and ground-truth answer, then mixed with difficulty-filtered Walton.Rewritten traces preserve the original question, image, and final answer while serving as training targets.
- Preprocessing: Final datasets use the organizer’s preprocessing tool for deduplication, length filtering, and format normalization.CoSyn image resizing occurs before final dataset construction, with images limited to 768 × 768 and aspect ratio preserved.
- Reproducibility: Repetition counts vary across random sampling, difficulty thresholds, dataset sizes, transfer checks, CoSyn mixtures, and diversity variants.Dataset-size ablations use four repetitions at 250 examples, five at 1k, four at 5k, three at 7.5k, and one at 10k.
- Compute and release: Full-corpus difficulty scoring required approximately 200 GPU-hours on A100-class GPUs, and fine-tuning followed the official DCVLR protocol.The authors plan to release code, scores, seeds, and curated-data identifiers where licensing permits.
- Submission context: Most top DCVLR submissions favor difficulty-aware filtering with expert reasoning distillation over large-scale synthetic generation.This comparison is summarized in Table 7’s curation-strategy overview.
D. Clustering and Category-Balancing Details
The Walton curation procedure combines Qwen2.5-VL-7B-Instruct embeddings, k-means clustering, randomized square-root-proportional sampling, and Mathematics Subject Classification-based balancing. The experiments vary clustering, cap, selection, exclusion, and weighting settings to control dataset composition.
- Clustering: Walton examples are embedded with Qwen2.5-VL-7B-Instruct and grouped using k-means clustering.
- Clustering: Sampling uses square-root-proportional allocation with per-cluster caps and Dirichlet-based randomization.
- Clustering: The study sweeps cluster granularity, per-cluster caps, and selection ratios.
- Category balancing: Each example receives a coarse Mathematics Subject Classification category, whose distribution is shown for the Walton dataset in Figure 8.
- Category balancing: Category-balancing experiments vary category exclusions, per-domain caps, and log-weighting.