PaperlayerView exact revision on arXiv

Source-linked AI summary

DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

Jiahao Xie, Zhongbin Guo, Qianle Wang, Ruiqi Lu, Dongling Xiao, Wanxuan Sun, Cheng Yang

arXiv:2607.24516v1cs.CVcs.AI

TL;DR

VLM data-mixture construction remains largely heuristic, with intuitive ratios and limited assessment of difficulty and diversity. DecoupleMix separates inter-class allocation from intra-class composition using iterative search, dataset-level scoring, and constrained convex optimization. It consistently outperforms heuristic baselines, transfers ratios across scales without retuning, and matches Qwen3-VL-4B at 62.5 overall average with 80B additional multimodal tokens.

  • Problem

    VLM mixture construction relies on intuitive ratios, brute-force stacking, and quality filtering that inadequately assesses data difficulty, diversity, and redundancy.

  • Method

    DecoupleMix decouples inter-class allocation and intra-class composition using iterative search, dataset-level quality–difficulty assessment, and constrained convex optimization with diversity control.

  • Results

    62.5 overall average matches Qwen3-VL-4B, while ratios transfer across larger data and model scales without retuning and the recipe consistently outperforms heuristic baselines.

  • Takeaways & Limitations

    Systematic data curation provides a scalable methodology for constructing high-performance VLM pretraining recipes.

  • Takeaways & Limitations

    The study does not establish scaling trends at the much larger token budgets used in frontier industrial pretraining.

Abstract

from arXiv · show

While data curation for Vision Language Models (VLMs) is increasingly active, public practice for constructing pretraining mixtures remains largely heuristic: practitioners stack datasets that pass quality filters, set cross-domain ratios by intuition, and lack a principled, attributable criterion for admitting new data, while frontier recipes remain undisclosed. We formulate data construction as a systematic mixture-optimization problem and turn it into a reproducible engineering discipline by decoupling the mixture into two orthogonal sub-problems: inter-class ratios across capabilities and intra-class ratios within a category. For inter-class allocation, we use a single-variable iterative search; for intra-class composition, we apply a multidimensional, dataset-level assessment scoring Quality and Difficulty, and formulate selection as a constrained convex optimization with a diversity objective. The DecoupleMix framework delivers two critical capabilities: guiding what data to collect next and rendering dataset validation a controlled, attributable experiment. Experiments show our approach consistently surpasses heuristic baselines. Moreover, optimal ratios discovered on small-scale proxies transfer seamlessly to larger scales without retuning. Using 80B additional multimodal continue-pretraining tokens, our VLM is competitive with strong open-source models trained with substantially larger multimodal budgets.

1. Introduction

DecoupleMix reframes VLM continue-pretraining data construction as a reproducible mixture-optimization problem by separating inter-class budget allocation from intra-class composition. This decoupling also enables controlled dataset-admission evaluation and proxy-scale ratio transfer without additional tuning.

  • Prevailing VLM data curation over-relies on quality filtering, neglecting systematic assessment of data difficulty and diversity.
  • Inter-class ratios are typically chosen by intuition, while intra-class practice stacks filtered datasets and ignores redundancy.Examples include Grounding versus Caption ratios and mixtures of multiple OCR datasets.
  • DecoupleMix separates mixture construction into inter-class allocation across capabilities and intra-class composition within categories.It uses single-variable search for inter-class allocation and constrained convex optimization for intra-class selection, supported by multidimensional dataset assessment.
  • Fixing the inter-class ratio, total budget, and intra-class strategy makes adding a candidate dataset a controlled, attributable intervention.The measured effect includes the resulting within-class reallocation and avoids confounding from re-stacking and retraining.
  • Ratios found at proxy scale transfer across data and model scales without additional tuning, while 80B additional multimodal continued-pretraining tokens yield competitiveness with strong open-model baselines.The recipe is validated and scaled up on a 2.5B→10B-token proxy progression.

2. Methodology

DecoupleMix is a closed-loop methodology that assesses datasets by capability, quality, and difficulty, then decouples macro-ratio search from within-category allocation. It also provides controlled, attributable dataset admission experiments under fixed global conditions.

  • Framework Overview: The framework combines automated dataset-level assessment, decoupled mixture optimization, and standardized data acceptance criteria in a closed loop.Assessment quantifies capabilities, quality, and difficulty; optimization plans inter-class and intra-class mixtures; acceptance evaluates newly collected datasets under a fixed recipe.
  • Automated Dataset-Level Assessment: An extensible hierarchical capability taxonomy uses natural-language criteria and automated classification to identify capability gaps and guide data production.The taxonomy is unbounded, allowing new capabilities to be introduced without manual labeling.
  • Automated Dataset-Level Assessment: Dataset-level LLM judging samples 150 instances per dataset to score Quality and Difficulty, reducing sample-level annotation noise in cohesive industrial datasets.Seed1.6 evaluates four Quality dimensions and six Difficulty dimensions, with hallucination penalties and cross-modal synthesis prioritization.
  • Automated Dataset-Level Assessment: Automated scores align strongly with human consensus, with Spearman r = 0.82 for Quality and r = 0.75 for Difficulty.This supports using dataset-level scores to drive downstream optimization.
  • Decoupled Mixture Optimization: Inter-class ratios are searched with a coordinate-style single-variable procedure, while intra-class allocations use constrained convex optimization with quality, difficulty, diversity, and allocation bounds.The entropy regularizer discourages concentration on a few high-scoring datasets and promotes broader candidate-pool coverage; ECOS solves the convex program.
  • Standardized Data Acceptance Criteria: Candidate admission fixes the inter-class ratio, budget, and optimizer while admitting data into exactly one class, making the resulting allocation shift attributable to that intervention.A source receiving zero allocation under the fixed convex rule can be screened out before empirical training.

3. Experiments

Experiments evaluate DecoupleMix across multiple data and model scales, combining end-to-end comparisons with controlled tests of its search, allocation, and dataset-admission components. The selected recipe is then scaled to 80B additional multimodal continued-pretraining tokens and compared with open-source baselines under a unified evaluation protocol.

  • Experimental Design: Experiments span multiple data and model scales, separating end-to-end comparisons from controlled tests of inter-class search, intra-class allocation, and dataset admission.The selected recipe is subsequently scaled for larger-scale comparison.
  • Experimental Design: 80B additional multimodal continued-pretraining tokens are used to scale the selected recipe for comparison with open-source model baselines.This scaling stage follows the controlled component evaluations.
  • Evaluation: 16 benchmarks are organized into five capability domains: General, Knowledge & Hallucination, OCR & Document, Math & Logic, and Video.The domains match the framework’s capability taxonomy.
  • Evaluation: All reported results, including external baselines, are obtained under one unified evaluation protocol, with video benchmarks fixed to 8 frames per video.MMBench and CharXiv average sub-scores, while MME is normalized to 0–100 using SUM / 2800.
  • Evaluation: Closed-Task Validation with MCQs evaluates multiple-choice benchmarks without instruction-tuning alignment by separating foundational knowledge from instruction-following.The method is intended to reflect pretraining data efficacy through internal-knowledge evaluation.

4. Experiment Results and Analysis

DecoupleMix consistently outperforms heuristic data stacking across scales, localizes the effects of admitting new datasets, and transfers proxy-searched recipes to larger data and model scales without retuning. Its gains arise from both capability-driven inter-class ratios and convex intra-class allocation, while end-to-end models remain competitive with Qwen3-VL under a smaller multimodal budget.

  • End-to-End Efficacy: The 4B model matches Qwen3-VL-4B at 62.5 overall, while the 1B model approaches Qwen3-VL-2B at 52.3 versus 53.7 using 80B additional multimodal tokens.Both models are evaluated before additional multimodal instruction tuning.
  • End-to-End Efficacy: The 4B model leads on reasoning- and difficulty-sensitive benchmarks, including PuzzleVQA 49.0 versus 43.0 and CharXiv 63.7 versus 59.9, but trails on several perception-heavy benchmarks.At 1B, it surpasses the larger Qwen3-VL-2B on PuzzleVQA by +16.8 and MVBench by +3.4.
  • Attributable Validation: Under controlled admission, the target OCR domain moves by +0.3 while every off-target domain stays within 0.6; stacking reaches 1.1 maximum off-target drift and displacement regresses General by −2.3.The protocol fixes the inter-class ratio and total budget, admitting the candidate through convex intra-class reallocation.
  • Scaling and Transfer: The method outperforms heuristic stacking at every data scale, with average gaps of +1.6 at 2.5B, +2.0 at 5B, and +1.4 at 10B tokens.Improvements span capability domains, including InfoVQA +3.1 at 10B.
  • Scaling and Transfer: Ratios searched at 2.5B transfer without retuning to 5B, 10B, and a 32B model, improving the matched-budget 32B overall average by +1.2 from 61.5 to 62.7.The proxy-scale search cost is paid once and amortized across larger data budgets and model capacities.
  • Ablations: Inter-class optimization raises average performance to 57.1 versus 56.6 for stacking, and convex intra-class allocation further raises it to 58.2.Under a fixed inter-class budget, convex allocation reaches 48.7 versus 46.0 for Quantity-Proportional and 47.1 for Quality-Only, with OCR at 52.7 versus 44.8.

5. Related Works

Prior work addresses multimodal example curation, language-model mixture optimization, and multimodal recipe design. DecoupleMix complements these directions by optimizing dataset-level allocation through separated inter-class search and intra-class composition under controlled budgets.

  • Multimodal data curation: Multimodal curation methods filter or score individual examples, whereas DecoupleMix allocates a finite training budget across retained datasets.The related methods include similarity-based filtering, model-based quality curation, and joint quality–diversity selection.
  • Data-mixture optimization: Language-model mixture optimization includes proxy-model weighting, small-scale prediction of unseen mixtures, dynamic schedules, and scaling-law studies for optimal mixtures.Examples include DoReMi, Data Mixing Laws, RegMix, and RegMix-D.
  • Multimodal mixture recipes: Multimodal recipes show that data proportions materially affect pretraining, while DecoupleMix separates inter-class budget search from intra-class dataset allocation under fixed rules.The passage also identifies LLaVA-OneVision-1.5 and Bee as reproducible reference pipelines.

6. Limitations

The study has limited scaling coverage and uses tractable optimization methods that may be suboptimal. It evaluates 2.5B–10B tokens with transfer at 32B parameters, without reporting frontier-scale token-budget trends.

  • Scaling scope: 2.5B–10B tokens are studied, with model-scale transfer evaluated at 32B parameters; trends at much larger frontier-industrial budgets remain unreleased.The limitation concerns both token-budget coverage and the absence of released frontier-scale trends.
  • Optimization algorithms: The single-variable inter-class search and convex intra-class program are tractable by design but likely suboptimal.The passage identifies richer joint search and more expressive objectives as possible improvements.

7. Conclusion

DecoupleMix transforms VLM pretraining data mixture construction from trial-and-error into principled engineering by decoupling inter-class and intra-class optimization. Across 2.5B to 10B tokens, its recipes outperform heuristic baselines and transfer to larger data and model scales without retuning.

  • DecoupleMix is a systematic data mixture optimization framework for transforming VLM pretraining from trial-and-error into principled engineering.
  • The framework uses multi-dimensional dataset-level assessment and decouples data composition into inter-class and intra-class optimization.
  • Validated across 2.5B to 10B tokens, DecoupleMix consistently outperforms heuristic baselines and transfers to larger data and model scales without retuning.

A. Detailed Evaluation Metrics

The evaluation uses 16 mainstream multimodal benchmarks organized into five capability domains. Video benchmarks follow a uniform eight-frame sampling and evaluation protocol across all models, including external baselines.

  • Evaluation Scope: The evaluation suite comprises 16 mainstream benchmarks spanning five primary multimodal capability domains.The domains align directly with the methodology’s capability taxonomy.
  • General: General evaluation covers multimodal understanding, real-world knowledge, and complex reasoning through MMBench, MME, MMStar, MMMU, and RealWorldQA.MMMU focuses on expert-level multi-discipline knowledge.
  • Knowledge & Hallucination: Knowledge and hallucination evaluation uses BLINK, HallusionBench, and AI2D to test visual perception, diagram understanding, and robustness against misleading prompts.These benchmarks also assess resistance to hallucinating non-existent visual elements.
  • OCR & Document: OCR and document evaluation uses OCRBench, InfoVQA, and CharXiv for text-rich image comprehension, layout parsing, and chart reasoning.The category targets document layouts and complex charts.
  • Math & Logic: Math and logic evaluation uses LogicVista, VisuLogic, and PuzzleVQA to test visual deduction, spatial intelligence, and geometric problem solving.These benchmarks target visual logic and geometry-related reasoning abilities.
  • Video: Video evaluation uses Video-MME and MVBench, uniformly sampling 8 frames per video with the same protocol for all models and external baselines.The temporal evaluation targets dynamic scene understanding, temporal localization, and cross-frame synergies.

B. Dataset-Level Assessment: Prompts and Formulations

The section defines dataset-level assessment through automated Quality and Difficulty scoring, semantic diversity categorization, and prompt-based evaluation of sampled data. Human-alignment results support using the automated scores to drive mixture optimization.

  • B. Dataset-Level Assessment: Prompts and Formulations: 150 samples per dataset are evaluated to estimate overall Quality and Difficulty distributions using system prompts from a SOTA VLM.The assessment covers sampled pretraining data at the dataset level.
  • B.1. Human Alignment of the Automated Judge: Automated scores correlate with human consensus at r = 0.82 for Quality and r = 0.75 for Difficulty, with both correlations statistically significant at P < 0.05.Ten instances were independently rated by three expert annotators on the same 1–5 rubrics, and the median was used as consensus.
  • B.2. Quality Assessment Formulation: Quality is computed as Q = 2 · Sacc + 1.5 · Shal + 1 · Scor + 0.3 · Sgra, weighting Accuracy and Hallucination most heavily.The four 1–5 dimensions are Accuracy, Hallucination, Correlation, and Grammar.
  • Quality Evaluation Prompt: The quality prompt asks an assessor to score multimodal samples across dimensions with evidence-based reasoning, emphasizing claims unsupported by the image or text context.The prompt covers image and text modalities, including user questions and assistant responses.
  • B.3. Diversity Categorization Prompt: An LLM classifier assigns each sample a primary category and subcategory to systematically measure semantic diversity and balance multimodal capability distributions.The classification distinguishes multimodal and text-only data using content and image dependency.
  • Diversity Classification Prompt: The diversity taxonomy includes Caption, OCR, Grounding, STEM, Interleave, General VQA, GUI, and Text-Only categories with task-specific subcategories.The prompt also specifies category-priority rules and distinguishes OCR from STEM by reasoning depth.
  • B.4. Difficulty Assessment Formulation: Difficulty aggregates six dimensions as D = 0.5 · Stxt + 0.5 · Simg + 2 · Scross + 1.5 · Sprior + 1 · Scue + 0.3 · Samb, emphasizing Cross-Modal Synthesis and Prior Knowledge Demand.The dimensions are Text Complexity, Image Complexity, Cross-Modal Synthesis, Prior Knowledge Demand, Visual Cue Sensitivity, and Task Ambiguity.
  • Difficulty Evaluation Prompt: The difficulty prompt screens for valuable multimodal samples requiring complex reasoning, cross-modal integration, expert knowledge, or precise visual perception while excluding low-value noise.It is designed for large language model training data.

C. Detailed Hardware and Hyperparameters

Stage 2 uses a memory-efficient training configuration with packed 8,192-token sequences, AdamW optimization, and DeepSpeed ZeRO-2. This stack supports rapid experimentation, requiring approximately 480 GPU hours for the 10B-token recipe and about 3,500 GPU hours overall.

  • Training configuration: Stage 2 uses a global batch size of 128 and a peak learning rate of 1e−4 with cosine decay.
  • Training configuration: Sequence packing with a maximum length of 8192 improves training efficiency, while DeepSpeed ZeRO-2 addresses context-length memory overhead.
  • Optimization: All models are trained with the AdamW optimizer.
  • Compute efficiency: Approximately 480 GPU hours are required to train the 10B token recipe in Stage 2, and all training experiments together cost about 3500 GPU hours.

D. Intra-class Sampling Visualization Analysis

Figure 2 shows that the convex intra-class allocation shifts token mass toward datasets scoring highly on both Quality and Difficulty while retaining broader coverage. This contrasts with quantity-proportional sampling’s emphasis on lower-scoring regions and quality-only sampling’s focus on Quality without Difficulty.

  • Allocation patterns: Convex allocation concentrates more token mass on datasets with high Quality and Difficulty scores while preserving broader coverage.This behavior reflects the diversity objective and joint consideration of both dimensions.
  • Allocation patterns: Quantity-proportional sampling assigns substantial mass to lower-quality, lower-difficulty regions.
  • Allocation patterns: Quality-only sampling concentrates toward higher Quality scores but does not explicitly account for Difficulty.
  • Visualization: Figure 2 encodes Difficulty on the X-axis, Quality on the Y-axis, and allocated token volume through log-scaled point size.The figure compares Quantity-Proportional, Quality-Only, and convex intra-class allocation from left to right.
Loading 2607.24516v1…