PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-DPO: Direct Preference Optimization for Target Model Data Selection in LLM Post-Training

Peng Sun, Yi Yang, Antong Zhang, Chunxiao Li, Yanbo Wang, Dianbo Liu, xin chen, Kai Yu, Lu Chen, Tianfan Fu

arXiv:2608.16926v1cs.LG

TL;DR

SFT data selection must identify small, effective subsets while accounting for how samples match a target model’s capabilities. Data-DPO learns target-model-aware preferences from one-step training feedback and combines them with quality and diversity signals, outperforming baselines across budgets and surpassing full-data training in multiple settings.

  • Problem

    SFT data selection seeks small, effective training subsets, but existing methods pay limited attention to compatibility between data and the target model’s capability distribution.

  • Method

    Data-DPO converts one-step target-model training feedback into pairwise data preferences, learns them with a lightweight reward model, and combines preference, quality, and diversity signals for subset selection.

  • Results

    Data-DPO consistently outperforms representative baselines across multiple budgets on Vision-Flan and LLaVA-CoT, surpassing full-data training in multiple settings.

  • Takeaways & Limitations

    Target-model-aware data preferences can support effective SFT subset selection under small data budgets.

  • Takeaways & Limitations

    Performance may degrade when external quality signals, embedding sources, or target-model preferences are highly biased, and the method remains less effective under substantial distribution shifts.

Abstract

from arXiv · show

Data selection in supervised fine-tuning aims to select a small set of effective samples from large-scale candidate data, reducing training cost while preserving model performance. However, existing methods usually treat data value as a relatively static property, and pay limited attention to the compatibility between data and the capability distribution of the target model. To address this issue, we propose Data-DPO, a target model-oriented SFT data selection method. Data-DPO observes the local training feedback of the target model on different samples through one-step probing, transforms activation differences among samples into pairwise data preferences, and trains a lightweight reward model to learn target-model-aware data preferences. In the final selection stage, Data-DPO further combines target model preference, external quality scores, and marginal diversity to construct a more stable and effective training subset. Experimental results on Vision-Flan and LLaVA-CoT show that Data-DPO consistently outperforms existing data selection baselines under multiple data budgets and stably surpasses full data training performance.

1 Introduction

The introduction frames SFT data selection as a target-model-conditioned problem: sample value depends on compatibility with the model’s capability distribution, not only static quality. It proposes Data-DPO, which learns model-aware preferences from one-step training feedback and combines them with quality and diversity for subset selection.

  • Motivation: Full-dataset SFT is computationally expensive and may not yield optimal performance, motivating selection of a small effective subset.The introduction positions efficient data selection as necessary as instruction datasets grow.
  • Limitations: Existing methods emphasize sample importance, diversity, or their combination, but usually treat data value as a relatively static property.Static value is commonly determined from sample quality and representation-space structure in advance.
  • Target-model conditioning: Because SFT activates and aligns existing capabilities, the same sample may play substantially different roles across models with different capability distributions.The introduction argues that intrinsic sample quality alone cannot determine data value.
  • Method: Data-DPO observes target-model feedback before and after a one-step update, converts it into pairwise data preferences, and combines preference, quality, and diversity signals.The method is designed to select subsets better suited to the target model’s current training needs.
  • Results: 100.76%, 102.63%, and 102.70% are Data-DPO’s average relative performance versus full-data training on Vision-Flan at 5%, 10%, and 15% budgets, respectively.On LLaVA-CoT, it reaches 102.73% and 103.93% of full-data performance at 5% and 10% budgets, respectively.

2 Related Work

Existing data selection methods primarily estimate sample importance or promote data diversity when constructing training subsets. Data-DPO instead uses target-model activation strength to derive pairwise preferences, rather than treating utility as static quality, external similarity, or representation coverage.

  • Sample importance and diversity: Existing methods select training subsets from two complementary perspectives: sample importance and data diversity.Sample importance estimates each instance’s contribution to the target task or model update.
  • Sample importance and diversity: Representative criteria incorporate sample informativeness, intra- and inter-cluster relations, and dialogue turns.The cited related methods design selection criteria across multiple dimensions.
  • Sample importance and diversity: COINCIDE extracts multi-layer target-model representations and samples from both intra- and inter-cluster structures.This approach combines representation extraction with structural coverage across clusters.
  • Target-model-oriented selection: Data-DPO characterizes each sample’s activation strength in the target model and transforms it into pairwise data preferences.This makes selection explicitly oriented to the target model.
  • Target-model-oriented selection: Unlike prior approaches, Data-DPO does not treat sample utility as static quality, external similarity, or representation-space coverage.Its preference construction is based directly on target-model activation behavior.

3 Method

Data-DPO formulates data selection as choosing a budgeted subset that maximizes downstream performance, then learns target-model-aware preferences from one-step updates. It combines learned preference, external quality, and marginal diversity for sequential final selection.

  • Problem Formulation: Data selection chooses a subset D′ ⊆ D under budget K to achieve performance close to or better than full-data training.The objective is to identify a limited subset that maximizes downstream model performance.
  • Dual View Encoding: Data-DPO uses dual-view sample representations: condition embeddings support probe construction and diversity, while supervision embeddings support preference learning and scoring.Both views are extracted with a frozen embedding model, independently centered, and L2-normalized.
  • Probe Set Construction: The probe set clusters candidate data by condition embeddings, samples proportionally from each cluster, and applies greedy k-center selection to broaden condition-space coverage.This produces more diverse and stable supervision for preference construction.
  • Activation Probing: Data-DPO performs repeated one-step SFT updates from the same target checkpoint, measuring each sample’s loss reduction to capture its local activation effect.Temporary updated models are discarded after each probing round, so probing reflects dynamics around the target checkpoint rather than a continuously trained surrogate.
  • Preference Learning and Sequential Selection: Pairwise activation preferences train a lightweight target-conditioned reward model, whose scores are combined with external quality and marginal diversity during sequential selection.The method uses a uniform empirical reference over the probe subset because data selection lacks a natural model-induced reference policy.

4 Experiments

Data-DPO is evaluated on Vision-Flan and LLaVA-CoT across three data budgets, using diverse data-selection baselines and separate target-model benchmarks. It achieves strong, stable performance, surpassing full-data training on both datasets and supporting target-model-aware preference modeling.

  • Experimental setup: Experiments use Vision-Flan for general instruction scenarios and LLaVA-CoT for reasoning scenarios, evaluating Data-DPO across different task complexities and data distributions.
  • Experimental setup: Data-DPO is evaluated under 5%, 10%, and 15% data budgets against baselines spanning importance estimation, diversity distribution, and combined selection paradigms.
  • Main results: On Vision-Flan, Data-DPO achieves ARP of 100.76, 102.63, and 102.70 under the 5%, 10%, and 15% budgets, respectively.These results surpass full-data training and are the best among all methods; gains over the strongest method are 1.31, 2.14, and 2.35 ARP, respectively.
  • Main results: Data-DPO consistently outperforms full-data training on both datasets and substantially outperforms other methods overall, validating explicit modeling of target-model preferences over data.The results support moving beyond static data quality or representation-space coverage alone.

5 Analysis and Ablation Studies

Data-DPO’s analyses show that target-model probing and the joint selection reward are central to performance, while the method remains robust across target models, quality scores, embeddings, and reward-model capacities.

  • Selection reward ablation: Combining target-model preference, quality, and marginal diversity achieves the best selection performance, whereas target-model preference alone yields 92.33 ARP versus 95.30 for random selection.Preference-only selection can concentrate on narrow, easy samples; quality and diversity provide complementary constraints.
  • Target-model probing: Using a randomly trained 5% checkpoint for probing reduces ARP from 100.76, 102.63, and 102.70 to 93.17, 97.76, and 101.72 under 5%, 10%, and 15% budgets.The degradation is especially clear at smaller budgets, supporting direct probing of the original target model.
  • Target-model robustness: With LLaVA-v1.5-13B on Vision-Flan, Data-DPO achieves ARP of 93.76, 94.53, and 94.39 under 5%, 10%, and 15% budgets, improving over random selection by 8.64, 2.27, and 3.25.On LLaVA-CoT with Qwen2VL-2B-Instruct, it reaches 100.13, 97.50, and 98.43 ARP, with gains of 5.83, 2.25, and 3.66.
  • Quality-score robustness: Using LLaVA-OneVision-1.5-4B-Instruct for quality scoring, Data-DPO achieves ARP of 101.23, 100.53, and 103.76 under 5%, 10%, and 15% budgets.The corresponding improvements over random selection are 8.27, 5.23, and 9.94, indicating quality is an auxiliary constraint rather than a single-source dependency.
  • Embedding robustness: Replacing the embedding model with Qwen3-VL-Embedding-8B yields ARP of 99.95, 100.36, and 100.75 under 5%, 10%, and 15% budgets.Data-DPO improves over random selection by 6.99, 5.06, and 6.93, despite some variation across embedding models.
  • Reward-model capacity: A simpler plain MLP achieves 95.36 ARP under the 5% budget, improving over random selection by 2.40 but trailing the residual MLP at 100.76.The plain MLP also surpasses full-data training at 10% and 15%, indicating preference supervision matters more than reward-model complexity.

6 Conclusion and Limitations

Data-DPO selects SFT data using target-model training feedback, learned data preferences, and a lightweight reward model. Its final subset construction combines target-model preference, external quality, and marginal diversity, but performance can degrade under biased signals or substantial distribution shifts.

  • Conclusion: Data-DPO uses one-step target-model training feedback to construct pairwise preferences among samples and learn target-model-aware selection signals with a lightweight reward model.This distinguishes the method from approaches treating data value as static quality.
  • Conclusion: Final subset construction combines target-model preference, external quality assessment, and marginal diversity.
  • Limitations: Performance may degrade when external quality signals, embedding sources, or target-model preferences are highly biased.The limitation arises from bias in the signals used by Data-DPO.
  • Limitations: Data-DPO may have limited effectiveness when the full dataset exhibits substantial distribution shifts.The method focuses on quality and preference modeling under distribution-preserving settings.

A Implementation Details of Data-DPO

Data-DPO uses dual-view embeddings, clustered probe-set construction, fixed activation-probing settings, and a lightweight residual-MLP reward model. Sequential selection incorporates normalized external quality ratings on a 0–1 scale.

  • Dual View Encoding: Dual-view encoding feeds c_i alone for the condition embedding and the concatenated pair (c_i, y_i) for the supervision embedding.
  • Probe Set Construction: Probe subsets use spherical clustering with 2000 clusters for Vision-Flan, 1000 for LLaVA-CoT, and a fixed probe ratio ρ = 0.05 per cluster.
  • Activation Probing: Activation probing runs for T = 16 rounds with batch size b = 1024, while retaining the SFT-stage training hyperparameters.
  • Preference Learning: The reward model is a lightweight residual MLP that maps 2048-dimensional supervision embeddings through a 1024-dimensional projection and four residual blocks to a scalar reward logit.Each residual block uses LayerNorm, linear layers, GELU, Dropout(0.1), and a residual connection.
  • Sequential Selection: External quality scoring uses discrete ratings from 1 to 5 normalized to [0, 1], mapping them respectively to 0, 0.25, 0.5, 0.75, and 1.Scores are obtained by feeding each sample into a multimodal model.

B Datasets and Preprocessing Details · B.1 Datasets · B.2 Preprocessing Procedures

The paper evaluates Data-DPO on two vision-language instruction datasets with complementary goals: broad visual task coverage and structured image-text reasoning. Preprocessing retains Vision-Flan unchanged while filtering and formatting LLaVA-CoT samples.

  • B.1 Datasets: Vision-Flan is a human-curated visual instruction tuning dataset targeting broad task coverage and strong cross-task generalization.It is built from publicly available academic vision benchmarks.
  • B.1 Datasets: Vision-Flan reorganizes its source benchmarks into 187 fine-grained visual tasks with manually written task instructions.Each task contains roughly 1,000 examples.
  • B.1 Datasets: Vision-Flan contains about 186K samples and emphasizes expert annotation and validation over purely synthetic instruction generation.This design helps reduce instruction artifacts and unreliable supervision.
  • B.1 Datasets: LLaVA-CoT-100k is an image-text reasoning instruction dataset designed to improve structured reasoning ability in vision-language models.It integrates approximately 99K image-question-answer pairs from general-purpose and science-targeted VQA datasets.
  • B.1 Datasets: Each LLaVA-CoT sample uses a structured reasoning response with summary, caption, reasoning, and conclusion stages.
  • B.2 Preprocessing Procedures: Vision-Flan uses the original dataset without additional filtering or format conversion, whereas LLaVA-CoT removes samples without associated images.
  • B.2 Preprocessing Procedures: After filtering, LLaVA-CoT contains 98,572 image-text reasoning samples.
  • B.2 Preprocessing Procedures: LLaVA-CoT wraps each response’s reasoning process and final answer with <think></think> and <answer></answer>, respectively.

C Training Hyperparameters · D Evaluation

The paper uses established official LLaVA settings for two target models and conservative full-parameter fine-tuning choices for two instruction-tuned checkpoints. For Qwen2-VL-2B-Instruct and Llama-3.2-11B-Vision-Instruct, the learning rate is 1 × 10−5 and batch size is 64.

  • C Training Hyperparameters: LLaVA-V1.5-7B and LLaVA-V1.5-13B use hyperparameter settings from the official LLaVA training code.
  • C Training Hyperparameters: Qwen2-VL-2B-Instruct and Llama-3.2-11B-Vision-Instruct use a conservative learning rate of 1 × 10−5 for full-parameter fine-tuning.Both models are already instruction-tuned checkpoints.
  • C Training Hyperparameters: The batch size is set to 64 for Qwen2-VL-2B-Instruct and Llama-3.2-11B-Vision-Instruct.The passage states that a batch size of 128 leads to too few optimization steps, but the sentence is truncated.

D.1 Evaluation Benchmarks · D.2 Detailed Evaluation Results

The evaluation uses capability- and modality-matched benchmarks for Vision-Flan and LLaVA-CoT. Detailed results report main comparisons and additional proxy-model, reward-ablation, and model-variant analyses.

  • D.1 Evaluation Benchmarks: Vision-Flan evaluation covers general multimodal understanding, visual question answering, OCR, document understanding, chart understanding, and science-diagram reasoning.
  • D.1 Evaluation Benchmarks: LLaVA-CoT evaluation combines multimodal mathematical and logical reasoning benchmarks with general vision-language benchmarks.
  • D.1 Evaluation Benchmarks: The benchmark design evaluates LLaVA-CoT reasoning ability and overall multimodal robustness.
  • D.2 Detailed Evaluation Results: Tables 3 and 4 provide full benchmark-level comparisons of Data-DPO and all data selection baselines on Vision-Flan and LLaVA-CoT.
  • D.2 Detailed Evaluation Results: The detailed evaluation also includes benchmark-level results for analyses and ablation studies discussed in the main paper and appendix.
  • D.2 Detailed Evaluation Results: Table 5 reports proxy-model analysis results, while Table 6 presents ablations on the selection reward.
  • D.2 Detailed Evaluation Results: Table 7 shows Vision-Flan results with LLaVA-V1.5-.

E Time Cost Analysis · Method MATH-Vision MME MMB-EN SQA-I AI2D ChartQA InfoVQA OCRBench ARP

The time-cost analysis evaluates Data-DPO on LLaVA-CoT and reports approximately 19.0 GPU hours, while the accompanying appendix examines robustness, ablations, and sensitivity across models, scoring systems, embeddings, and reward-model capacities.

  • E Time Cost Analysis: Data-DPO takes approximately 19.0 GPU hours on a NVIDIA A6000 for LLaVA-CoT selection.Its cost is slightly higher than other baselines but remains acceptable compared with full SFT training.
  • Method MATH-Vision MME MMB-EN SQA-I AI2D ChartQA InfoVQA OCRBench ARP: The reported evaluation materials include benchmark-result tables for Vision-Flan and LLaVA-CoT, plus scoring prompts for estimating quality scores on both datasets.The benchmark listings include metrics such as SQA-I, MME, MMB-EN, AI2D, ChartQA, InfoVQA, OCRBench, and ARP.
  • Method MATH-Vision MME MMB-EN SQA-I AI2D ChartQA InfoVQA OCRBench ARP: The LLaVA-CoT experiments evaluate methods across MATH-Vision, We-Math, LogicVista, DynaMath, MMStar, MME, MMBench-EN, ScienceQA-IMG, AI2D, ChartQA, InfoVQA, OCRBench, and ARP.The listed evaluations cover 5% and 15% data-subset settings in the supplied passages.
  • Method MATH-Vision MME MMB-EN SQA-I AI2D ChartQA InfoVQA OCRBench ARP: The appendix compares proxy-model choices for activation probing under 5%, 10%, and 15% Vision-Flan data budgets.It contrasts probing the original LLaVA-v1.5-7B target model with probing a checkpoint trained on randomly sampled 5% Vision-Flan data.
  • Method MATH-Vision MME MMB-EN SQA-I AI2D ChartQA InfoVQA OCRBench ARP: The selection-reward ablation evaluates combinations of target-model preference, quality score, and marginal diversity on Vision-Flan with a 10% data subset.The experiment uses LLaVA-v1.5-7B as the target model.
  • Method MATH-Vision MME MMB-EN SQA-I AI2D ChartQA InfoVQA OCRBench ARP: Robustness analyses vary target models, quality-scoring models, embedding sources, and reward-model capacity across Vision-Flan and LLaVA-CoT.The supplied tables include LLaVA-v1.5-13B, Qwen2VL-2B-Instruct, LLaVA-OneVision-1.5-4B-Instruct, and plain versus residual MLP reward models.
  • E Time Cost Analysis: Table 12 compares the time cost of different methods on LLaVA-CoT.The comparison is conducted in GPU hours on a NVIDIA A6000.
Loading 2608.16926v1…