Pick the Mixture, Not the Samples: Combinatorial Data Selection for LLMs

1 INTRODUCTION

• Samples can help evaluation tasks unevenly, while training on samples together can yield more or less utility than the sum of their individual contributions.The motivating setting is choosing a small subset from a large pool spanning skill domains.Source: §1 · p. 1
• Combinatorial Data Selection separates sample attribution from subset selection and models complementary and substitutive interactions across multiple downstream tasks.It extends submodular selection, which captures substitutive interactions, and accepts any per-sample attribution score.Source: §1 · p. 1
• MtS first chooses how many samples to take from each domain using per-domain utility, then selects the highest-attribution samples within each domain.The method's guarantee depends on stated structural and predictor-accuracy assumptions; its additive gap grows linearly with selection size and is independent of pool size.Source: §1 · pp. 1–2
• MtS outperforms eight baselines across four selection principles by 20 to 54% relative in held-out utility.The paper also reports robustness across evaluation tasks and domain partitions, and applies MtS to purchasing data under varying prices and a fixed budget.Source: §1 · p. 2
• The paper frames LLM data selection as combinatorial allocation across domains and presents MtS as a practical method for choosing training data to buy under a fixed budget.Its contributions also include theoretical and empirical performance guarantees.Source: §1 · p. 2

2 RELATED WORK

• Table 1 compares methods on sample-pair interactions, cardinality or nonuniform-price budgets, subset-utility guarantees, and domain allocation.The discussion distinguishes MtS as selecting individual samples before training under cardinality or price budgets while accounting for pairwise interactions.Source: Table 1 · p. 2, §2 · p. 3
• Per-sample attribution methods differ in cost, from single-pass quality scores to gradients, Hessian-based influence approximations, and repeated training-trace queries.Downstream selection ranks or thresholds these scores.Source: §2 · p. 2
• Subset-level approaches include submodular and coreset selection, determinantal point processes, and dataset distillation; domain-mixture methods learn or update domain proportions.The cited approaches include DoReMi [Xie et al., 2023], RegMix [Liu et al., 2025], and DIDS [Shi et al., 2025].Source: §2 · p. 2
• Data-pricing work studies contribution valuation, reinforcement-learning valuation, market design, LLM fair-share pricing, and creator compensation.The cited threads include Shapley [Ghorbani & Zou, 2019], Banzhaf [Wang & Jia, 2023], and LLM-specific pricing [Zhang et al., 2025].Source: §2 · p. 3

3 COMBINATORIAL DATA SELECTION

• A selected subset's value depends on sample interactions during fine-tuning, making training-data selection a combinatorial optimization problem.The section develops a constrained formulation, an efficient algorithm, and a theoretical guarantee.Source: §3 · p. 3

3.1 DATA SELECTION AND PAIRWISE INTERACTIONS

• The selection objective aggregates utility across evaluation tasks and supports arbitrary task weights and heterogeneous sample costs under a total budget.Pairwise approximation decomposes utility into singleton values, pairwise interactions, and a higher-order remainder; on the data pool, it captures ≥90% of utility variance across ∼100 measured triplets (R2 = 0.90).Source: §3.1 · pp. 3–4
• The pairwise interaction matrix is split into within-domain interactions, cross-domain block means, and residual interactions, aligning within-domain choices with the first component and mixture choices with the second.The residual captures deviations from cross-domain block means.Source: §3.1 · p. 4

3.2 THE MIXTURE-THEN-SELECT ALGORITHM

• MtS searches feasible domain mixtures with a learned utility predictor, then fills each domain with top-ranked samples using their singleton values.The predictor is fit on measured probe-run utility; the per-sample values can come from different attribution sources, and the residual interaction term is uncontrolled by either stage.Source: §3.2 · p. 4

3.3 APPROXIMATION GUARANTEES

• Under the stated structural and global-mixture-identification assumptions, MtS's utility gap from the optimum grows linearly with selection size and is independent of data-pool size.The bound is tight, and estimation error in the interaction matrix or singleton values adds a penalty proportional to that error.Source: §3.3 · pp. 5–6
• The spectral guarantee controls each stage's residual-interaction contribution with an operator norm, which is reported to yield a factor-m improvement over an entrywise bound.On the calibrated data pool, the spectral form is approximately 51× tighter than the entrywise Bounded-Complementarity form.Source: §3.3 · p. 5
• The guarantee separates within-domain and cross-domain residual gaps, both linear in selection size and independent of the number of domains.It applies at sub-domain budgets; its constants are pessimistic, and the bound describes scaling and estimation robustness rather than predicting the observed gap.Source: §3.3 · p. 6

4.1 SETUP

• The main pool has 50K samples from 13 public sources across eight skill domains; evaluations also use a 200K pool and multiple models.Qwen3-8B is the main base model, and selection sizes span approximately 1% to 20% of the pool.Source: §4.1 · p. 6

4.2 PERFORMANCE COMPARISONS

• MtS beats every baseline by 20% to 54% relative, and its 1,000-sample subset beats full 50K-data-pool training by ∆=+0.146.All MtS rows significantly outperform Stratified (paired t-test, p<0.05).Source: Table 2 · p. 7, §4.2 · pp. 7–8
• MtS reaches 0.694 utility, versus 0.6566 for RegMix and 0.6272 for DoReMi, with reported relative gains of 5.7% and 10.7%.RegMix and DoReMi use a 1K subset with fixed within-domain selection; DIDS reaches 0.6343 using online full-pool sampling with matched training exposures.Source: §4.2 · p. 8

4.3 COMPUTATIONAL EFFICIENCY

• MtS uses 43% less total compute than full-data training and 77% less than LESS, with approximately 27% higher utility under Table 4’s accounting.The comparison includes calibration; LESS’s gradient-extraction pipeline is roughly 4.26× more expensive than MtS at the same selection size.Source: §4.3 · pp. 8–9, Table 4 · p. 9

4.4 ANALYSIS OF MECHANISM

• Singleton and pairwise terms explain 90% of utility variation across approximately 100 measured triplets; adding pairwise terms reduces prediction error from 0.014 to 0.004.The controlled comparisons also favor MtS over other selection rules using the same attribution scores.Source: §4.4 · p. 9
• At m≤500 on a 50K pool, Stratified matches or edges MtS; Qwen-derived within-domain rankings also do not transfer reliably to Llama without recalibration.The authors recommend re-measuring rankings on the target model and suggest using a pilot predictor’s gain-versus-uncertainty as an empirical screening rule.Source: §4.4 · p. 10
• MtS retains utility advantages of 3.5% after coarsening eight domains into four and 3.2% with utility-derived clusters, compared with 4.3% under the original partition.The predictor-input ablation gives utility 0.6927 with mixture and value-statistics features; keeping only the domain histogram lowers utility by 0.088, and only value statistics by 0.063.Source: §4.4 · pp. 9–10

4.5 APPLICATION: BUNDLE-LEVEL DATA PRICING

• Under a purchasing budget, MtS recovers ≥89% of the oracle’s optimality gap at every budget and exactly matches the oracle on 58% of instances.In this setting, MtS replaces the size constraint with a budget constraint while retaining its two-stage allocation approach.Source: Table 5 · p. 10, §4.5 · pp. 10–11

5 CONCLUSION

• Mixture-then-Select beats strong baselines on real instruction-tuning data, works across models and data sizes, and extends to buying data under a variable-price budget.The conclusion frames domain allocation as a design lever alongside model building.Source: §5 · p. 11

A.3 CORE ASSUMPTIONS

• The guarantees assume monotone task utilities, bounded positive pairwise complementarity, and geometrically decaying higher-order interactions; the observed third-order remainder is one to two orders below the pairwise term.Under substitution-only interactions, the pairwise approximation is submodular.Source: §A.3 · pp. 16–17, §A.8 · p. 19

A.4 TWO SOURCES OF COMPLEMENTARITY

• Linear aggregation preserves submodularity (β=0), whereas concave aggregation yields complementarity when domains cover different tasks, even if each task’s interactions are substitutive.Source: §A.4 · p. 17

A.5 OPERATOR-NORM REFINEMENT

• Matching domain composition cancels the cross-domain block mean; controlling residual interactions with operator norms yields an O(m) rather than O(m^2) gap.Source: §A.5 · p. 17

A.6 LOCAL-MARGIN CONDITION AND FINITE-SAMPLE NOISE

• Exact mixture identification requires a global accuracy-and-separation condition; a local margin alone controls displacement only after localization is established.Source: §A.6 · p. 18

A.7 NP-HARDNESS

• CDS is NP-hard even with zero pairwise interactions, and unconstrained CDS remains NP-hard when all interactions are substitutive.Source: §A.7 · pp. 18–19

A.9 HIGHER-ORDER REMAINDER BOUND

• Under Assumption 3, the higher-order remainder is bounded when nρ_d < 1, with its bound increasing as the selection size and interaction-decay term grow.Source: §A.9 · p. 19

A.10 SUBMODULARITY RATIO UNDER CONDITIONAL BOUNDED COMPLEMENTARITY

• The adaptive extension needs conditional complementarity control after conditioning, beyond the singleton-based bound used for the fixed-ranking guarantee.This additional condition is used only for the adaptive extension.Source: §A.10 · pp. 19–20

A.11 PROOF OF THEOREM 1 (MIXTURE-THEN-SELECT SPECTRAL GUARANTEE)

• The spectral bound is independent of pool size, tightens with smaller optimal per-domain counts, and grows linearly with selection size.The proof matches the selected and optimal domain counts under the uniform-accuracy event; block-constant cross-domain terms then cancel.Source: §A.11 · p. 20

A.12 ADAPTIVE INTERACTION-AWARE EXTENSION AT T = 1

• Theorem 11 guarantees adaptive IA-Greedy under monotonicity and a submodularity-ratio condition, but this is distinct from Algorithm 1’s fixed-ranking guarantee.The analysis requires valid conditional-marginal bounds; a centered operator norm alone does not certify the ratio.Source: §A.12 · p. 21

A.13 DOMAIN-UNIFORM COST EXTENSION

• With domain-uniform prices, the guarantee retains Theorem 1’s form with m replaced by m*, where floor(B/max_c p_c) ≤ m* ≤ floor(B/min_c p_c).At uniform price p, m* = floor(B/p); unit prices with B = m recover the cardinality-bound case.Source: §A.13 · pp. 21–22

A.14 TIGHTNESS CONSTRUCTIONS

• The lower-bound constructions show that either within-domain or cross-domain interactions can force two-stage algorithms to lose utility, with each axis matching its spectral upper bound up to factor 2.An empirical SVD finds 16 degenerate ±σ pairs among the top 32 singular values, with σ_i/σ_1 in [0.77, 1.0], supporting the cross-axis construction’s relevance to observed interactions.Source: §A.14 · pp. 22–23, §A.15 · p. 23

A.16 PREDICTOR ACCURACY AND MIXTURE IDENTIFICATION

• Exact mixture recovery requires a uniform predictor-error certificate smaller than half the unique optimum’s mixture gap; probe count alone does not establish this condition.Under the stated uniform-error hypothesis, the corollary gives a sufficient probe budget of O((rT + log(1/δ))/ϵ²), with constants depending on A and B.Source: §A.16 · p. 23

A.17 NUMERICAL INSTANTIATION

• At m = 1000, the reported R(m) is approximately 37 vmax conditional on zero composition displacement, and Mixture-then-Select matches or exceeds IA-Greedy on every calibration benchmark.The displacement claim requires the global identification event; the calibration count L = 5000 does not establish it. The hybrid therefore coincides with Mixture-then-Select on those benchmarks.Source: §A.17 · pp. 23–24

A.18 IA-GREEDY: THE T=1 SINGLE-DOMAIN BASELINE AND VARIANTS

• Lazy IA-Greedy uses cached marginal gains and, in the reported experiments, achieves identical selections to exact IA-Greedy at m ≤ 10 and stays within 1.3% on ũ at m = 20 with 6–8× fewer evaluations.Other variants include stochastic sampling for extreme scale and singleton-enumeration or local-search methods with stated approximation guarantees; these are alternatives to the fixed Stage 2 rule.Source: §A.18 · p. 24

A.19 COMPUTATIONAL HARDNESS

• CDS is APX-hard even with substitutive pairwise interactions, while alternative minimum and concave aggregators encode floor-raising or balanced task performance and can induce aggregate complementarity.Under linear aggregation, β = 0 when each task utility is submodular; under concave or minimum aggregation, β can be positive even when every task utility is submodular, and Theorem 1 applies with recalibrated constants.Source: §A.19 · p. 25, §A.20 · p. 25, §A.21 · p. 25

A.21 PROOF OF PROPOSITION 3 (TWO SOURCES OF COMPLEMENTARITY)

• When cross-task coverage gains exceed pairwise substitution losses, samples strong on different tasks can have positive joint complementarity.The bottleneck task receives contributions from both samples, making joint utility exceed the sum of singleton utilities.Source: §A.21 · pp. 25–26

A.22 PROOF OF THEOREM 2 (OPERATOR-NORM ROBUSTNESS)

• The operator-norm robustness proof yields a total additive gap of 2ϵop m vmax under the stated perturbation bound.The cross-domain gap is (δop + ϵop) m vmax, while the residual perturbation bound is ϵop vmax.Source: §A.22 · p. 26

A.23 vj-NOISE ROBUSTNESS

• With per-sample value noise bounded by ϵv vmax, the added vj-noise contribution is at most 2ϵv m vmax.The stated bound is informative for ϵv < 1/2 and the contribution scales linearly with selection size m.Source: §A.23 · pp. 26–27

A.24 MULTI-OBJECTIVE EXTENSION

• For non-negative weights, scalarizing K utilities preserves a submodularity ratio at least min_m γm.The result applies to utilities with individual submodularity ratios γ1, …, γK.Source: §A.24 · p. 27

A.25 MATROID CONSTRAINT EXTENSION

• Under partition matroid constraints, IA-Greedy retains its (1 − e^−γ) guarantee by restricting each step to feasible additions.The constraints are |S ∩ Cm| ≤ bm; the augmentation axiom ensures a feasible addition from the optimal set while |St| < |S*|.Source: §A.25 · p. 27

B.1 SYNTHETIC EXPERIMENT DETAILS

• Synthetic tests use 100-item instances, selection size m = 20, and compare IA-Greedy with exact optima across complementarity levels.Fifty instances are generated per level; interactions combine substitutive and complementary components, and optima use exhaustive enumeration at β = 0 or branch-and-bound at β > 0.Source: §B.1 · p. 27
• Pilot and scaled-proxy settings span four tasks and Pythia-1B or Qwen3 models, with proxy analysis using 864 samples and 3,000 measured pairs.The pilot uses 40 training and 200 evaluation samples total; the scaled proxy compares gradient cosine with regressors and reports pairwise Spearman correlation and positive-interaction AUC.Source: §B.1 · p. 28

B.2.1 CALIBRATION REUSE AND DOMAIN SCALING

• 43% less total compute than full-data training and 77% less than LESS, with approximately 27% higher utility than either under Table 4’s accounting.Calibration-free Stratified remains cheaper for a one-off selection; reusing calibration reaches break-even at approximately 35 final training runs in this comparison.Source: §B.2.1 · p. 28

B.2 EXPERIMENTAL DETAILS (PILOT N=40 + SCALED-PROXY STUDY)

• At N=50K, MtS uses 16.58 GPU-hours for 0.694 utility, versus LESS at 70.58 GPU-hours and 0.548; calibration costs roughly 17× less than LESS’s selection pipeline at N=200K.Table 10 charges calibration to MtS but not oracle-ranked baselines; downstream fine-tuning is excluded from Table 9’s selection-side comparison.Source: §B.2 · p. 28, Table 9 · p. 28, Table 10 · p. 29

B.2.1 CALIBRATION REUSE AND DOMAIN SCALING

• Taskwise probe outcomes can be reused for new selection sizes, price constraints, and objective weights on the same pool and base model; probe requirements grow with domain count and predictor rank, not pool size.At fixed assumptions, flat calibration for 1,000 domains requires approximately 125 times as many measurements as for eight domains; the authors support grouping related tasks into operational families.Source: §B.2.1 · p. 29

B.3 PILOT MODEL-SCALING STUDY (N=40): COMPLEMENTARITY EMERGES WITH CAPACITY

• In the 40-sample pilot, interactions were measured across three base models, and IA-Greedy and IA-Enum were best or tied-best at three of four selection budgets.The Qwen3-8B pilot reports interaction signs and aggregate strength by model, with task-level cross-domain complementarity measured on 60 pairs per task.Source: §B.3 · p. 30, Table 11 · p. 30, Table 12 · p. 30

B.4 PAIRWISE PREDICTION AND LARGER-CARDINALITY SELECTION

• Pairwise terms explain 90% of measured utility variation across approximately 100 triplets; MtS also beats Stratified in all six larger-pool conditions tested.Adding pairwise terms reduces mean absolute prediction error from 0.014 to 0.004; the larger-pool checks cover selections up to 10K from 50K and 3.5K, 10K, and 20K from 200K.Source: §B.4 · pp. 30–31, Table 14 · p. 31

B.5 SCALED-PROXY EVALUATION (N=864): GRADIENT COSINE VS. CALIBRATED PREDICTOR

• The N=864 proxy comparison evaluates a calibrated Ridge predictor using 14 per-task-value features against gradient cosine computed from 7.7M-dimensional LoRA gradients.The scaling protocol measures singleton values, calibrates on approximately 5,000 strategically sampled pairs, then estimates N^2 interactions in seconds.Source: §B.5 · p. 31, Table 15 · p. 31

B.6 BUDGET SWEEP: WHERE THE DOMAIN-MIXTURE ADVANTAGE HOLDS

• At m=500, MtS is 0.065 below Stratified (p=0.018), but at m=1,000 it is 0.028 above Stratified (p=0.0013); larger-budget positive gaps are not significant.The sweep covers m values from 500 to 10,000 on Qwen3-8B at N=50K.Source: Table 16 · p. 32, §B.6 · p. 32

B.7 MIXTURE-THEN-SELECT ABLATIONS: π∗AND PER-DOMAIN vAGG ARE BOTH LOAD-BEARING

• 0.6927 utility with all 11 D1 features; histogram-only and value-statistics-only predictors lose 0.088 and 0.063, respectively.The compared feature groups are the domain histogram, mixture entropy, and value statistics.Source: Table 17 · p. 32, §B.7 · p. 32

B.8 ROBUSTNESS PROFILE: MIXTURE-THEN-SELECT ACROSS 14 CONDITIONS

• Mixture-then-Select wins or trends in 14 of 18 cells against Stratified; the only significant loss is the out-of-regime m=500.The table covers budget, LoRA rank, attribution signal, and base model.Source: §B.8 · p. 32, Table 18 · p. 33
• At m=5000, Mixture-then-Select scores 0.691/0.624/0.674 under oracle/LESS/EL2N attribution, winning or tying Stratified in every cell.The corresponding Stratified scores are 0.678/0.613/0.600; Figure 5 summarizes robustness across attribution signals.Source: §B.8 · p. 32, Figure 5 · p. 33, Table 19 · p. 34
• At N=200K, Mixture-then-Select retains the MtS≥Strat>Greedy ordering across attribution signals at m=3500 and 10,000.Under LESS at m=10,000, MtS is 0.617±0.014 versus Stratified at 0.603±0.033, a +0.014 gap.Source: §B.8 · pp. 33–34
• Training on all 50K samples yields 0.548±0.019 utility, compared with 0.694 for Mixture-then-Select’s 1,000-sample selection.The full-data reference uses three epochs and five seeds.Source: §B.8 · p. 34
• On Llama-3.2-3B with Qwen-derived selections at m=5000, Mixture-then-Select × EL2N reaches +0.020 while Stratified × EL2N is −0.123.At m=1000, naive Qwen-to-Llama transfer has a −0.030 MtS–Stratified gap; the cross-model comparison is summarized in Figure 5.Source: §B.8 · pp. 32, 34, Figure 5 · p. 33

B.9 ROBUSTNESS TO DOMAIN DEFINITIONS

• Merging eight domains into four retains a 3.5% utility advantage over Stratified, while utility-vector clustering retains a 3.2% advantage.The clustering uses measured per-sample utility vectors and does not bypass their measurement.Source: §B.9 · p. 34
• Fixed 20% symmetric label corruption without recalibrating the clean allocation decreases the four-task downstream composite by 0.7% relative.Table 20 distinguishes this downstream endpoint from utility comparisons under alternative partitions.Source: §B.9 · p. 35, Table 20 · p. 35
• With 30% two-domain ambiguity, reassignment substantially improves recovery of the clean selection, measured by Jaccard overlap rather than downstream performance.The diagnostic holds allocation and singleton values fixed and uses frozen embeddings, utility profiles, or both for reassignment.Source: §B.9 · p. 35, Table 21 · p. 35
• At 30% ambiguity, the lambda-0.50 variant improves held-out utility by 3.09% and Equal4 by 2.42% relative to Stratified.Both aggregate gains occur in all three seeds; this end-to-end test uses fresh calibration after embedding-based reassignment.Source: §B.9 · pp. 35–36, Table 22 · p. 35
• Source or task metadata, coarsening, and utility-derived clustering are supported operational options, but jointly choosing domains with calibration remains an open question.Samples can express several skills while receiving one reproducible assignment for selection.Source: §B.9 · pp. 34, 36

B.10 ADDITIONAL ROBUSTNESS ABLATIONS: DOMAIN, RANK, AGGREGATION

• With four macro-domains, Mixture-then-Select retains a positive lead over Stratified, though the gap shrinks from +0.029 to +0.022.The four-domain calibration uses fresh per-domain fine-tunes on top-125 samples by aggregate value.Source: §B.10 · p. 36, Table 23 · p. 36
• On Llama-3.2-3B at ranks 8, 16, and 32, Mixture-then-Select scores 0.314/0.281/0.243 versus Stratified’s 0.250/0.190/0.133.The gap grows from +0.064 at rank 8 to +0.110 at rank 32, matching the reported Qwen3-8B pattern.Source: §B.10 · p. 36
• Under mean, minimum, and geomean aggregation on Llama-3.2-3B, Mixture-then-Select scores +0.281/+0.233/+0.280 versus Stratified’s +0.190/+0.106/+0.189.Under minimum aggregation, its advantage is +0.127; it is the only method whose worst domain remains positive on this model.Source: §B.10 · p. 36
• Mixture-then-Select leads Stratified on every Qwen3-8B skill domain, including creative at +0.582 versus +0.540 and math at +0.766 versus +0.736.The reported aggregate gap of approximately 0.07 reflects per-domain gains rather than a single-domain win.Source: §B.10 · p. 36, Table 24 · p. 37
• Under five task-weighting schemes that emphasize individual skills 3×, the Mixture-then-Select advantage is preserved.The re-aggregation uses the same per-domain data as the uniform-weight headline.Source: §B.10 · p. 37, Table 25 · p. 37

B.11 FROZEN-SELECTION TRANSFER TO A LARGER MODEL

• The frozen Qwen3-8B selection on Qwen2-72B tests reuse without recalibration or reselection, but does not compare selection methods on 72B.The models differ in size and instruction alignment; benchmark scores rise on MMLU, GSM8K, and ARC-Challenge while IFEval is lower.Source: §B.11 · p. 37, Table 26 · p. 37

B.12 LLAMA-3.2-3B NATIVE RUN: RESOLVES THE SELECTION-TRANSFER TIE

• On Llama-3.2-3B, recalibrating sample values while retaining the Qwen-derived mixture changes the MtS–Stratified gap from −0.030 to +0.091.The recalibrated gap is significant (p=0.0067); the transferred-selection gap is not (p=0.45).Source: §B.12 · p. 37

B.13 CAPACITY-GATED DOMAIN MIXTURE: DATA POOL-N AND RANK BEHAVIOR

• The referenced Llama-3.2-3B transfer uses Qwen-derived mixture weights with Llama-recalibrated rankings, while other rows transfer Qwen-selected subsets.The cited table passages provide table scope but no numerical outcomes for these rows.Source: §B.13 · p. 38, Table 27 · p. 38, Table 28 · p. 38
• The pool-size comparisons span three pool-size/rank settings, but differing ranks across rows mean the comparison does not isolate pool size.Table 29 identifies the 50K condition as rank 16 and the 100K and 200K conditions as rank 8.Source: §B.13 · p. 38, Table 29 · p. 38
• The additional pool-size comparisons use ranks 8 and 16.Cross-model transfer is reported separately in Appendix B.12.Source: §B.13 · p. 38

Per-instance results

• Mean utility is 0.240 for joint-aware MtS and 0.160 for additive LP across the reported per-instance results.MtS exceeds the restricted Stage-B oracle in two cells; that oracle refines only the top 50 single-seed candidates.Source: §B.14 · p. 38

Within-cluster scoring controls

• Neither cumulative-maximum solo-utility curves nor allocation caps at each cluster’s solo-utility maximizer changed additive LP’s selection in any of 30 cells.Increasing the budget also does not consistently increase its measured joint-training utility.Source: §B.14 · p. 38

• Figure 6 reports per-domain utility on Llama-3.2-3B at N=50K and m=1K; its numerical values are in Table 28.Source: Figure 6 · p. 39
• Figure 7 shows selected domain proportions on Qwen3-8B at N=200K and m=3.5K, with a dashed uniform-allocation reference of 1/8.The corresponding 50K comparison is in Figure 2A.Source: Figure 7 · p. 39

Positive impacts

• Interaction-aware selection can reduce fine-tuning waste by helping practitioners assemble smaller, higher-value subsets, especially when exhaustive trial-and-error over large pools is unaffordable.Source: C BROADER IMPACT · pp. 39–40

Negative impacts

• If redundancy or complementarity tracks demographic or linguistic attributes, utility-only selection may under-select already underrepresented data.Repeatedly judging particular domains or styles redundant could amplify existing dataset biases.Source: C BROADER IMPACT · p. 40

Mitigations

• Matroid and multi-objective extensions can encode diversity or coverage constraints, while explicit interaction modeling can help audit systematic exclusions.Source: C BROADER IMPACT · p. 40