PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaMiCS: Fine-tuning LLMs with Performance Constraints using Dynamic Mixtures

Eleonora Gualdoni, Sonia Laguna, Louis Bethune, Joao Monteiro, Pierre Ablin, Marco Cuturi

arXiv:2605.10770v1cs.LG

TL;DR

Multi-domain fine-tuning must improve target capabilities without losing constrained capabilities, but existing mixture strategies cannot explicitly enforce preservation. DynaMiCS uses short domain-specific probes to estimate local cross-domain effects and optimizes simplex mixture weights under loss constraints. Across varied scenarios, it achieves stronger target improvements and higher constraint satisfaction than fixed-mixture baselines at lower cost.

  • Problem

    Fine-tuning must improve specialization domains while preserving capabilities such as general knowledge, instruction following, and safety, which existing mixture strategies cannot explicitly guarantee.

    Source: §1 · p. 1

  • Method

    DynaMiCS performs short domain-specific probing runs to estimate a slope matrix, then optimizes mixture weights over the probability simplex under constrained-domain loss limits.

    Source: §1 · p. 2, §3.1 · p. 4, §3.2 · p. 4

  • Results

    DynaMiCS consistently achieves stronger target-domain performance and constraint satisfaction than fixed-mixture baselines at a fraction of grid-search cost.

    Source: §1 · pp. 2–3, §6 · p. 10

  • Takeaways & Limitations

    DynaMiCS provides dynamic, constraint-aware fine-tuning without auxiliary models, per-example scoring, or manual mixture tuning.

    Source: §6 · pp. 9–10

    Local linear approximations, additive-effect assumptions, narrow evaluations, and greedy updates can limit performance or fail to preserve broader capabilities.

    Source: §3.3 · p. 4, §6 · p. 10

Abstract

from arXiv · show

Multi-domain fine-tuning of large language models requires improving performance on target domains while preserving performance on constrained domains, such as general knowledge, instruction following, or safety evaluations. Existing data mixing strategies rely on fixed heuristics or adaptive rules that cannot explicitly enforce preservation of such capabilities. We propose DynaMiCS, a dynamic mixture optimizer that casts multi-domain fine-tuning as a constrained optimization problem. At each update, DynaMiCS performs short domain-specific probing runs to estimate a slope matrix of local cross-domain effects, capturing how training on each fine-tuning dataset affects each evaluation domain. These estimates are then used to compute mixture weights through optimization over the probability simplex, with the objective of improving target-domain performance while keeping constrained-domain losses below reference levels. Across multi-domain fine-tuning scenarios with varying numbers of target and constrained domains, DynaMiCS achieves stronger target-domain improvements and higher constraint satisfaction than fixed-mixture baselines, at lower computational cost and without reference models, per-example scoring, or manually tuned mixture weights.

1 Introduction

DynaMiCS addresses the tension between specialization and retention by treating fine-tuning mixture selection as constrained optimization. It estimates local cross-domain effects and dynamically chooses mixtures to improve targets while preserving constrained capabilities.

Motivation

  • Fine-tuning must improve specialization domains while retaining general knowledge, instruction following, safety, and other previously learned abilities.Small supervised fine-tuning datasets can promote rapid adaptation, overfitting, and loss of broader generalization.Source: §1 · p. 1
  • Fixed, uniform, dataset-size, and temperature-based mixtures incur costly trial and error and cannot guarantee preservation of critical capabilities.The number of trials needed to cover the mixture simplex grows exponentially with the number of fine-tuning datasets.Source: §1 · pp. 1–2

Contribution

  • DynaMiCS dynamically selects constraint-aware mixture weights from local cross-domain effects during fine-tuning.It estimates a slope matrix through short domain-specific probes, then optimizes the mixture over the probability simplex.Source: §1 · p. 2, Figure 1 · p. 2

Empirical scope

  • The method evaluates 50 multi-domain scenarios across 1–3 target and 3–10 constrained domains, over 7,000 fine-tuning runs.Experiments include Gemma3-12B, Qwen3-8B, and Qwen2.5-3B configurations, including multiple LoRA ranks for the latter.Source: §1 · p. 2

Results

  • DynaMiCS achieves stronger target-domain improvements and higher constraint satisfaction than fixed-mixture baselines at substantially lower cost.The introduction positions this result against fixed-mixture baselines and best-of-many baseline selection.Source: §1 · pp. 2–3

2 Related Work

Prior work includes static heuristics, optimization-based mixtures, and dynamic reweighting, but these approaches generally do not adapt to changing transfer and interference while explicitly enforcing preservation constraints.

Forgetting, overfitting, and data mixtures

  • Fine-tuning data mixtures are used to improve target domains while limiting regressions on stable domains affected by catastrophic forgetting and overfitting.Supporting or general-domain data can act as regularizers alongside specialized data.Source: §2 · p. 3

Static and dynamic mixture selection

  • Mixture selection commonly uses uniform, dataset-size-proportional, or temperature-based sampling, while more principled static methods use scaling-law predictions.Static mixtures cannot react to evolving transfer and interference effects during fine-tuning.Source: §2 · p. 3

Relation to pre-training mixture optimization

  • Pre-training mixture optimization spans proxy-model, benchmark-driven, scaling-law, online reweighting, gradient-alignment, and domain-affinity approaches.These methods constitute a broader pre-training literature distinct from the fine-tuning setting addressed here.Source: §2 · p. 3

3 DynaMiCS: Dynamic, Slope-Informed Data Mixture Optimization

DynaMiCS formulates mixture selection as constrained optimization over target and constrained evaluation losses. It repeatedly estimates local dataset effects with temporary probes, predicts short-horizon losses, and updates simplex weights using a constraint-penalized objective.

3.1 Constrained data-mixture optimization

  • The fine-tuning setting separates datasets used for training from evaluation domains divided into target and constrained sets.Constrained domains may be held-out safety or general-knowledge benchmarks unavailable as training data.Source: §3 · p. 3
  • Mixture weights lie on the probability simplex, and each training batch samples datasets according to those weights.The predicted target losses are minimized while constrained losses are compared with reference losses from the base model.Source: §3.1 · pp. 3–4

3.2 Objective and constraint approximation using finite-differences

  • DynaMiCS estimates a slope matrix by temporarily probing each fine-tuning dataset and measuring its loss change on every evaluation domain.Negative entries indicate beneficial transfer, whereas positive entries indicate interference or potential forgetting.Source: §3.2 · p. 4, §3.3 · p. 5

3.3 Adaptive weight optimization

  • The slope matrix provides a local linear prediction of short-horizon evaluation losses from candidate mixtures.Per-domain effects are assumed additive and extrapolated only until the next mixture update.Source: §3.3 · p. 4
  • DynaMiCS solves a low-dimensional simplex optimization with squared-hinge penalties, selecting feasible weights or the least-violating mixture when necessary.Penalty strength and safety margin control constraint enforcement, and the resulting weights remain active until the next scheduled update.Source: §3.3 · pp. 4–5

4 Experimental Setup

DynaMiCS adaptively re-estimates mixture weights during fine-tuning by probing domains, then evaluates target improvement under explicit constraint and compute-cost criteria across diverse scenarios and models.

DynaMiCS procedure

  • At scheduled updates, Algorithm 2 probes domains to estimate effects, optimizes mixture weights on the simplex, and trains the next step using the resulting mixture.Reference losses are initialized on constrained domains, and the best feasible checkpoint is selected post-hoc.Source: §4.1 · p. 5, §4.6 · p. 7

Evaluation scenarios and models

  • 50 scenarios span targets including reasoning, code, instruction following, medical QA, function calling, and multilingual tasks, with constrained domains evaluated against pre-training references.Three models from 3B to 12B parameters are fine-tuned with LoRA, while the optimization procedure is also applicable to full fine-tuning.Source: §4.1 · p. 5, §4.2 · p. 5

Update schedules

  • Geometric schedules share updates at steps 64, 128, 256, 512, and 1024, while warmup variants add earlier updates at steps 8–32 or 2–32.Probe length follows the interval through c_t = min(H_t, c_max), with c_max = 128, so early updates use short probes.Source: §4.3 · p. 6

Baselines and cost

  • Fixed-weight baselines require choosing allocations without knowing cross-domain interactions, while gradient alignment uses gradient dot products but produces unreliable constraint satisfaction.The study compares dynamic methods and fixed-weight trials on a common compute axis; gradient dot products indicate direction but poorly predict losses after several fine-tuning steps.Source: §4.4 · p. 6, §4.5 · p. 6

Compute cost

  • DynaMiCS incurs per-update probing and reduced evaluation overhead, but later geometric intervals make the capped probe budget relatively small and aligned evaluations can recycle anchor losses.The simplex optimization itself is treated as negligible relative to GPU training and evaluation.Source: §4.5 · p. 7

Metrics and feasibility

  • The primary metric is held-out target perplexity reduction at the best checkpoint satisfying every constraint; infeasible runs receive 0%.Feasibility requires constrained losses at or below reference levels and a decrease in the sum of target losses relative to step 0.Source: §4.6 · p. 7

5 Results

Across models and scenarios, DynaMiCS improves target perplexity reduction over fixed-weight baselines while remaining cost-efficient and adapting its behavior to constraints, architectures, and training schedules.

  • DynaMiCS consistently outperforms fixed-weight baselines, reaching 8–9% versus 0.4% on Qwen2.5-3B, 11% versus 0% on Qwen3-8B, and 24% versus 18% on Gemma3-12B.These comparisons use median test perplexity reduction; Figure 5 also shows DynaMiCS finding feasible checkpoints where baselines are mostly infeasible.Source: §5 · p. 7, Figure 5 · p. 9
  • DynaMiCS achieves its gains at 1.5–2× the cost of one baseline run, while Qwen baselines remain unreliable even at 10× cost and Gemma3-12B baselines require 10× cost to approach 20%.On Gemma3-12B, a single DynaMiCS run at approximately 1.5× cost already exceeds the baseline level reached at 10×.Source: §5 · pp. 7–8, Figure 2 · p. 8
  • Denser warmup schedules improve early adaptation: on Qwen3-8B, median reduction rises from 5.8% without warmup to 11% with light warmup, while light and dense warmup perform similarly.On Qwen2.5-3B, denser warmup improves target quality but slightly reduces feasibility rate.Source: §5 · p. 8
  • Performance remains stable across LoRA ranks on Qwen2.5-3B, with warmup schedules producing 7–9% median reduction; rank 32 lies near the capacity-curve knee.Ranks below 32 underperform, while higher ranks provide no added benefit in the extended sweep.Source: §5 · p. 8
  • As constraints increase from 3 to 7, DynaMiCS remains largely stable while baselines degrade; with 10 constraints both struggle, but DynaMiCS still performs better on Gemma3-12B and Qwen3-8B.Even when feasibility is not reached, the constraint-aware method limits violation relative to the best baseline.Source: §5 · pp. 8–9, Figure 3 · p. 8
  • DynaMiCS adapts allocation over training and across architectures, initially emphasizing target domains before shifting toward FineWeb as a later regularizer.In scenario 28, Glaive dominates steps 2–16, while the three model architectures follow distinct allocation variants.Source: §5 · p. 9, Figure 4 · p. 9

6 Conclusion

DynaMiCS frames multi-domain fine-tuning as constrained optimization, using local cross-domain effects to choose mixtures that improve targets while preserving capabilities. It reports broad gains while acknowledging locality, additive-interaction, evaluation-quality, and trajectory-bias limitations.

  • DynaMiCS dynamically derives constraint-aware mixture weights from short probing runs, balancing target improvement with explicit knowledge-preservation constraints without auxiliary models or manual tuning.Source: §6 · pp. 9–10
  • Across model families, scales, and scenarios, DynaMiCS achieves stronger target performance and maintains constraint satisfaction at a fraction of grid-search cost.Source: §6 · p. 10
  • The method applies constraints through finite differences of evaluation metrics, allowing any metric computable on a held-out batch without requiring a differentiable surrogate.Source: §6 · p. 10
  • Limitations include local linearity, additive-effect assumptions, dependence on representative evaluations, and possible early over-specialization from greedy local updates.Source: §6 · p. 10
  • Fine-tuning must balance specialization gains against catastrophic forgetting of capabilities such as general knowledge, instruction following, and safety behavior.Source: §A.1 · p. 15, §A.2 · p. 15

A.3 Dynamic data mixture selection

Dynamic mixture selection adapts training allocations as domain relationships evolve, but DynaMiCS distinguishes itself by enforcing explicit preservation constraints in the SFT setting. It operates at domain level to reduce overhead relative to finer-grained influence methods and avoids proxy-model strategies used in pre-training.

Motivation

  • Dynamic methods are motivated by evolving domain relationships, because a dataset’s usefulness can change as fine-tuning progresses and representations reorganize.Source: §A.3 · p. 15

Granularity

  • Domain-level control is cheaper than sample-level influence estimation while still permitting mixtures to adapt during training.Source: §A.3 · pp. 15–16

Dynamic constrained selection

  • DynaMiCS estimates each fine-tuning dataset’s local effects on target and constrained evaluation domains, then solves a constrained optimization problem for mixture weights.Source: §A.3 · p. 16, §A.4 · p. 16

SFT distinction

  • Unlike pre-training methods using proxy models, static proportions, or gradient alignment, DynaMiCS targets SFT improvement under explicit constraints on specified domains.Source: §A.4 · p. 16

Evaluation setting

C 50 Evaluation Scenarios

The study evaluates DynaMiCS across 50 scenarios spanning multiple target and constraint counts, models, schedules, and computational settings. Results and implementation choices emphasize feasible target improvement, adaptive probing, and low overhead from geometric schedules.

50 Evaluation Scenarios

  • The benchmark contains 50 scenarios with 1–3 target domains and 3–10 evaluation-only constraint domains, increasing difficulty through the number of constrained capabilities.Source: C 50 Evaluation Scenarios · p. 19

Training setup

Compute cost

Metrics

  • DynaMiCS feasibility excludes the base checkpoint and requires a post-step checkpoint that both improves target perplexity and satisfies every constraint.Source: F Feasibility Rates · p. 25

Results

  • Across schedules, DynaMiCS substantially outperforms fixed-weight baselines in feasibility, although schedule choice affects the trade-off between feasibility and median perplexity reduction.Source: F Feasibility Rates · p. 25, Figure 7 · p. 25

G Sensitivity to Tightness of U

Fixed-interval sweeps show that more frequent mixture updates generally improve target perplexity reduction, while geometric schedules provide better cost efficiency by updating densely early and spacing updates later.

Experimental design

Fixed-interval sweep

Cost efficiency

H LoRA Rank Sensitivity

DynaMiCS remains effective across LoRA ranks, with performance largely stable from ranks 8 to 64 and limited gains beyond rank 32.

  • 7–9% median PPL reduction remains stable across LoRA ranks 8, 32, and 64, with a consistent advantage over fixed-weight baselines.Source: H LoRA Rank Sensitivity · p. 28
  • Ranks 32, 64, and 128 perform comparably in feasibility and PPL reduction, indicating little benefit from adapter capacity beyond rank 32.Source: H LoRA Rank Sensitivity · p. 28
  • Ranks 2 and 8 achieve lower feasibility rates than rank 32, consistent with insufficient capacity, while higher ranks remain comparable.Source: H LoRA Rank Sensitivity · p. 28
  • Figure 10 compares per-scenario PPL reductions on Qwen2.5-3B and median reduction against total compute for ranks 8, 32, and 64.Source: Figure 10 · p. 28

I Results by Number of Constraints

DynaMiCS preserves a clear advantage as the number of constraints increases, degrades more gracefully when feasibility is impossible, and trades extra compute for greater robustness with curve fitting.

Results by number of constraints

Infeasible scenarios

Curve-fitting predictor

Gradient alignment

  • Gradient alignment uses averaged evaluation gradients and Adam update directions, but its instantaneous slope estimates accumulate prediction errors over longer horizons.Source: §L.1 · p. 34, §L.2 · p. 34

L.3 Results

Gradient alignment consistently underperforms DynaMiCS because its instantaneous estimates are poorly calibrated for multi-step loss prediction and constraint optimization. The evaluation also extends DynaMiCS beyond perplexity constraints to task-accuracy constraints.

Gradient alignment

  • Gradient alignment consistently underperforms DynaMiCS across three models and struggles to calibrate slope magnitudes for the optimization horizon.Source: §L.3 · p. 34, Figure 17 · p. 35
  • Its instantaneous rate estimates can identify beneficial directions but produce overly conservative or overly aggressive mixture allocations.Source: §L.3 · p. 34

Accuracy constraints

N Evolution of weight allocation

Dynamic weight allocations vary with model architecture and scenario difficulty, but FineWeb receives larger later-training weights in several constrained settings.

Related papers

Cited by

Loading 2605.10770v1…