PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Human-Readable Text Necessary for Effective LLM Fine-Tuning?

Jinhao Zhang, Zeyu Liu, Zicheng Yan, Yunquan Zhang, Daning Cheng, Song Tang

arXiv:2609.35868v1cs.AI

TL;DR

The paper asks whether LLM fine-tuning can remain effective without human-readable inputs. It proposes DASA, which uses a frozen reference model’s internal signals to optimize continuous synthetic inputs. Across the evaluated settings, these inputs provide adaptation comparable to real text, often outperform GRADMM, and reduce synthesis time.

  • Problem

    It remains unclear whether human-readable expressions are necessary for models to benefit from fine-tuning, since readability and downstream training utility are distinct properties.

    Source: §1 · p. 1

  • Method

    DASA uses feedback from a frozen reference model to optimize continuous synthetic inputs for useful adaptation signals rather than source-text reconstruction or linguistic fluency.

    Source: §1 · p. 2

  • Results

    Across six models and six benchmarks, DASA embeddings provide LoRA adaptation often comparable to real-text fine-tuning, outperform GRADMM in most evaluated model–task settings, and cut generation time by 3.6–4.9× with comparable peak GPU memory.

    Source: §1 · p. 2, §4.5 · p. 8, §5 · p. 10

  • Takeaways & Limitations

    In the evaluated adaptation settings, useful training signals can be carried by model-conditioned continuous inputs without preserving readable text.

    Source: §5 · p. 10

    Three runs per setting are insufficient to establish statistical significance for the observed mean gains.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

Is human readability necessary for effective fine-tuning of large language models? We investigate whether model-conditioned training representations can preserve or improve adaptation utility without requiring a human-readable textual form. We propose Desired-Update-Aligned Synthetic Data (DASA), which uses activation-gradient feedback from a frozen reference model to guide the optimization of continuous synthetic input embeddings. Inspired by the role of activation gradients in local risk reduction, DASA targets useful adaptation updates rather than source-text reconstruction or linguistic fluency. The resulting embeddings are used directly for downstream fine-tuning; discrete token projections are employed only for qualitative inspection. Experiments on six models from the Llama and Qwen families, ranging from 1B to 32B parameters, cover six benchmarks spanning knowledge, mathematical reasoning, code generation, and commonsense reasoning. Under matched LoRA adaptation settings, DASA achieves performance comparable to the source natural-language data and surpasses it in multiple configurations, while outperforming GRADMM in most comparisons. Further experiments cover general-domain and task-specialized source data. Under the evaluated synthesis settings, DASA provides a $3.6$--$4.9\times$ speedup over GRADMM with comparable peak GPU memory.

1 INTRODUCTION

The introduction asks whether effective fine-tuning requires human-readable inputs, distinguishing readability from downstream training utility. It proposes DASA and reports comparable or better adaptation results against natural-language data across evaluated settings.

1 INTRODUCTION

  • The work asks whether human-readable expressions are necessary for effective fine-tuning, treating readability and downstream training utility as distinct properties.Training utility is defined by performance on independent evaluation tasks after fine-tuning.Source: §1 · pp. 1–2
  • DASA uses internal feedback from a reference model on source data to guide alternative training sequences toward retaining or improving adaptation value.The motivating design shifts the objective from preserving human-readable form to retaining signals useful for adaptation.Source: §1 · p. 2
  • DASA constructs alternative fine-tuning data by aligning model-internal signals, without optimizing for source-text reconstruction or fluent language.The introduction frames unreadability as relaxed linguistic constraints, not as a mechanism for improving performance.Source: §1 · p. 2
  • DASA yields adaptation gains comparable to original natural-language data and surpasses it on multiple model–task combinations, while outperforming GRADMM in most evaluated settings.The evaluation spans Llama and Qwen models from 1B to 32B parameters, with experiments covering general-domain and task-relevant data.Source: §1 · p. 2
  • The contributions include evaluating DASA across model scales, tasks, data budgets, source types, repeated runs, and generation costs.The introduction states that DASA can outperform original text in some settings.Source: §1 · p. 2

2 RELATED WORK

Related work situates DASA within text dataset-distillation approaches and research on model-internal signals and adaptation. Its distinguishing focus is optimizing the first-order effect of continuous synthetic inputs on reference risk, without requiring source-text reconstruction or fluency.

2 RELATED WORK

  • Text dataset-distillation methods include continuous embedding optimization, discrete text generation, and training-trajectory matching.GRADMM matches parameter gradients and projects optimized embeddings into low-perplexity text in its main procedure [Nguyen et al., 2025].Source: §2 · p. 2
  • Prior work also studies difficult-to-read instructions, learnable prompts, model-internal synthesis signals, local update directions, and efficient adaptation with adapters and LoRA.The passage cites these strands respectively as Duan et al. (2025), Shin et al. (2020), Yin et al. (2020), Cheng et al. (2026), and Houlsby et al. (2019).Source: §2 · p. 2
  • DASA optimizes continuous synthetic inputs for a favorable first-order change in reference risk, rather than reconstructing source text or producing fluent language.The residual expansion analysis motivates this objective but does not guarantee finite-step fine-tuning outcomes.Source: §2 · p. 3

3 METHODOLOGY

DASA turns a local directional criterion into an objective for constructing continuous synthetic inputs, then uses those inputs directly for downstream fine-tuning. Its theoretical motivation is an existence result, not a guarantee that a particular dataset or optimizer will find an improving direction.

3.1 START POINT

  • The cited residual-expansion result is existential: it does not establish that a particular dataset or optimizer will find an improving direction.The stated theorem relies on conditions including differentiability and realizability.Source: §3.1 · p. 3
  • DASA optimizes continuous synthetic inputs so their induced parameter-space update direction produces a favorable first-order change in reference loss.The empirical reference average substitutes for the population expectation, and gradients of the objective are propagated through the induced direction to update the inputs.Source: §3.1 · p. 3

3.2 ALGORITHM DESIGN

  • The method outputs a continuous input tensor for subsequent fine-tuning, initialized from reference token-sequence embeddings with standard shifted-token prediction targets.The tensor represents n sequences of L continuous embeddings each; the shifted targets are standard causal-language-modeling targets, not separately annotated labels.Source: §3.2 · p. 4
  • During synthesis, the frozen model and prediction targets remain fixed while SGD updates synthetic embeddings using clipped induced directions and a first-order reference-risk objective.The induced direction is clipped to an ℓ2 norm of at most 2.0, and optimized embeddings are converted to BF16.Source: §3.2 · p. 4
  • For downstream evaluation, the continuous embeddings are paired with retained prediction targets and fine-tuned directly with LoRA, without projection to discrete tokens.Evaluation uses independent tasks and compares with original-text fine-tuning under the same adaptation configuration and training budget.Source: §3.2 · p. 4

4 EXPERIMENTS

Across six Llama and Qwen models and six benchmarks, the experiments compare DASA with natural-text and synthetic-data baselines under matched adaptation settings. DASA’s advantages span model scales, tasks, calibration sources, and synthesis efficiency, with additional tests of run variability, LoRA settings, and synthetic-input readability.

4.2 MAIN RESULTS

  • At the 200 × 2048 budget, DASA outperforms GRADMM across all six tasks for Llama-3.2-1B, Llama-3.1-8B, Qwen3-14B, and Qwen3-32B, with gains spanning knowledge, reasoning, coding, and commonsense tasks.The reported results suggest useful updates persist as the synthetic set grows and can improve task performance without calibration samples drawn from the target benchmark distribution.Source: §4.2 · pp. 5–6
  • DASA beats GRADMM in 31/36, 35/36, and 35/36 model–task configurations as the C4 calibration budget increases from 50 × 2048 to 100 × 2048 and 200 × 2048.DASA also improves over the unadapted model across model families and scales.Source: §4.2 · p. 5, Table 1 · p. 6

4.3 ROBUSTNESS ACROSS RANDOM SEEDS

  • Across six representative settings, DASA has a higher mean than GRADMM and Random, and the highest mean in four; three runs per setting are insufficient to establish statistical significance.DASA’s standard deviations are generally comparable to those of Real-text and GRADMM, but are higher on Qwen3-14B MBPP.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 TASK-SPECIALIZED CALIBRATION DATA

  • With task-specialized calibration, DASA beats GRADMM in all 12 model–task comparisons and has the strongest overall result in eight, while Real-text remains slightly stronger on GSM8K and MBPP.The advantage persists when calibration data come from each benchmark’s training split, rather than generic C4 data.Source: §4.4 · p. 7, Table 3 · p. 8

4.5 COMPUTATIONAL EFFICIENCY

  • 3.6–4.9× faster synthesis: DASA takes 5.8–23.3 minutes across the reported budgets, while peak GPU memory remains similar for both methods at 17.3–17.9 GB.Measurements use matched synthesis settings on one NVIDIA A800 GPU with FP16 precision.Source: Table 4 · p. 8, §4.5 · p. 8

4.6 SENSITIVITY ANALYSIS

  • DASA performs competitively across LoRA ranks and downstream step counts; smaller ranks generally benefit from more steps, while larger ranks reach strong performance sooner and can lose accuracy with continued adaptation.Several intermediate configurations achieve similar accuracy, suggesting rank and steps are partly substitutable.Source: Figure 1 · p. 9, §4.6 · p. 9

4.7 QUALITATIVE ANALYSIS OF SYNTHETIC DATA

  • The decoded synthetic sample contains fragmented words, mixed scripts, and locally incoherent tokens, but downstream adaptation uses continuous embeddings rather than their token projection.The authors assess training utility through task performance because token projection loses information from the continuous embeddings.Source: §4.7 · p. 9, Figure 2 · p. 10

5 CONCLUSION

The conclusion finds that continuous synthetic inputs can support useful adaptation without readable language in the evaluated settings. DASA is often comparable to real-text fine-tuning, outperforms GRADMM in most evaluated settings, and reduces generation time.

5 CONCLUSION

  • Across six models, six benchmarks, and three data budgets, DASA often matches real-text adaptation and outperforms GRADMM in most evaluated model–task settings.The comparisons use downstream LoRA adaptation.Source: §5 · p. 10
  • 3.6–4.9× lower generation time than GRADMM under matched synthesis workloads, with comparable peak GPU memory.Source: §5 · p. 10
  • DASA optimizes continuous synthetic input embeddings for their first-order effect on a reference loss while keeping the pretrained model fixed during synthesis.Source: §5 · p. 10
  • In the evaluated adaptation settings, useful training signals can be carried by model-conditioned continuous inputs without preserving readable language.Source: §5 · p. 10
  • The directional objective is a local criterion, not a guarantee of finite-step fine-tuning outcomes; its relation to downstream updates and cross-model transfer remains future work.Source: §5 · p. 10

A BASIC SETUP AND NOTATION

This section sets up DASA's local analysis around a fixed pretrained model and defines the synthetic-input direction and reference-loss quantities. Its assumptions and local improvement criterion do not guarantee finite-step fine-tuning outcomes.

A BASIC SETUP AND NOTATION

  • For synthetic inputs Z_s, activation gradients define a parameter-space tangent direction whose first-order effect on reference activations is evaluated through the empirical reference-loss variation.The setup uses synthetic activations and prediction targets to form the synthetic activation gradients.Source: A BASIC SETUP AND NOTATION · p. 13
  • A negative first-order risk variation indicates a locally improving direction only for a sufficiently small positive step under the stated conditions, not a finite-step guarantee.Source: A BASIC SETUP AND NOTATION · p. 13
  • The activation-gradient statistics support a decomposition and bound for D_Q(v); they are not optimization targets for Z_s.Source: A BASIC SETUP AND NOTATION · p. 13
  • The analysis assumes differentiable mappings and finite second moments so the stated expectations, covariances, and first-order quantities are well defined.Source: A BASIC SETUP AND NOTATION · p. 13
  • Z_s denotes continuous synthetic inputs, S_syn the stored inputs used for fine-tuning, v the induced tangent direction, and u_v(z) its first-order activation change.Source: A BASIC SETUP AND NOTATION · p. 13
  • D_Q(v) is first-order risk variation, while μ(Q) and Σ_Q denote the mean and covariance of activation gradients under Q.Source: A BASIC SETUP AND NOTATION · p. 13
  • During synthesis, the pretrained model and auxiliary residual branch remain fixed; only Z_s is updated, and LoRA fine-tuning follows separately.Source: A BASIC SETUP AND NOTATION · p. 14

B THEORETICAL BACKGROUND

The background develops a residual-branch framework for reasoning about local risk reduction, then distinguishes that local argument from guarantees about a trained model’s test loss.

Auxiliary residual branch

  • The residual branch is added at a selected insertion point to the fixed reference model, with a zero-output initialization that preserves the original function.The framework uses the branch’s realizable representation change to define the direction whose risk effect is evaluated.Source: B THEORETICAL BACKGROUND · p. 14

Population risk and the population jumpboard

  • Under the stated differentiability and realizability conditions, an admissible direction with negative population-risk derivative yields an expanded comparison model with lower population risk.This establishes that an improving point exists in the expanded model class, not that an optimization algorithm will find it.Source: B THEORETICAL BACKGROUND · p. 14

Empirical risk and the empirical jumpboard

  • A small step along a training-selected direction can reduce test loss when the independent test-gradient mean aligns positively with the training mean, provided the branch realizes the perturbation.The empirical jumpboard is distinct from the final expanded model and depends on the training sample.Source: B THEORETICAL BACKGROUND · p. 14

Probability of gradient alignment

  • The gradient-alignment bound separates failures into training-mean deviation and test-mean fluctuation, but does not lower-bound the magnitude of test-loss improvement.Its stated assumptions include independent training and test samples and a covariance bound.Source: B THEORETICAL BACKGROUND · p. 15

From a jumpboard to the final expanded model

  • A strict test-loss improvement for the final expanded model follows only when realized gains exceed the generalization terms under the cited model-selection and uniform-generalization conditions.The argument first compares training losses with the jumpboard, then applies a uniform train–test loss bound.Source: B THEORETICAL BACKGROUND · p. 15

C COVARIANCE CONTROL UNDER LORA

The LoRA analysis bounds the covariance contribution to a first-order update and explains how synthetic inputs can shape the update direction toward favorable mean improvement.

C COVARIANCE CONTROL UNDER LORA

  • Under the standard LoRA initialization B = 0, the perturbation VA has no first-order contribution to the residual branch.The analysis specializes its first-order decomposition to a residual adaptation branch with LoRA factors A and B.Source: C COVARIANCE CONTROL UNDER LORA · p. 15
  • The covariance-control bound makes the first-order implication strict when the resulting upper bound is negative.The section derives the bound by decomposing the first-order variation and bounding its factors.Source: C COVARIANCE CONTROL UNDER LORA · pp. 16–17
  • Because synthetic inputs induce the LoRA direction, optimizing them can favor mean improvement while controlling direction geometry relative to the covariance residual.For fixed Q, the activation statistics are fixed, whereas v = v(Zs) can be shaped during synthesis; a fixed real-text corpus does not directly optimize this trade-off.Source: C COVARIANCE CONTROL UNDER LORA · p. 17

D TRAINING DETAILS

The training details hold downstream LoRA adaptation settings constant across models, tasks, and methods, while optimizing only DASA’s continuous synthetic inputs during synthesis.

Downstream LoRA adaptation

  • Downstream adaptation freezes pretrained backbones and uses LoRA with rank 32, scaling 32, and dropout 0.05 for 32 AdamW steps.The shared configuration also specifies the adapted projection layers, optimizer settings, sequence length, precision, batch size, and gradient accumulation.Source: D TRAINING DETAILS · p. 17

Synthetic-data optimization

  • DASA and GRADMM use identical sequence counts, sequence lengths, synthesis steps, and downstream adaptation configurations.The synthetic inputs are stored in BF16 and used directly as input embeddings for downstream adaptation.Source: D TRAINING DETAILS · p. 17
Loading 2609.35868v1…