PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Repeat Yourself: Self-Supervised Fine-Tuning for Coverage

Eric Fithian, Kirill Skobelev, X. Y. Han

arXiv:2609.31688v2cs.CL

TL;DR

In verifiable domains, coverage can matter more than an individual attempt’s pass rate, yet post-training can concentrate outputs and temperature has limited control over diversity. DRY-SFT sequentially elicits solutions conditioned on prior attempts and fine-tunes on them independently, raising pass@100+ across three coding benchmarks. Its gains are associated with lower structural diversity in the base model, while pass@1+ decreases across datasets.

  • Problem

    In verifiable tasks, post-training can favor pass@1 over pass@k, while temperature-based sampling has limited ability to increase valid output diversity.

    Source: §1 · pp. 1–2

  • Method

    DRY-SFT sequentially generates solutions conditioned on earlier attempts, then fine-tunes on each attempt independently without rewards, verifiers, or correctness filtering.

    Source: §1 · pp. 2–3, §3 · p. 4

  • Results

    Across three coding benchmarks, DRY-SFT raises pass@100+ by 10.8, 12.5, and 12.4 points over the base model and increases structural diversity.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Across nine open-weight models, base-model structural diversity is significantly related to DRY-SFT effectiveness, offering a practical test to estimate coverage gains.

    Source: §1 · p. 3, §5 · p. 8

    DRY-SFT decreases pass@1+ across all datasets.

    Source: §5 · p. 8

Abstract

from arXiv · show

In verifiable domains such as math and coding, finding one correct solution among many attempts can matter more than the pass rate of each attempt. Post-training can concentrate large language model outputs around a few modes, while increasing sampling temperature has limited effectiveness. We introduce Don't Repeat Yourself Supervised Fine-Tuning (DRY-SFT), a post-training method that increases output diversity and coverage: the probability of at least one correct solution among many attempts. DRY-SFT has two stages. First, for each problem, sequentially generate K solutions, showing the model all prior attempts and asking for a different solution. Second, fine-tune on each attempt independently, removing prior attempts from the context. The process uses no reward, verifier, or correctness filter. On HumanEval+, MBPP+, and DS-1000, DRY-SFT raises pass@100 by 10.8, 12.5, and 12.4 percentage points, respectively, at a small cost to pass@1. Structural diversity, measured by abstract syntax tree edit distance among passing solutions, rises significantly on all three benchmarks. DRY-SFT also solves 244 of 600 problems that the base model did not solve in the same 200 attempts. Across nine open-weight models, lower structural diversity of the base model significantly predicts larger DRY-SFT gains, indicating that the method is especially effective on more mode-collapsed models.

1 INTRODUCTION

In verifiable tasks, coverage depends on generating meaningfully different attempts, but post-training can concentrate outputs and temperature alone offers limited control. DRY-SFT trains on sequentially elicited solutions to broaden output diversity and improve coverage without rewards, verifiers, or correctness filtering.

  • Coverage, or pass@k, is the probability that at least one of k samples is correct, and it rises only when later samples solve what earlier ones could not.Repeated sampling over only a few solution modes can cause pass@k to plateau at low k; cited work also reports post-training methods that prioritize pass@1 over pass@k.Source: §1 · pp. 1–2
  • DRY-SFT raises pass@100+ by 10.8, 12.5, and 12.4 points over the base model on three coding benchmarks.The introduction also reports that neither a matched i.i.d. control nor P-SFT reproduces these gains.Source: §1 · pp. 2–3
  • DRY-SFT sequentially elicits K solutions conditioned on prior attempts, then fine-tunes on each attempt independently, without a reward, verifier, or correctness filter.The method is presented as changing the output distribution through self-generated rollouts rather than reweighting it at decoding time.Source: §1 · pp. 2–3
  • Across nine open-weight models, lower base-model structural diversity correlates with larger DRY-SFT gains (R2=0.65, p=0.0088).Source: §1 · p. 3

2 RELATED WORK

Related approaches seek coverage through rewards, filtering, privileged context, altered training objectives, or inference-time prompting. DRY-SFT instead constructs diverse self-generated training data through sequential conditioning, without changing the objective or requiring a verifier or pre-existing multi-response corpus.

Rewarding coverage with a verifier

  • Verifier-based coverage rewards and entropy-regularized exploration target pass@k directly, but both require a verifier, unlike DRY-SFT.Source: §2 · pp. 3–4

Filtering self-generated data

  • Correctness-filtered self-training methods refit on accepted attempts, while GiFT conditions on problem rephrasings rather than prior solutions; DRY-SFT’s single-cycle experiments do not establish stability under repeated self-training.Source: §2 · p. 3

Generation under privileged context

  • Privileged-context approaches train answers from prompts after generating reasoning traces, while SESA elicits sequential sketches for later rollouts; DRY-SFT trains the sequential diversity step into the model with SFT.Source: §2 · p. 3

Changing the objective, or presupposing variation in the data

  • P-SFT, SSFT, SED-SFT, and related methods alter the objective or rely on multiple existing solutions, whereas DRY-SFT requires neither a changed objective nor a pre-existing multi-response corpus.Source: §2 · pp. 3–4

Injecting variation during inference

  • Inference-time context-variation methods avoid training, but require applying their prompting method at every generation; DRY-SFT trains diversity into the model for independent sampling from a bare prompt.Source: §2 · p. 4

3 DON’T REPEAT YOURSELF SUPERVISED FINE-TUNING

DRY-SFT elicits solutions sequentially with prior attempts in context, then trains on each solution conditioned only on the original problem. This fits a mixture of conditionally generated solutions, unlike an i.i.d. self-training control that reproduces the model’s existing distribution.

Generation and fine-tuning

  • Stage 1 generates K solutions sequentially with prior attempts in context; Stage 2 removes those attempts and the diversity prompt, then fine-tunes on N · K pairs without reward, verification, or correctness filtering.At test time, solutions are sampled independently and in parallel using a standard prompt.Source: §3 · p. 4

Objective

  • Stage 1 draws the first solution from πθ(· | x) and each later solution from πθ(· | x, y<i), where y<i denotes earlier attempts for problem x.Source: §3 · p. 4
  • Training on the sequentially conditioned solutions fits a mixture over K solutions, while the matched i.i.d. control fits the distribution the model already has.The passage links this distinction to evidence that fitting each problem to a single reasoning mode can degrade coverage even when the dataset is balanced across modes.Source: §3 · p. 4

4 EXPERIMENTS

The experiments evaluate DRY-SFT on three coding benchmarks using coverage and AST-based structural diversity, alongside comparisons with controls, temperature changes, held-out data, and other models. DRY-SFT improves coverage and structural diversity, generalizes across settings, and its gains are associated with lower base-model diversity.

4 EXPERIMENTS

  • Experiments use HumanEval+, MBPP+, and DS-1000, comparing DRY-SFT with the base model and two fine-tuning controls using five unfiltered self-generated solutions per problem.Coverage is measured by pass@k+; structural diversity is normalized mean pairwise AST tree-edit distance among passing solutions.Source: §4 · pp. 4–5

4.1 PASS@K PERFORMANCE

  • DRY-SFT raises pass@100+ by 10.8 points on HumanEval+ (0.851 to 0.960), 12.5 on MBPP+ (0.749 to 0.874), and 12.4 on DS-1000 (0.507 to 0.632).The i.i.d. control and P-SFT do not produce the same improvement on any benchmark; DRY-SFT pass@1+ is lower, but its coverage curves cross the base model by k=2 on HumanEval+ and MBPP+ and k=3 on DS-1000. It also uniquely solves 244 problems the base model never solves within the same 200 attempts.Source: §4.1 · p. 5, Figure 2 · p. 6

4.2 MEANINGFUL DIVERSITY

  • Normalized AST edit distance rises from 0.072 to 0.264 on HumanEval+, 0.051 to 0.286 on MBPP+, and 0.057 to 0.308 on DS-1000.The matched i.i.d. control and P-SFT have lower diversity values on all three benchmarks; sequentially conditioned targets have still higher values than the fine-tuned model.Source: §4.2 · pp. 5–6, Figure 2 · p. 6

4.3 TEMPERATURE

  • On HumanEval+, the base model peaks at pass@100+ = 0.910 at T=2.6, while DRY-SFT’s point estimate is 5.0 points higher and its structural diversity is 0.264 versus 0.175.The coverage intervals overlap, so the coverage difference is a point-estimate comparison; no temperature reaches DRY-SFT’s structural-diversity level.Source: §4.3 · p. 6, Figure 3 · p. 7

4.4 HELD-OUT AND CROSS-DATASET PERFORMANCE

  • On held-out data, pass@100+ rises from 0.888 to 0.956 on HumanEval+, 0.751 to 0.881 on MBPP+, and 0.523 to 0.696 on DS-1000.Cross-dataset evaluation yields pass@100+ = 0.871 when trained on HumanEval+ and evaluated on MBPP+, only −0.003 versus same-domain training; the reverse setup differs by −0.015.Source: §4.4 · p. 7, Figure 4 · p. 7

4.5 WHEN DRY-SFT WORKS

  • Across nine open-weight models, base structural diversity explains 65% of the variation in DRY-SFT’s change in pass@100+ (R2=0.65, p=0.0088).The six models with the lowest base diversity gain between 2.8 and 21.5 pass@100+ points; the fitted slope is −6.14 points per 0.1 of base diversity, with 95% CI [−10.18, −2.10].Source: §4.5 · p. 8, Figure 5 · p. 8, §4.4 · p. 8

5 CONCLUSION

The conclusion reports that DRY-SFT improves pass@100+ and structural diversity across three coding benchmarks, while identifying limits and directions for further work. It also notes that reported runs share generation and fine-tuning settings across methods.

5 CONCLUSION

  • DRY-SFT improves pass@100+ and structural diversity on three coding benchmarks, transfers gains across benchmarks, and relates base-model diversity to its effectiveness.The conclusion also reports that increasing temperature does not fully recover the gains and can eventually degrade pass@100+.Source: §5 · p. 8
  • DRY-SFT decreases pass@1+ across datasets, did not improve coverage for already-diverse models, and was evaluated only in coding domains.Stage 1 generation is sequential and costs more than drawing K independent samples, but this cost occurs only during data generation; inference has no added cost.Source: §5 · p. 8
  • Whether elicited variation resides in base-model weights or is computed during sequential generation remains an open question.The authors suggest resolving this could explain why DRY-SFT works and help optimize its mechanism.Source: §5 · p. 9
  • Reinforcement learning could complement DRY-SFT after it or by alternating with it, because reinforcement learning moves pass@k toward pass@1.Source: §5 · p. 9
  • Future work could tune sequential rollout prompts to spread outputs along a chosen diversity axis, such as lexical versus semantic variation.Source: §5 · p. 9

A HYPERPARAMETERS

  • DRY-SFT, P-SFT, and the matched i.i.d. SFT control share generation and fine-tuning settings; their target-generation procedures differ.Table A.1 lists the settings used for reported runs.Source: A HYPERPARAMETERS · p. 11

B CHOOSING THE TRAINING EPOCH

Checkpoint scoring shows that the fixed three-epoch budget can train too long, while validation-based epoch selection is robust to a larger holdout. The selected epoch nevertheless depends on K and training-set size, and is not a benchmark-only property.

B CHOOSING THE TRAINING EPOCH

  • Each fine-tuned arm holds out 10% of problems, trains for five epochs, and restores the checkpoint with the lowest held-out cross-entropy.The split is by problem because K targets from one problem share a prompt; Table A.1 lists reported-run hyperparameters.Source: B CHOOSING THE TRAINING EPOCH · p. 11, Table A.1 · p. 12
  • 0.963 is HumanEval+ pass@100+ at epoch 2, versus 0.957 at epoch 1 and 0.955 at epoch 3; epoch 3 is worst on both benchmarks.On MBPP+, epoch 2 exceeds epoch 3 by +0.012 points, with a paired problem-bootstrap interval of [+0.001, +0.024].Source: B CHOOSING THE TRAINING EPOCH · p. 12
  • A 30% holdout selected the same epoch on every arm, and the minimum-validation-loss epoch was among those maximizing pass@100+ among scored epochs.The authors note that cross-entropy does not measure the goal of spreading across solution structures.Source: B CHOOSING THE TRAINING EPOCH · p. 12
  • Epoch count is not fully separable from K because update counts scale with training-set size, and selected epochs vary with the amount of training data.Held-out arms select later epochs than their main counterparts: 3 versus 2 on HumanEval+ and 2 versus 1 on MBPP+.Source: B CHOOSING THE TRAINING EPOCH · p. 13

C CHOOSING THE CHAIN DEPTH K

Across the chain-depth sweep, shorter chains usually match or improve coverage over K=10, while deeper chains can cost pass@1+. The reported runs use K=5, though the sweep does not separate chain depth from the number of training targets.

C CHOOSING THE CHAIN DEPTH K

  • Every tested depth raises pass@100+ over the base model by 7.9 to 12.3 points, and 11 of 12 contrasts favor shorter chains over K=10.Four shorter-chain contrasts exclude zero: K=3, K=5, and K=7 on DS-1000, and K=5 on MBPP+; no contrast favors K=10 by an interval excluding zero.Source: C CHOOSING THE CHAIN DEPTH K · p. 13, Table C.1 · p. 13
  • pass@1+ declines overall as K grows; on DS-1000, the regression shrinks from −7.4 points at K=10 to −4.6 points at K=2.On HumanEval+ and MBPP+, the pass@1+ regression is not resolvable from zero at K ≤3.Source: C CHOOSING THE CHAIN DEPTH K · p. 13
  • The reported runs use K=5, which sits at the top of the coverage plateau and avoids the larger pass@1+ cost seen at greater depth.K=10 is the weakest setting on both axes, and buys no additional coverage.Source: C CHOOSING THE CHAIN DEPTH K · p. 13
  • The sweep varies conditioning depth and target count together, so it cannot distinguish those effects from drift under more optimizer updates.A fixed-total-target chain-length comparison was not run; the table's sweep also holds training at three epochs rather than using the validation-selected protocol.Source: C CHOOSING THE CHAIN DEPTH K · pp. 13–14, Table C.1 · p. 13

D TARGET QUALITY ALONG THE CHAIN

Pass rates tend to be lower at later positions in a K=20 chain, but vary nonmonotonically. The lower endpoint is consistent with a correctness–diversity tradeoff, not evidence that later-target quality causes the K=5 optimum.

D TARGET QUALITY ALONG THE CHAIN

  • In a K=20 chain, pass rate is 0.787 at position 1 and 0.616 at position 20, with substantial nonmonotonic variation between them.The figure reports position-wise pass rates on the extended test suite with 95% confidence intervals.Source: D TARGET QUALITY ALONG THE CHAIN · p. 14, Figure D.1 · p. 14

E CORRECTNESS FILTERING BEFORE STAGE 2

Correctness filtering before Stage 2 changes coverage little overall in the held-out comparison, and both filtered and unfiltered variants retain coverage gains over the held-out base. The comparison has scope limits, including one training seed per arm.

E CORRECTNESS FILTERING BEFORE STAGE 2

  • The filtered arm keeps only targets passing the applicable tests, while Stage 1 still conditions later generations on all earlier attempts, including failed ones.The comparison uses the same K=5 candidates and seed-0 50/50 task splits, then evaluates on the untouched task half.Source: E CORRECTNESS FILTERING BEFORE STAGE 2 · pp. 14–15
  • At k=100, filtering changes coverage by +0.1, −0.6, and +3.2 percentage points on HumanEval+, MBPP+, and DS-1000, respectively.Paired intervals include zero on HumanEval+ and MBPP+, but exclude zero in favor of filtering on DS-1000; at k=1, filtering improves all three benchmarks with intervals excluding zero.Source: E CORRECTNESS FILTERING BEFORE STAGE 2 · p. 15, Figure E.1 · p. 15, Table E.1 · p. 15
  • Both variants retain substantial coverage gains over the held-out base, so the gains require neither correctness filtering nor fitting test-failing Stage 2 targets.Filtering retains 276/410, 586/945, and 689/2500 candidate targets on HumanEval+, MBPP+, and DS-1000, respectively, without replacing discarded targets.Source: E CORRECTNESS FILTERING BEFORE STAGE 2 · p. 15
  • The filtering comparison does not isolate incorrect-target effects at a fixed training budget, rule out failed attempts' role in Stage 1 conditioning, or establish equivalence across seeds.Intervals quantify held-out task uncertainty for one training seed per arm.Source: E CORRECTNESS FILTERING BEFORE STAGE 2 · p. 15, Table E.1 · p. 15

F PER-MODEL COVERAGE CURVES

  • Figure F.1 compares each model's pass@k+ curves for DRY-SFT against its own matched i.i.d. SFT control; the models are ordered by base structural diversity.The supplied passage describes the comparison and protocol, not its per-model outcomes.Source: Figure F.1 · p. 16, F PER-MODEL COVERAGE CURVES · p. 16

G PROMPT TEMPLATES

HumanEval+ Stage 1

MBPP+ Stage 1

HumanEval+ Stage 2 and evaluation

The HumanEval+ Stage 2 prompt asks for a function body only, while the system prompt requests a correct Python function implementation.

HumanEval+ Stage 2 and evaluation

SYSTEM PROMPT

  • The system prompt requests a correct Python function implementation and output consisting only of the full function definition.Source: SYSTEM PROMPT · p. 19

MBPP+ evaluation

The supplied material gives code-generation instructions: one requests a complete Python function, another requests code to replace an unfinished portion, and a heading labels a Stage 2 evaluation.

  • One prompt requests only valid Python code for a full function definition, with no explanation or markdown.Source: SYSTEM PROMPT · p. 19
  • The cited passage identifies the section heading as DS-1000 Stage 2 and evaluation.Source: SYSTEM PROMPT · p. 19
  • Another prompt asks for code to replace an incomplete portion and prefers a Python code block.Source: SYSTEM PROMPT · p. 19

Related papers

Loading 2609.31688v2…