PaperlayerView exact revision on arXiv

Source-linked AI summary

ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training

Zhihao Zhang, Mingqi Wu, Qiaole Dong, Enyu Zhou, Shuo Li, Boyang Liu, Jiazheng Zhang, Honglin Guo, Xin Guo, Shaofan Liu, Junzhe Wang, Dingwei Zhu, Zhiheng Xi, Minlong Peng, Yuan Hua, Qi Zhang, Tao Gui, Xuanjing Huang

arXiv:2609.16639v1cs.AI

TL;DR

Continual post-training must learn new capabilities without sacrificing prior ones, but SFT and on-policy methods separately favor supervision or policy proximity. ReDraft revises failed on-policy rollouts with an expert response as reference, verifies the revisions, and trains on the retained targets. Across three tasks and two model scales, it improves acquisition and retention relative to SFT and OPSD, while its effectiveness depends on correction competence and an automatic verifier.

  • Problem

    Continual post-training needs explicit supervision for cold-start learning while maintaining proximity to the current policy.

  • Method

    ReDraft revises incorrect current-policy rollouts using expert responses as references, verifies the revisions, and fine-tunes on successful targets.

  • Results

    56.9 points versus 52.9 for SFT on target-task gain, while prior-task loss falls from 16.6 to 1.5 points; ReDraft also outperforms OPSD on both axes.

  • Takeaways & Limitations

    Self-revising the model’s own output combines explicit correction with policy proximity across the evaluated tasks and model scales.

  • Takeaways & Limitations

    ReDraft requires correction competence and an automatic verifier, and its one-time revised targets become staler as training changes the policy.

Abstract

from arXiv · show

Continual post-training of large multimodal models should add new capabilities while preserving those from pre-training, and the two goals pull in opposite directions. SFT gives explicit target supervision that learns a task from near-zero accuracy, but its off-policy targets move the model far enough to cause forgetting; on-policy methods such as RLVR and self-distillation preserve policy proximity yet supply little signal when the policy cannot yet solve the task. We introduce ReDraft (Reference-Driven Revision and Fine-Tuning), which obtains both from the model's own failures: using an expert response only as a reference, it has the model revise its own incorrect rollout, keeps the revision only if a verifier accepts it, and fine-tunes on what survives. Each retained target is therefore explicit, yet still close to the current policy. Across Counting, Clock Reading, and Jigsaw on Qwen2.5-VL-3B/7B, two of them with near zero accuracy, ReDraft gains 56.9 points on the target task against SFT's 52.9 while cutting prior-task loss from 16.6 to 1.5 points (11.3x less forgetting), and improves on OPSD along both axes (19.3 gain, 6.2 loss). Data- and parameter-space analyses match the design: revised targets are more probable under the base model, and the updates they induce stay compact and follow SFT's direction more closely than OPSD's. Repairing the model's own output, rather than replacing it with an expert's, is what lets one objective do both.

1 INTRODUCTION

Continual post-training must acquire capabilities while preserving prior ones, but explicit supervision and policy proximity are difficult to obtain together. ReDraft uses self-revised, verifier-approved targets to combine both, outperforming SFT and OPSD across acquisition and retention.

  • 1 INTRODUCTION: Continual post-training requires explicit supervision for cold-start learning while keeping updates close to the current policy.Effective acquisition with little base competence needs informative targets, whereas retention favors policy-proximal updates.
  • 1 INTRODUCTION: SFT supplies explicit expert targets but can induce forgetting, whereas RLVR preserves policy proximity but provides little signal when sampled responses are almost always incorrect.These methods emphasize opposite sides of the acquisition–retention trade-off.
  • 1 INTRODUCTION: Self-distillation methods provide implicit token-level corrections that may emphasize stylistic or pivot tokens rather than task-determining decisions.Per-token divergence can be dense but misaligned with the errors that determine task success.
  • 1 INTRODUCTION: ReDraft samples the current policy, revises incorrect responses using an expert reference, verifies revisions, and fine-tunes only on successful targets.The complete revised response externalizes correction while remaining close to the policy that produced the initial rollout.
  • 1 INTRODUCTION: ReDraft exceeds SFT’s target-task gain while causing 11.3× less forgetting, and outperforms OPSD on both target acquisition and prior-task retention.Across three tasks and two model scales, ReDraft matches or exceeds SFT-level acquisition while consistently outperforming OPSD along both axes.
  • 1 INTRODUCTION: ReDraft’s revised targets remain closer to the base model, while induced parameter updates stay compact and follow SFT’s direction more closely than OPSD’s.These data- and parameter-space properties align with the method’s combination of explicit supervision and policy proximity.

2 RELATED WORK

Related work spans off-policy supervision, on-policy self-distillation, continual-learning remedies, and self-training with revision or filtering. ReDraft combines these lines through self-revised, verifier-approved targets without replay or frozen task-specific parameters.

  • Off-policy and On-policy Post-training: SFT uses fixed curated demonstrations, while related objectives clip or reweight updates against a reference policy to limit drift.These approaches address explicit imitation targets and policy stability through different objective designs.
  • On-policy Self-distillation: On-policy distillation evaluates teacher guidance along student-sampled trajectories, and context distillation strengthens the model through privileged prompts or demonstrations.This line reduces train–test mismatch while retaining teacher-conditioned guidance.
  • Catastrophic Forgetting: Continual-learning methods regularize important parameters, replay prior examples, or allocate task-specific parameters to mitigate catastrophic forgetting.Such remedies are difficult for foundation-model post-training when pre-training data are unavailable.
  • 2 RELATED WORK: ReDraft reduces forgetting without replay, frozen parameters, or waiting for rare successful rollouts.This distinguishes its self-revised-target approach from common continual-learning remedies and rejection-based self-training.
  • Self-training and Expert-revision: Self-training and self-refinement methods generate, filter, or revise model outputs using correctness, reward, or model and environment feedback.ReDraft belongs to this family while differing in how it constructs training targets from the model’s own generations.

3 BACKGROUND AND METHOD

The method frames continual post-training as maximizing target correctness while limiting policy change. ReDraft operationalizes this by revising failed rollouts, verifying complete responses, and fine-tuning on retained targets that combine explicit correctness with policy proximity.

  • 3 BACKGROUND AND METHOD: The problem assumes a pretrained policy with near-zero target-task accuracy, expert demonstrations, and a verifier that judges complete responses.Training begins from π0 and updates the current policy πθ.
  • 3 BACKGROUND AND METHOD: Continual post-training is evaluated by target-task gain and forgetting on a prior-capability benchmark suite.Accuracy on the corresponding evaluation sets supplies both axes.
  • 3 BACKGROUND AND METHOD: SFT directly imitates expert trajectories, while OPSD distils a reference-conditioned teacher into the unconditioned policy along the student’s rollout.OPSD’s forward-KL formulation takes gradients through the student branch only.
  • 3 BACKGROUND AND METHOD: The ideal target is the closest distribution to the current policy subject to verifier correctness, whereas SFT discards policy weighting and self-distillation lacks an explicit correctness indicator.ReDraft approximates this target with complete verified responses; its candidate acceptance rate is not an optimal repair rate.
  • 3 BACKGROUND AND METHOD: OPSD’s token-level KL can overemphasize stylistic disagreement rather than task-critical errors, and clipping limits contributions without identifying corrected continuations.The reported stylistic-token KL contribution is 6–15× larger than the mathematical-token contribution.
  • 3 BACKGROUND AND METHOD: ReDraft samples a rollout, keeps it if verified, otherwise revises it with the expert response as reference, then trains on the retained response from the prompt alone.Edited spans provide explicit supervision, while unchanged spans preserve the model’s trajectory and encourage policy proximity.
  • 3 BACKGROUND AND METHOD: ReDraft combines SFT-like explicit natural-language targets and cross-entropy supervision with OPSD-like student-generated trajectories, uniquely supplying all four desired properties.This design predicts SFT-like acquisition with the retention benefit of on-policy data.

4 EXPERIMENTAL SETUP

Experiments evaluate ReDraft on three vision-language target tasks using Qwen2.5-VL-3B and 7B, including two cold-start tasks, while measuring retention across prior-capability benchmarks. The setup compares supervision methods on target acquisition and retained capability.

  • 4 EXPERIMENTAL SETUP: Table 1 compares which methods provide explicit targets, cross-entropy supervision, student-generated trajectories, and policy proximity.ReDraft is described as the only method supplying all four properties.
  • 4 EXPERIMENTAL SETUP: Experiments use Counting, Jigsaw, and Analog Clock Reading on Qwen2.5-VL-3B and 7B, with Jigsaw and Clock Reading serving as near-zero-accuracy cold-start settings.Counting instead measures refinement of an existing skill because the base model is partly competent.
  • 4 EXPERIMENTAL SETUP: Target datasets contain 2k Counting images, 10k Clock Reading images, and 10k shuffled COCO images for Jigsaw, each paired with an expert response.The expert responses are generated by GPT-5.5.
  • 4 EXPERIMENTAL SETUP: Retention is measured across representative prior-capability axes including OCR, chart and document understanding, and general visual question answering.The evaluation includes AI2D, DocVQA, InfoVQA, ChartQA, MME, and MMStar.
  • 4 EXPERIMENTAL SETUP: Figure 4 plots target-task acquisition against prior-capability retention for Qwen2.5-VL-3B and 7B, with arrows starting at the base model and higher values preferred on both axes.Reject Sampling appears only for Counting, where the base policy already has substantial competence.

5 RESULTS AND ANALYSIS

ReDraft preserves SFT-level or better target acquisition while substantially reducing forgetting, including in cold-start, mixed, and sequential training. Compared with OPSD, extending the optimization budget does not close the acquisition gap, while ReDraft retains stronger performance across tasks and scales.

  • 5.1 MAIN RESULTS: 56.9 versus 52.9 points of mean target-task gain, while prior-task loss falls from 16.6 to 1.5 points, or 11.3× less forgetting than SFT.ReDraft leads SFT on target gain in most settings and forgets less in every comparison.
  • 5.1 MAIN RESULTS: 79.4 versus 73.8 versus 26.0 target-accuracy points in cold-start tasks for ReDraft, SFT, and OPSD, with prior-task losses of 2.2, 7.8, and 4.1 points.At 3B, OPSD gains only 2.4 points on Jigsaw versus ReDraft’s 77.5, and 43.1 versus 84.9 on Clock Reading.
  • 5.1 MAIN RESULTS: GRPO and rejection sampling remain limited by scarce correct rollouts, whereas ReDraft converts failed attempts into positive training targets.On Counting, rejection sampling gains only 1.0/1.3 points for 3B/7B, versus ReDraft’s 12.8/11.4.
  • 5.2 IS OPSD SIMPLY UNDER-TRAINED?: 70% more OPSD steps increase gain only from 19.3 to 23.4 points, while ReDraft reaches 56.9 and remains 31.7 points ahead on Clock Reading.The longer budget changes little across most task-scale cells, consistent with an objective-limited plateau.
  • 5.3 MULTI-TASK TRAINING: In mixed training, ReDraft reaches 62.9 versus SFT’s 59.7 on 3B target accuracy and retains 75.3 versus 63.3 on original tasks, while the retention advantage also holds on 7B.On 7B, ReDraft ends just below SFT in target accuracy, 68.6 versus 69.5, but retains 78.4 versus 31.6.
  • 5.3 MULTI-TASK TRAINING: In sequential Clock→Counting→Jigsaw training, ReDraft reaches three-task means of 53.4 versus 51.0 on 3B and 66.3 versus 60.9 on 7B, with stronger retention at both scales.Retention is 74.5 versus 63.1 on 3B and 75.9 versus 33.9 on 7B; SFT leads only on final-stage 3B Jigsaw.

6 DATA- AND PARAMETER-SPACE ANALYSIS

Data- and parameter-space analyses place ReDraft between expert SFT and on-policy OPSD: its targets remain policy-proximal, while its updates are compact and comparatively aligned with SFT. OPSD’s token-level credit often misses the answer tokens that determine task correctness.

  • DATA SPACE: Self-revision median PPL is 1.62 versus 5.33 on 3B and 1.47 versus 5.90 on 7B for expert trajectories, indicating greater policy proximity.Own rollouts are lowest at 1.30 on both scales; low PPL indicates likelihood under the model, not correctness.
  • PARAMETER SPACE: ReDraft’s update magnitude stays between SFT and OPSD throughout language-model depth, while vision-encoder updates nearly coincide.The profile is measured using relative Frobenius displacement from the pretrained weights and averaged across tasks and scales.
  • PARAMETER SPACE: ReDraft’s effective rank is 1454.7, intermediate between OPSD’s 1506.4 and SFT’s 1397.6, indicating a larger update packed into fewer directions than OPSD.The measurement covers language-model attention and MLP matrices.
  • DATA SPACE: OPSD credit is concentrated on incidental reasoning, style, and formatting tokens rather than the answer digits that decide the task, with some answer tokens pushed incorrectly.ReDraft instead writes the correction as a verified target.

7 CONCLUSION

ReDraft addresses continual post-training on tasks the model cannot yet perform by making corrections explicit while remaining close to the model’s own distribution. Across tasks, scales, and curricula, it recovers SFT-like acquisition with much less forgetting and improves on OPSD along both axes.

  • 7 CONCLUSION: ReDraft edits incorrect rollouts using an expert response as reference, verifies the revisions, and trains only on successful self-revisions.This combines explicit token-level supervision with on-policy data.
  • 7 CONCLUSION: Across three tasks, two model scales, and mixed and sequential curricula, ReDraft recovers SFT’s acquisition at a fraction of its forgetting and improves on OPSD along both axes.The conclusion attributes the approach’s scope to repairing the model’s own output rather than replacing it with an expert response.

LIMITATION

ReDraft’s theoretical target is conditioned on verifier acceptance, preserving the current policy’s relative probabilities among correct responses while discarding incorrect ones. The approach remains limited by finite-β softness and requires nonzero initial acceptance probability.

  • Theoretical limitation: The appendix derives the tilted-distribution solution, its binary-verifier hard-constraint limit, and the associated sampling-cost comparison.It states that rejection sampling costs 1/Zθ(x) samples, whereas revision costs 1/Sθ(x).
  • Theoretical limitation: The trust-region solution uses an exponential tilt that gives accepted responses a factor e1/β relative to rejected responses.β weights the proximity penalty in the objective, but appears in the solution through the acceptance advantage e1/β.
  • Theoretical limitation: At finite β, correctness remains a soft preference because the optimum still assigns probability to incorrect responses.Larger β keeps the solution closer to πθ, while smaller β increases the advantage of accepted responses and suppresses rejected mass.
  • Theoretical limitation: The hard correctness limit is valid only when the initial policy has nonzero acceptance probability Zθ(x) > 0.As β approaches zero, accepted responses retain mass and rejected responses vanish, provided an accepted rollout exists with positive probability.
  • Theoretical limitation: ReDraft’s limiting target keeps the current policy’s relative probabilities over accepted responses and removes rejected responses.This target is not uniform over correct responses; it remains weighted by πθ(y | x), preserving policy proximity.

A.4 WHERE EACH OBJECTIVE SITS

The objectives differ by whether they preserve the current policy proposal or enforce verified correctness, exposing distinct cold-start and forgetting trade-offs. ReDraft retains verifier filtering while replacing direct sampling with model-generated revisions, making accepted targets available even when initial correct rollouts are absent.

  • Rejection Sampling and RLVR: Rejection sampling and RLVR retain the current-policy proposal and keep the first verifier-accepted rollout, so their target supply depends on successful rollouts.Their expected sampling cost is 1/Zθ(x), and no target is produced when Zθ(x)=0.
  • SFT: SFT fits a point mass on the expert response, dropping current-policy weighting; this supports cold-start learning but can produce updates large enough to cause forgetting.The expert response receives probability 1 under SFT, versus πθ(y⋆|x)/Zθ(x) under the policy-proximal target, with the discrepancy growing in cold-start regimes.
  • Self-distillation: Self-distillation keeps the current-policy proposal but replaces the hard verifier indicator with an unconstrained soft divergence, so correctness is not guaranteed.Its per-token correction may concentrate on stylistic or pivot tokens rather than decisions that determine verifier acceptance.
  • ReDraft: ReDraft keeps verifier-based correctness while replacing the proposal with a rollout followed by model revision, and trains only on the resulting candidate.The model leaves already accepted rollouts unchanged, revises rejected ones using the expert response as a reference, and retains only verifier-approved revisions.
  • ReDraft: At cold start, revision can yield accepted targets when direct sampling yields none because revised responses have support unavailable to the current policy.The revision acceptance rate can remain positive even when Zθ(x)=0, since the reference-conditioned revision distribution differs from πθ.
  • ReDraft: 92.27% and 56.85% of 3B Jigsaw and Clock Reading prompts produce targets after revision, versus 0.21% and 0.03% solved by unaided first rollouts.On Counting, revision raises target yield from 65.85% to 92.00%.

B.4 EXTENDED OPSD TRAINING

The extended OPSD budget produces little additional acquisition in most task-scale settings. The main exception is 7B Clock Reading, where ReDraft remains substantially ahead on the same run.

  • Extended budget: +17.0 points is the largest OPSD gain from the longer budget, occurring on 7B Clock Reading as performance rises from +44.4 to +61.4.Most other cells change little despite 50–100% more updates, while Counting remains below 9 points at either scale.
  • Extended budget: ReDraft reaches +83.5 on the same 7B Clock Reading run, ending 22.1 points above OPSD.This comparison identifies the principal gap between the longer-budget OPSD result and ReDraft.
  • Evaluation scope: Table 5 reports per-task, per-scale OPSD gains, with cumulative steps and base accuracies specified for Counting, Jigsaw, and Clock Reading.Column-wise averages across the two scales reproduce Table 2.

C PER-TASK ANSWER-PERPLEXITY DISTRIBUTIONS

ReDraft produces verified-correct targets that are closer to the model’s own response distribution than expert targets across tasks and scales, though Jigsaw remains harder at 3B.

  • Revised targets have lower answer-only PPL than expert targets in every task and scale, approaching the model’s own-rollout PPL on Counting and Clock Reading.The revised medians are 1.59, 1.44, 1.32, and 1.41 versus expert-target values between 4.19 and 6.27.
  • Jigsaw shows a scale-dependent proximity gap: revision reaches a 2.75 median at 7B versus 2.06 for own rollouts, but remains at 4.00 versus 1.28 at 3B.Thus, verified correctness does not guarantee equally close alignment with the model distribution.
  • Low PPL for incorrect rollouts separates policy proximity from correctness, making verified correctness an essential part of ReDraft’s target construction.

D PER-SETTING WEIGHT-SPACE RESULTS

Across tasks and scales, ReDraft induces intermediate update magnitude and concentration between SFT and OPSD while aligning more closely with SFT’s update direction.

  • The weight-space summary reports means over task–model pairs, while detailed tables decompose displacement by module group and task–scale setting.
  • SFT > ReDraft > OPSD for total displacement, LM-linear drift, and output-head drift across every task and model scale.The gap is smallest on Jigsaw and larger on Counting and Clock Reading.
  • All three concentration measures rank SFT as most concentrated and ReDraft as more concentrated than OPSD.The measures describe update shape independently of update size and use singular-value spectra of language-model Attention and MLP matrices.
  • ReDraft’s full-parameter update aligns more closely with SFT than OPSD does in every task–scale pair, with vocabulary cosines of 0.644–0.961 versus 0.314–0.492.LM-linear layers favor ReDraft in most settings, except the two Clock Reading settings.
  • A Clock Reading revision changes an incorrect 06:08:30 rollout into the verified 03:30:38 answer with minimal edits.

E.2 COUNTING

The Counting demonstration shows ReDraft minimally repairing an incorrect five-person rollout into a verified six-person answer while preserving the response structure and enumeration.

  • E.2 COUNTING: ReDraft corrects the counting rollout from 5 to 6 by adding the missing person rather than changing only the final number.The revised response explicitly enumerates six individuals and ends with the verified count 6.
  • E.2 COUNTING: The task asks the model to count people in an image and provide a step-by-step answer ending with a boxed total.
  • E.2 COUNTING: The broader revision procedure uses an incorrect rollout, an expert reference, and a verifier-accepted revised response as the training target.
  • E.2 COUNTING: The counting revision template requires the model to repair the enumeration and keep the list and total consistent.Changing only the final number while leaving the incorrect list untouched is invalid.
  • E.2 COUNTING: The template instructs the model to preserve correct wording, order, and style while changing only what is wrong.
Loading 2609.16639v1…