PaperlayerView exact revision on arXiv

Source-linked AI summary

Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data

Takayuki Yamamoto, Daisuke Kawahara

arXiv:2609.16532v1cs.CL

TL;DR

CPT can store new knowledge without ensuring it is elicited correctly, and standard DPO may push down correct model-generated answers when they differ from gold answers only in style. SD-DPO scores factual agreement and inverts and weights style-only pairs to cancel their style signal. Across both benchmarks, it succeeds where objectives that push down rejected answers fail when correct rejected responses dominate.

  • Problem

    CPT can store knowledge without ensuring correct elicitation, and when knowledge is partially known, standard DPO may suppress correct rejected responses that differ from gold answers in style.

    Source: §1 · pp. 1–2

  • Method

    SD-DPO uses a judge LLM to distinguish factual errors from style-only pairs, then inverts and weights the latter to cancel style gradients.

    Source: §1 · p. 2, §3 · p. 4, §3.3 · p. 5

  • Results

    Across both benchmarks, SD-DPO succeeds where vanilla DPO, IPO, and KTO fail to beat baseline when correct rejected responses dominate; on QuALITY it reaches 59.17% after 50% preference-data usage (+2.96 over baseline).

    Source: §4.2 · p. 7, §5 · p. 9

  • Takeaways & Limitations

    A preference stage that respects what differs within each pair updates knowledge at a fraction of the CPT cost and answers with the fact holding at the queried period.

    Source: §5 · p. 9

    AToKE results use a single seed, all judging is by an LLM, and excessive budget can cause repetition at the length cap.

    Source: §5 · p. 9

Abstract

from arXiv · show

Continued pretraining (CPT) with data augmentation such as paraphrasing can store inside a large language model (LLM) the knowledge of a small source corpus. The stored knowledge, however, is not always retrieved correctly. We study the eliciting side rather than the storing side: we use preference optimization, which learns from pairs of a preferred (chosen) and a dispreferred (rejected) response, so that the model elicits its stored knowledge more accurately. One proposed approach takes the model's own erroneous response as rejected and the gold answer as chosen, so as to suppress the error. When the target knowledge is partially known, however, most of these rejected responses are factually correct. Using direct preference optimization (DPO) then pushes down rejected responses that contain correct knowledge and differ from the chosen answer only in style, such as length and wording. We propose style-debiased DPO (SD-DPO), which scores whether the rejected response of each pair is factually correct, inverts the preference of such pairs, and weights them so that the learning signal due to differences in style cancels out as a whole. We first test whether, on top of EntiGraph, a representative storing-side method that runs CPT on text synthesized from the corpus, our method adds accuracy efficiently. On QuALITY, the reading-comprehension QA benchmark on which EntiGraph was evaluated, SD-DPO exceeds a baseline we CPT on EntiGraph's synthetic data from the same base model and evaluate with the same procedure. The training tokens this requires are a few dozen times fewer than the additional CPT needed for the same gain. For knowledge updating, the main goal of this work, we use AToKE, a knowledge-editing benchmark for facts that change over time. There, SD-DPO reaches an overall accuracy of 0.982 and answers with the new or the old fact according to the queried period.

1 INTRODUCTION

CPT with augmented data can store new knowledge, but does not ensure the model can retrieve it correctly. SD-DPO addresses a self-preference failure mode in which standard DPO pushes down factually correct responses that differ from gold answers in style.

1 INTRODUCTION

  • Augmentation makes storing knowledge feasible, but accuracy levels off as additional synthetic training tokens teach less, leaving correct elicitation unresolved.EntiGraph’s scaling curve exposes diminishing accuracy gains as training proceeds.Source: §1 · p. 2
  • The paper separates knowledge acquisition by CPT from retrieval by preference optimization, using the gold answer as chosen and the CPT model’s sample as rejected.This self-preference recipe follows PASTA’s Self-DPO and Rozner et al. (2024).Source: §1 · p. 2
  • When target knowledge is partially known, most model-generated rejected responses are factually correct, so standard DPO can suppress correct knowledge along with stylistic differences.Chosen gold answers and rejected model samples can differ in format, verbosity, and phrasing as well as factual content.Source: §1 · p. 2
  • SD-DPO inverts preferences for factually correct rejected responses and weights them to cancel style gradients, retaining the factual learning signal.The judge LLM distinguishes factual errors from pairs that differ only in style.Source: §1 · p. 2
  • On QuALITY, SD-DPO exceeds an EntiGraph-setting CPT baseline, using only a few percent of the additional CPT tokens needed for the same accuracy gain.For AToKE, the paper reports high new-fact acquisition and old-fact recall simultaneously.Source: §1 · p. 3

2 RELATED WORK

Prior work studies synthetic-data CPT for storing knowledge, preference optimization for updating it, and remedies for style bias or temporal knowledge editing. The paper positions its focus at the intersection: preference-based retrieval when self-generated rejected answers are often factually correct.

2 RELATED WORK

  • EntiGraph and related synthetic-data CPT work improve knowledge storage, but the cited approaches do not add a preference stage to improve retrieval after CPT.EntiGraph reports QuALITY accuracy scaling with synthetic-token count; Cheng et al. (2024) convert raw text to reading-comprehension training format.Source: §2 · p. 3
  • PASTA and Rozner et al. (2024) apply preference optimization to knowledge updates, but their rejected responses are mostly factually wrong, unlike this paper’s partially known setting.Rozner et al. cast single-fact editing as DPO with the old fact rejected.Source: §2 · p. 3
  • Existing noise-robust DPO methods treat imperfect pairs as noise or low quality, rather than identifying what differs within each pair.The cited work also reports that format can mislead reward models and preference tuning can amplify style over content.Source: §2 · p. 3
  • Length remedies target a specific feature or add randomly labeled equal-utility ties; SD-DPO instead identifies style-only pairs and labels them to cancel the mean style signal.The paper also compares with SamPO under identical conditions because its pairs differ greatly in length.Source: §2 · p. 3
  • On AToKE, standard ROME or MEMIT preserves old-fact accuracy at only about 2%; METO raises it to 20–30%, below their reported 86–100% new-fact accuracy.The paper approaches temporal updating with CPT plus preference optimization and retains both facts.Source: §2 · p. 3

3 METHOD

The method first stores source-corpus knowledge through synthetic-data CPT, then uses SD-DPO to improve retrieval without pushing down correct rejected answers. A factual-agreement judge separates factual errors from style-only differences, whose inverted, weighted gradients cancel under the method’s assumptions.

3 METHOD

  • The two-stage method runs CPT on synthetic texts or paraphrases, then trains on gold-answer and CPT-model-response preference pairs to improve elicitation.Figure 1 illustrates style-gradient cancellation: vanilla DPO ends at (style 5, fact 2), while SD-DPO ends at (style 0, fact 2).Source: §3 · pp. 3–4, Figure 1 · p. 4

3.1 PROBLEM SETUP AND NOTATION

  • At initialization, DPO raises the chosen response’s likelihood and lowers the rejected response’s likelihood, an operation to avoid when the rejected answer is correct.The CPT model is the reference policy and initialization of the policy being trained; the preference data pairs each question with a gold answer and a sampled response.Source: §3.1 · p. 4

3.2 MEASURING FACTUAL AGREEMENT

  • The judge scores factual agreement from 0 to 1; pairwise scoring compares responses, while context-grounded scoring checks the rejected answer against the source and accounts for document grounding.Pairwise scoring is used for single-fact questions; context-grounded scoring is used when correctness depends on reasons or causes beyond the chosen answer.Source: §3.2 · p. 5

3.3 STYLE-DEBIASED DPO

  • Under additive margin decomposition and Assumption A2, SD-DPO keeps the usual direction for factual differences and inverts style-only pairs with weight w = p/(1 −p), canceling style terms at initialization.The cancellation guarantee is for the expected dataset gradient at initialization. The trained loss also uses judge-confidence weights, rescales group weights, and adds a knowledge-preservation hinge term; replay pairs use standard DPO and EOS is excluded from likelihood sums.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

The experiments test whether preference optimization improves retrieval after CPT on QuALITY and whether SD-DPO updates time-sensitive facts on AToKE. They also examine training-budget effects and retention of general capabilities.

4.1 SETUP

  • QuALITY uses 4,609 closed-book questions after CPT on 449M synthetic tokens; AToKE tests 8,819 time-stamped edits, with results also checked on two other model families.The AToKE procedure applies CPT before SD-DPO; its facts include old and new answers tied to the periods when each held.Source: §4.1 · p. 6, §4.3 · p. 9

4.2 QUALITY: EXCEEDING THE CPT BASELINE EFFICIENTLY

  • 59.17% QuALITY accuracy (+2.96 over baseline) uses 50% of preference data; the final checkpoint reaches 57.98% (+1.77).Vanilla DPO remains below baseline at every checkpoint, while fmax filtering without inversion stays near baseline.Source: §4.2 · p. 7, Table 1 · p. 7
  • The ablations attribute the gain to inversion rather than budget tuning: removing inversion drops SD-DPO from 59.17% to 55.82%, and vanilla DPO stays near baseline.Halving the learning rate shrinks the late-training drop to −1.19; SamPO reaches 57.32% (+1.11), 1.85 points below SD-DPO.Source: §4.2 · pp. 7–8, Table 2 · p. 7
  • About 409M additional CPT tokens would be needed to reach 59.17%, roughly forty times the preference data SD-DPO used.The comparison uses the EntiGraph tokens-to-accuracy curve; the paper notes that it is a single measured point and the trade-off reversal is unmapped.Source: §4.2 · p. 8, Figure 2 · p. 8

4.3 ATOKE: RETRIEVING THE FACT THAT HOLDS AT THE QUERIED PERIOD

  • 0.982 Overall on AToKE; SD-DPO raises HES from 0.782 to 0.954, while vanilla DPO falls to 0.738, below CPT only.HES measures old-fact answers with an explicit period; the AToKE table compares the shared-data, shared-model rows directly.Source: §4.3 · pp. 8–9, Table 3 · p. 8

4.4 RETENTION OF GENERAL CAPABILITIES

  • The six-benchmark average changes by −0.071 from pre-CPT to final SD-DPO on QuALITY, with −0.063 in CPT and −0.008 in SD-DPO; AToKE changes by −0.013.All preference-optimization variants show similarly small general-capability drops, within measurement error (±0.5); replay contributes to stability, but its share is not isolated.Source: §4.4 · p. 9

5 DISCUSSION AND CONCLUSION

The discussion separates storing knowledge from eliciting it and argues that preference optimization should respond to factual differences rather than stylistic ones. The authors report strong benchmark results while noting limits in AToKE evaluation and late training.

5 DISCUSSION AND CONCLUSION

  • When factually correct rejected responses dominate, vanilla DPO, IPO, and KTO fail to beat the baseline, while style-cancelling inversion succeeds on both benchmarks.The contrast is consistent across tested base models of different families and sizes; the reported factual-error shares are cFA p≈0.33 and pFA p=0.148.Source: §5 · p. 9
  • AToKE results use a single seed and LLM judging; at the final checkpoint, 14% of responses reach the length cap and are less accurate.The authors identify suppressing this behavior without budget control as future work.Source: §5 · p. 9
  • The conclusion treats storing and eliciting as distinct optimization problems and reports updating at a fraction of CPT cost while answering for the queried period.This is the paper's stated conclusion across its knowledge-updating framing.Source: §5 · p. 9

A.1 STORAGE AND ACCESS

  • CPT raises storage but does not optimize emitting stored facts when asked; about 66% of CPT responses already agree with gold answers on the facts.The appendix frames accuracy as storage multiplied by access and says SD-DPO's improvement is consistent with improving the access factor.Source: §A.1 · p. 12

A.2 PREFERENCE OPTIMIZATION STAYS NEAR THE REFERENCE

  • The ideal preference optimizer stays near its reference policy, approximately preserving stored knowledge; learned training can overshoot when continued too long.The paper presents the near-zero general-benchmark degradation as corroboration, not proof, of the bound's prediction.Source: §A.2 · p. 12

A.3 DERIVATION OF PROPOSITION 1

At initialization, the dataset-averaged style-gradient contribution cancels under the stated shared-style assumption. When style differs by answer format, pooled cancellation can leave a small residual.

A.3 DERIVATION OF PROPOSITION 1

  • At initialization, weighting inverted and non-inverted pairs cancels the shared style term in the dataset-averaged gradient, leaving only factual components.The guarantee assumes A2 and applies to the dataset average; averaging 32 pairs per step reduces mini-batch noise, while budget control limits drift from the initial regime.Source: §A.3 · pp. 12–13

A.4 WHEN ASSUMPTION A2 BREAKS: FORMAT-SPECIFIC STYLE COMPONENTS

  • With MCQ and free-text styles separated, pooled weighting may leave a per-format residual; in standard data it is below 0.5% of one group's style gradient.The reported group ratios are p = 0.334 for MCQ and p = 0.336 for free text; an MCQ-only test is left to future work.Source: §A.4 · p. 13

A.5 RELATION TO TIE TRAINING AND SPURIOUS-CORRELATION LEARNING

SD-DPO treats factual content as causal and style as spurious, automatically identifying style-only ties and inverting them to cancel the initial style signal. Unlike random-label tie training, this cancellation depends on an assumption and does not remove correlation leakage.

A.5 RELATION TO TIE TRAINING AND SPURIOUS-CORRELATION LEARNING

  • SD-DPO identifies style-only pairs as ties automatically, rather than constructing them by hand, addressing the open tie-construction problem in Moya et al. (2026).Here, fact is the causal feature and style the spurious feature.Source: §A.5 · p. 13
  • Weighting inverted ties by w = p/(1−p) cancels the strict pairs’ style mean difference at initialization and adds curvature in style directions.This differs from random labeling, which removes the ties’ own mean difference but only shrinks the strict pairs’ style bias.Source: §A.5 · p. 13
  • When fact and style correlate across pairs, cancellation does not remove leakage; SD-DPO relies on budget control to stay local rather than guaranteeing equilibrium.The guarantee also depends on Assumption A2; random labeling is robust to A2 violations but leaves strict-pair bias.Source: §A.5 · p. 13

B IMPLEMENTATION DETAILS OF SD-DPO

Implementation choices calibrate factual-agreement scoring and preserve the intended cancellation while limiting side effects of preference inversion.

B IMPLEMENTATION DETAILS OF SD-DPO

  • The 0.5 threshold separates equivalent from non-equivalent answers; shifting it to 0.4 or 0.6 reclassifies at most 0.4% of pFA pairs and 7% of cFA pairs.The fmax filter uses the same threshold.Source: B IMPLEMENTATION DETAILS OF SD-DPO · pp. 13–14
  • The implemented loss adds normalized confidence weights, rescales group weights, preserves correct rejected responses with a hinge penalty, replays general-purpose preferences under standard DPO, and excludes EOS from likelihood sums.The ideal-form cancellation carries over to the implemented loss in expectation; the preservation term is excluded from the analysis because its max argument is zero at initialization.Source: B IMPLEMENTATION DETAILS OF SD-DPO · p. 14
  • On inverted pairs, the shared answer-ending template makes EOS likelihood fall on most data, causing termination collapse in preliminary runs; excluding EOS removes this preference gradient and the collapse.The exclusion is specific to preference inversion; vanilla DPO, IPO, and KTO do not invert and do not exhibit the phenomenon.Source: B IMPLEMENTATION DETAILS OF SD-DPO · p. 14

C FACTUAL-AGREEMENT SCORING

Factual-agreement scoring distinguishes wrong rejected answers from correct ones that differ in style, using source-grounded judgments for QuALITY. The section then describes how QuALITY preference pairs are constructed and illustrates why correct rejected answers are inverted.

C FACTUAL-AGREEMENT SCORING

  • pFA compares factual content while ignoring format and verbosity; cFA additionally judges accuracy and nuance against the source article.Documents longer than 15,000 characters are truncated with a note appended.Source: C FACTUAL-AGREEMENT SCORING · pp. 14–15
  • For cFA only, a source-wording match factor discounts the judge score when the rejected answer is less grounded than the chosen answer, limiting high scores for generic prose.The factor compares matched-word fractions in the two responses.Source: C FACTUAL-AGREEMENT SCORING · p. 15
  • Table 4’s bottom-row p is the share of genuinely wrong rejected answers, while 1−p is the inverted group and p also gives the post-filter fraction.The table covers pFA and cFA score distributions on 26,291 MCQ pairs; the free-text side behaves similarly.Source: Table 4 · p. 15
  • About 36% of pFA pairs judged nearly equivalent fall below 0.7 under cFA, which exposes differences in nuance, causality, and character description.The authors use cFA for QuALITY because of this finer judgment; AToKE uses pFA with p = 0.148, which still supports cancellation.Source: C FACTUAL-AGREEMENT SCORING · p. 15

D.1 QUALITY PREFERENCE DATA

  • QuALITY questions are generated from A–B–C entity chains, with two relation texts sharing bridge entity B, so answering requires combining both relations.The generator samples up to 100 chains per article and creates one four-way question per chain.Source: §D.1 · p. 15
  • 52,582 preference pairs use 21.7M tokens, from 26,291 questions rendered in MCQ and free-text formats with sampled CPT-model rejections.Chosen responses share the reasoning across formats; rejected responses are sampled at temperature 0.7 and top-p 0.9.Source: §D.1 · pp. 15–16
  • In the example, cFA scores of 1.0 and 0.9 mark both longer rejected answers as factually correct, so SD-DPO inverts both pairs.The MCQ rejected answer is 116 words against a 76-word chosen answer; the free-text rejected answer is 253 words.Source: §D.1 · p. 16

D.2 ATOKE SYNTHETIC DATA: A WORKED EXAMPLE

The AToKE data construction covers old and new facts with explicit and implicit time, and evaluation tests whether answers match the queried period. The reported overlap check has little effect on overall accuracy, while cross-paper numerical comparisons are limited by protocol differences.

D.2 ATOKE SYNTHETIC DATA: A WORKED EXAMPLE

  • Each AToKE case supplies old and new facts with validity periods; four seed sentences cover explicit and implicit time, and each is paraphrased 10 times to generate 44 QA per case.Across 8,819 cases, the CPT corpus contains 775,311 texts and 19.7M tokens; preference data contains 387,287 pairs.Source: §D.2 · p. 16
  • Evaluation uses 35,276 template-generated questions, free-generation answers, and a judge that accepts the expected answer or an alias unless the response contradicts the queried period.Questions cover four combinations: current or historical fact, each with explicit or implicit time.Source: §D.2 · p. 16
  • 6.1% of SD-DPO responses judged correct mention both periods; 11 of 8,689 explicit-period questions (0.13%) were judged correct despite assigning another period’s fact to the queried period.A correctly period-attributed fact from another period is not counted as a contradiction.Source: §D.2 · p. 16
  • Numerical comparisons with prior-work ROME and MEMIT rows are for reference only because their evaluation protocols differ.Source: §D.2 · p. 16
  • 1,228 of 35,276 evaluation questions (3.48%) overlap verbatim with training questions; removing them changes overall accuracy by at most 0.001 (CPT 0.884, SD-DPO 0.981).All reported numbers use the full evaluation set.Source: §D.2 · p. 17

E HYPERPARAMETERS AND PROTOCOLS

The QuALITY and AToKE experiments use specified, mostly shared training settings and a common evaluation pipeline. QuALITY evaluation uses closed-book questions from the source articles, with protocol choices intended to support comparisons against the CPT baseline.

E HYPERPARAMETERS AND PROTOCOLS

  • All training is full-parameter under FSDP on four H100 GPUs; QuALITY uses sequence lengths 1024/512, seed 42, and replay from HH-RLHF helpful-base.DPO variants use a constant scheduler without warmup, while CPT uses cosine scheduling with a 0.05 warmup ratio; AToKE uses different CPT and SD-DPO settings.Source: Table 5 · p. 17
  • 4,609 questions about 265 QuALITY train and dev articles form the closed-book evaluation set; the official test set is excluded because its labels are hidden.The evaluated articles are also the CPT source corpus, so the protocol tests whether their knowledge was stored and can be elicited.Source: E HYPERPARAMETERS AND PROTOCOLS · p. 17
  • QuALITY uses 0-shot chat prompting and eight samples per question, with accuracy based on one uniformly drawn parseable answer and parseable-generation rate tracked separately.Unfiltered vanilla DPO runs have 59–84% valid-answer rates, so some accuracy loss reflects broken response formatting.Source: E HYPERPARAMETERS AND PROTOCOLS · p. 17
  • Under constant learning rate, no warmup, and one epoch, checkpoint progress ratio r approximates training on fraction r of the data.At r=0.25, the accuracy difference was 0.0087, comparable to rerun-to-rerun noise of about 0.01.Source: E HYPERPARAMETERS AND PROTOCOLS · p. 17
  • The EntiGraph paper’s 56.22% result is not directly comparable to the reproduction because its prompting and parse-success denominator differ from the paper-consistent protocol.The reproduction from the base model is about 47% under that protocol, and its replay corpus substitutes for RedPajama Books, which is no longer distributed.Source: E HYPERPARAMETERS AND PROTOCOLS · p. 17
  • The shared evaluation pipeline uses Llama-3-8B-Instruct, 0-shot chat, eight samples, random-valid scoring, and closed-book evaluation for both method and baseline.The reproduced CPT result is 56.21%, agreeing with EntiGraph’s 56.1% curve prediction at 449M tokens, though across evaluation protocols and only for token-equivalence analysis.Source: E HYPERPARAMETERS AND PROTOCOLS · pp. 17–18

F.1 ATOKE ACROSS BASE MODELS

Across three base models, SD-DPO exceeds both CPT-only and vanilla DPO, while vanilla DPO falls below CPT-only. The preference-stage capability changes remain small, though individual AToKE metrics can trade gains for losses and QuALITY outputs can become excessively long.

F.1 ATOKE ACROSS BASE MODELS

  • Across all three tested models, SD-DPO exceeds both CPT-only and vanilla DPO, while vanilla DPO falls below CPT-only.SD-DPO gains are Gemma4 +0.232, Llama +0.097, and Qwen3 +0.046; the vanilla DPO drop is largest for Qwen3 (p = 0.132: −0.359) and smallest for Gemma4 (p = 0.734: −0.011).Source: §F.1 · p. 18, Table 7 · p. 18
  • 0.901 to 0.870: only CES on Qwen3 falls below CPT-only, as 1,001 previously correct questions are lost and 725 missed questions are gained.Among lost responses, 71.8% name another edit case’s answer and 5.0% name the same case’s old fact; CRS, HES, and HRS instead gain +781, +603, and +512 questions, respectively.Source: §F.1 · p. 18
  • Rejected QuALITY responses are longer than chosen responses in 98–99.5% of pairs, but the factual-difference and style-only groups have similar lengths.The groups average 146 versus 134 words on QuALITY MCQ; their lengths are also similar across the listed AToKE models, so the split is not a length proxy.Source: §F.1 · p. 19, Table 12 · p. 21
  • At the final QuALITY checkpoint, 14.0% of generations hit the length cap; these retain answer format less often and are correct less often than normal-length generations.On AToKE, 18% exceed 1,000 words and are still judged correct at 0.988, while 95% of sentences in sampled long answers are repetitions.Source: §F.1 · pp. 19–20
  • At the best QuALITY checkpoint, SD-DPO answers 443 questions correctly that both baselines answer incorrectly, versus 303 in the opposite direction.Its longer reasoning can expose stored knowledge, but when wrong it typically narrates a coherent but incorrect story; SD-DPO outputs reach 200 words at the best point and 382 at the final checkpoint.Source: §F.1 · pp. 19–20
  • Across unfiltered runs, average six-benchmark capability finishes 1.36–1.63 points below baseline despite QuALITY accuracy ranging from 47.01% to 57.98%.The only run with a smaller average-capability drop (−0.21), vanilla at 5×10−7, barely moves on QuALITY; HellaSwag and Winogrande remain within ±0.02 in every condition.Source: §F.1 · p. 20
  • 59.17/57.98: SD-DPO’s best/final QuALITY accuracy exceeds vanilla DPO’s 56.01/54.64 under identical conditions; SamPO alone among the baselines exceeds the CPT baseline.IPO reaches 55.68/45.98 and KTO 56.05/54.47; SamPO reaches 57.32/57.32.Source: §F.1 · p. 20
  • The preference-stage general-capability change is +0.002 on Qwen3 and −0.021 on Gemma4, comparable to −0.013 for the Llama AToKE run.The paper reports that raising AToKE accuracy substantially while leaving general capability nearly intact does not depend on the base model.Source: §F.1 · p. 22, Table 13 · p. 22

G LIMITATIONS

The evaluation covers a limited set of model families and benchmarks, with several training and evaluation choices left untested. The reported results also have limitations involving generation length, gradient assumptions, and the scope of efficiency comparisons.

G LIMITATIONS

  • The experiments cover three model families on AToKE and one on QuALITY; models above 8B parameters and other architectures are untested, and domain and format diversity is limited.The AToKE experiments use three families, while QuALITY uses one by design; the authors identify broader robustness testing as relevant to their main goal.Source: G LIMITATIONS · p. 22
  • True iterative DPO, broader hyperparameter search, and the effect of EOS exclusion on baselines are untested or unmeasured.Sequential training on four data splits used a different loss; the EOS exclusion is applied only to SD-DPO.Source: G LIMITATIONS · p. 22
  • Judge scores and AToKE correctness lack human verification, and the AToKE experiments use a single random seed, 42.The authors also report that 3.48% of AToKE evaluation questions overlap verbatim with training questions, though removing them changes accuracy by no more than 0.001.Source: G LIMITATIONS · p. 22
  • Prior-work accuracy comparisons on AToKE are for reference only because the base model and evaluation protocol differ.The original paper does not publish how its accuracies are computed, while this work judges free generations with an LLM.Source: G LIMITATIONS · p. 22
  • General-capability retention partly reflects a 0.5 replay ratio, so the separate contributions of replay and the KL constraint are not identified.Data-side regularization such as generative replay is not compared as an alternative to the KL constraint.Source: G LIMITATIONS · p. 23
  • 14% of generations at the final checkpoint repeat to the length cap and are less accurate than normal generations.SD-DPO also lengthens outputs substantially and leaves some runaway generations; length-aware objectives and stopping control are untested.Source: G LIMITATIONS · p. 23
  • SD-DPO removes the mean style gradient but does not address leakage through correlation between factual and style features, and relies on staying near initialization rather than an equilibrium guarantee.Assumption A2 is also violated by variation in rejected-response length, because the style component scales with token count.Source: G LIMITATIONS · p. 23
  • The CPT-versus-elicitation efficiency comparison rests on one scaling-curve point, at 449M tokens.Improving CPT itself through better synthetic-data generation is outside the paper's scope.Source: G LIMITATIONS · p. 23

Related papers

Loading 2609.16532v1…