FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Models

1. Introduction

• Pruned models show more severe text degeneration than unpruned models in both comparison cases, despite fine-tuning to recover knowledge.The comparison generates 200 tokens from a 50-token WikiText-103 prefix and classifies text as repetitive when repeated segments make up most of the generation.Source: Table 1 · p. 1, §1 · pp. 1–2
• Prior repetition-mitigation approaches lower probabilities for previously generated tokens, but do not guide coherent continuation and can degrade perplexity (Holtzman et al., 2020; Welleck et al., 2019).The paper motivates token-level guidance that also promotes plausible alternatives.Source: §1 · p. 2
• Repetition-loop degeneration has distinct entry risk and persistence, with persistence dominating long-horizon repetition as consecutive in-loop transitions compound.The analysis finds that replacing only a few high-probability tokens at loop onset can steer generation away from repetition; pruning and naive distillation can raise both entry risk and persistence.Source: §1 · p. 2
• FOCUS reweights distillation toward high-confidence teacher regions, while RePAIR uses onset-centered positive and negative continuations with a margin loss to promote plausible alternatives.Together, the objectives target leakage that raises loop entry risk and early commitment that raises persistence.Source: §1 · p. 2
• FOCUS and RePAIR consistently reduce repetition and improve distributional and semantic quality across open-ended continuation and instruction-based generation, with only a small perplexity increase.FOCUS also improves generation quality when combined with existing training-based mitigation objectives.Source: §1 · p. 2

2.1. LLM Pruning & Distillation

• Post-pruning knowledge distillation transfers teacher knowledge to pruned students, but naive distillation can be ineffective or transfer incorrect answers.Studies attribute limitations to teacher biases and the teacher–student capacity gap, motivating selective transfer of useful information.Source: §2.1 · p. 2

2.2. Mitigation of Text Degeneration

• Decoding-time sampling and training-time objectives offer complementary approaches to mitigating text degeneration.Top-k and top-p sampling restrict generation to high-probability tokens (Holtzman et al., 2020); training approaches include unlikelihood (Welleck et al., 2019), ScaleGrad (Lin et al., 2021), and synthetic repetition data in DITTO, though unlikelihood can degrade perplexity.Source: §2.2 · p. 3

3.1. Coverage-based Repetition Metrics and Onset Sensitivity

• CREP measures sequence-level degeneration by the dominant N-gram's token coverage, treating repetition as a dominance event over a small set of recurring local contexts.Coverage counts the fraction of tokens explained by an N-gram's occurrences, using exact token matches or task-specific evaluation equivalence.Source: §3.1 · p. 3
• 5.9% to 0.7% under sampling and 26.6% to 10.8% under greedy decoding: replacing only two onset tokens reduced CREP in 1,000 generated sequences.The replacements came from top-2 candidates at the same prefix, keeping the continuation plausible while shifting it away from a repetitive basin.Source: Table 2 · p. 3, §3.1 · p. 3

3.2. Decoding Dynamics: Entry Risk and Persistence

• Degeneration arises when decoding enters a recurrent context region and persists there; its upper bound combines horizon-T entry risk with persistence raised to the required run length.Because persistence is exponentiated by the minimum in-loop run length, modestly reducing it can suppress long repetition runs even if entry is not eliminated.Source: §3.2 · pp. 3–4

3.3. Nucleus Alternatives and Escape Mass

• Under nucleus sampling, reducing loop persistence requires increasing escape mass among sampleable nucleus tokens, not merely increasing entropy across the full vocabulary.Persistence depends on how the nucleus mass is divided between tokens that continue the loop and tokens that exit it.Source: §3.3 · p. 4

3.4. How Pruning and Naive Distillation Reshape Risks

• Pruning and naive distillation can raise both loop entry risk and persistence by leaking mass to teacher-suppressed tokens and collapsing teacher-supported alternatives at loop-sensitive contexts.The resulting supervision targets are to suppress leakage that can enter the nucleus and preserve plausible onset-adjacent alternatives that increase nucleus-level escape mass.Source: §3.4 · pp. 4–5

4.1. FOCUS: FOcus on Confident Token Under Teacher Supervision

• FOCUS emphasizes tokens where the teacher is highly confident and preserves the standard knowledge-distillation form with a reweighted teacher distribution.The reweighted objective is described as easy to optimize.Source: §4.1 · p. 5

4.2. RePAIR: Repetition-aware PAIRwise Alignment

• RePAIR provides token-level corrective signals where degeneration begins, guiding the student toward non-repetitive, contextually coherent generation.The passage notes that runtime guidance is not possible because detecting repetition-loop onset requires future tokens.Source: §4.2 · p. 5

4.2.1. PAIRWISE DATA COLLECTION

• RePAIR data collection labels outputs above a repetition threshold as negatives, then regenerates from the prefix before the first repetition to obtain positives.Negative samples come from the pruned model; positive samples come from the unpruned model using top-p decoding, with the threshold set to 0.3.Source: §4.2.1 · p. 5

4.2.2. REPAIR TRAINING OBJECTIVE

• The training objective uses a token-level pairwise margin loss to make positive continuations more likely than negative ones.Unless otherwise specified, α1 is fixed at 0.05 and α2 at 1.Source: §4.2.2 · p. 5, §4.2.3 · p. 6

5.1. Setup

• Experiments prune 25% of Llama model width with LLMPruner, then use two-epoch LoRA fine-tuning on Alpaca with self-distillation; evaluation also considers 35% and 45% pruning.The tasks use 1,000 WikiText-103 instances with 50-token prefixes and 100 generated tokens, or 1,000 Self-Instruct prompts, with top-p sampling at p = 0.9.Source: §5.1 · p. 6

5.2. Open-ended Generation Task

• On WikiText-103, RePAIR alone achieves the highest MAUVE score of 0.68 and lowest CREP score of 2.23; adding FOCUS improves MAUVE by about 0.06 on average across methods.With FOCUS, methods improve all metrics with only a slight increase in perplexity; RePAIR also has unique n-gram distributions close to real WikiText-103 text.Source: §5.2 · pp. 6–7, Table 3 · p. 7

5.3. Instruction-based Generation Task

• With FOCUS, most instruction-based metrics achieve the best results; RePAIR records CREP 0.63 and EAD1 0.32 while maintaining comparable perplexity to UL.FOCUS combinations also improve BERTScore and maintain stable zero-shot accuracy, according to the reported results.Source: Table 4 · p. 7, §5.3 · pp. 7–8

5.4. LLM-as-a-Judge Evaluation

• In Llama-3.1-8B WikiText continuation judged by GPT-120B (reasoning), the proposed method consistently outperforms baselines.The authors use this LLM-as-a-judge evaluation to complement automatic metrics, which may not fully capture perceptual quality.Source: Table 6 · p. 8, §5.4 · p. 8

FOCUS Distribution Study

• 148.6 →202.1 viable head candidates accompany increased head agreement, while tail agreement remains stable and tail correlation rises substantially under FOCUS.The study compares token distributions on 200 teacher-generated responses, with the head defined by top-p (p = 0.9).Source: §6 · p. 8

Robustness to Likelihood Instability

• Despite increased WikiText PPL, FOCUS improves TruthfulQA performance over vanilla KD, while RePAIR achieves the largest overall gain.The authors relate FOCUS’s result to token-level supervision and capacity-aware distillation prioritizing salient teacher signals.Source: §6 · p. 8

Onset-Level vs. Sequence-Level Supervision

• RePAIR achieves larger repetition-reduction gains than sequence-level DPO in WikiText-103 continuation, consistent with onset-level corrective supervision targeting early loop entry.DPO is described as providing sequence-level preferences, whereas RePAIR supplies corrective alternatives at repetition onset (Rafailov et al., 2024).Source: §6 · p. 9

Empirical Analysis of Loop Persistence

• FOCUS delays and lowers loop-persistence growth relative to KD, and combining it with RePAIR suppresses persistence further.In synthetic WikiText-103 repetition contexts, KD’s loop-continuing token increasingly dominates escape mass as sentence occurrence rises; FOCUS delays that increase and saturation.Source: §6 · p. 9, Figure 4 · p. 9

Comparison with Distribution-shaping Distillation

• FOCUS reduces repetition more effectively than ToDi; combined with RePAIR, it achieves the lowest repetition scores while retaining generation quality and MAUVE comparable to ToDi.The comparison uses the same experimental setting as Table 3 on Llama 3.1-8B; ToDi is a token-level hybrid-KL method (Jung et al., 2025).Source: Table 7 · p. 9, §6 · p. 9

Number of Pairwise Data

• About 4k of RePAIR’s 12k pairwise samples suffice for repetition rates comparable to the full dataset, preserving more data for standard training than DITTO.The authors describe this as greater data efficiency because DITTO uses half of the training set for its auxiliary loss.Source: §6 · p. 9, Figure 5 · p. 9

• FOCUS uses token-probability-weighted distillation, and RePAIR uses a pairwise margin objective to guide models toward alternatives; both reduce repetition and improve generation quality.Source: §7 · p. 9

A. Repetition Analysis

• Repeated-token probabilities rapidly saturate in the repeated-sentence diagnostic, approximating high persistence and making escape increasingly difficult once a loop forms.The experiment tracked the model’s probability for each observed next token across within-sentence positions; persistence probability ρ(c) was not directly observable.Source: A. Repetition Analysis · p. 12, Figure 6 · p. 12

B. Gradient Analysis of FOCUS

• FOCUS replaces the standard KL distillation objective with a modified loss.The passage introduces the loss but does not provide enough readable formula content to specify its weighting rule.Source: B. Gradient Analysis of FOCUS · p. 13
• FOCUS’s gradient can be viewed as optimizing the student distribution with respect to a reweighted teacher distribution.The analysis defines a reweighted teacher distribution before giving this interpretation.Source: B. Gradient Analysis of FOCUS · pp. 13–14

C. An Example of Pairwise Data

• RePAIR identifies degeneration onset in pruned-model output, uses the preceding sequence as a prefix, and pairs it with a degeneration-free continuation from the unpruned model.Training pairs combine the prompt and prefix with negative and positive continuations for margin-loss optimization.Source: C. An Example of Pairwise Data · p. 14
• The pairwise dataset contains 12k examples, while the passage says 4k pairs are sufficient.The stated sufficiency is attributed to Figure 5; no further result details are supplied here.Source: C. An Example of Pairwise Data · p. 14

D. Zero-shot Accuracy

• Zero-shot evaluation covers ARC c, PIQA, BoolQ, OpenQA, Hellaswag, and Winogrande.The passages describe these as science reasoning, physical commonsense, yes/no comprehension, open-domain question answering, plausible continuation, and pronoun-resolution tasks, respectively.Source: D. Zero-shot Accuracy · pp. 14–15
• Zero-shot results are reported across six benchmarks for Llama-3.1-8B and Llama2-13B.The supplied table captions describe the evaluation scope but provide no cell values or model comparisons.Source: Table 8 · p. 15, Table 9 · p. 15

E. Implementation Details

• Baseline methods use the Hugging Face framework and official implementations; distillation uses temperature T = 2, while token-level unlikelihood is adopted because sentence-level variants challenge large-scale models such as LLaMA under memory constraints.The experiments set α = 0.5, described as balancing repetition suppression with fluency and perplexity.Source: E. Implementation Details · pp. 15–16
• Implementation settings use α = 0.5 for unlikelihood, γ = 0.5 for ScaleGrad, and half the training data with MSE loss and λ = 0.5 for DITTO.ScaleGrad’s γ setting is chosen to balance repetition mitigation and perplexity; DITTO constructs pseudo-repetition data by repeatedly appending the output to the instruction and input prefix.Source: E. Implementation Details · p. 16

F. Parameter Search

• FOCUS’s β and γ increases consistently improve unique n-gram scores but slightly raise perplexity, reflecting greater reliance on high-confidence tokens.The study uses open-ended WikiText-103 generation and notes that higher values improve generation-related metrics, so perplexity alone is not the sole quality criterion.Source: Figure 8 · p. 17, F. Parameter Search · p. 17
• With β fixed at 15, α1 = 0.05 balances perplexity and repetition reduction because perplexity rises sharply from 0.05 to 0.1 while the n-gram rate saturates from 0.1 onward.The search tracks perplexity and unique n-grams as α1 increases.Source: Figure 8 · p. 17, F. Parameter Search · p. 17

G. CREP: Coverage-based Repetition Metric

• CREP sweeps n-gram lengths, finds recurring positions and their modal gap, reconstructs a repeated segment, and returns the fraction of generated texts exceeding the coverage threshold.The procedure tracks each text’s maximum coverage before incrementing the degeneration count.Source: G. CREP: Coverage-based Repetition… · p. 18
• CREP marks a sentence degenerate when its maximum global coverage of a reconstructed repeated segment exceeds a predefined threshold across n-gram lengths 4–16.For each sentence, it identifies the most frequently recurring r-gram and measures how much of the remaining output the repeated segment covers.Source: G. CREP: Coverage-based Repetition… · p. 18

H. Experiment on Other Pruning Methods

• Across LLMPruner and other width- and depth-pruning methods, the method consistently reduces repetition on WikiText-103.The reported settings include Shortened-LLaMA, FLAP, and LLMPruner.Source: H. Experiment on Other Pruning Methods · p. 19, Table 11 · p. 19, Table 12 · p. 19
• The Qwen 2.5-3B results show that the method’s effectiveness is not limited to the LLaMA family.Table 13 reports open-ended WikiText-103 generation for Qwen 2.5-3B pruned with LLMPruner at ratio 0.25.Source: H. Experiment on Other Pruning Methods · p. 19, Table 13 · p. 19

I. Aggressive Pruning Settings

• Both FOCUS and RePAIR consistently improve repetition-related metrics, including n-gram rate, at 35% and 45% LLMPruner pruning ratios.These more aggressive settings test robustness where degradation is more prominent.Source: §I · p. 20, Table 14 · p. 20
• Pruning ratios above 30% typically induce noticeable degeneration, making the 35% and 45% LLMPruner settings a test of robustness under more prominent degradation.The cited prior work is Jaiswal et al. (2024).Source: §I · p. 20, Table 14 · p. 20
• MAUVE declines as pruning ratios increase, reflecting expected losses in model capacity and distributional fidelity rather than a limitation of the repetition-mitigation strategy.Source: §I · p. 20, Table 14 · p. 20
• Recovering semantic richness under severe compression may require capacity-recovery or representation-restoration techniques beyond this work’s scope.Source: §I · p. 20

J. DPO Training Details

• DPO uses RePAIR’s positive non-repetitive and negative repetitive pairs, matching its data and supervision for a fair sequence-level baseline.The objective follows the original DPO formulation (Rafailov et al., 2024).Source: J. DPO Training Details · p. 20
• Both methods use WikiText-103 continuation evaluation without KD or FOCUS, isolating sequence-level versus token-level supervision; DPO uses βdpo = 0.1.The temperature is set consistently with prior work.Source: J. DPO Training Details · p. 20

K. LLM-as-a-Judge

• The judge compares anonymized candidates A and B for the same example, selecting A, B, or TIE based on overall quality.Candidate order is randomized, and a preference that changes after swapping order is counted as TIE.Source: K. LLM-as-a-Judge · p. 21
• Judging prioritizes less degeneration and repetition, then encyclopedic style and coherence, local quality and factual plausibility, and sentence completion.The judge is instructed to penalize loops, duplicated spans, and obvious degeneration, and prefer neutral, expository, coherent Wikipedia-style continuations.Source: K. LLM-as-a-Judge · p. 21
• The judge must compare alternatives for the same example using only the candidates, then return exactly one JSON object with a single winner field.The permitted values are A, B, or tie; no explanation is allowed.Source: K. LLM-as-a-Judge · p. 21