PaperlayerView exact revision on arXiv

Source-linked AI summary

XTC: Head-Aware Sampling by Excluding Top Choices

Philipp Emanuel Weidmann, Allen Roush, Judah Goldfeder, Sanjay Basu, Ravid Shwartz-Ziv

arXiv:2608.22758v1cs.CLcs.AIcs.LG

TL;DR

Autoregressive decoding can leave probability overly concentrated on generic choices even when several continuations are plausible, motivating a method that targets this head-ambiguity regime. XTC identifies plausible tokens, probabilistically removes dominant eligible choices, and renormalizes the remainder. Across models and tasks, it improves the diversity-repetition frontier while preserving quality within supported operating boundaries.

  • Problem

    Existing decoding rules overlook head ambiguity, where several plausible continuations exist but probability remains concentrated on the generic choice.

  • Method

    XTC applies an absolute plausibility threshold and, when at least two tokens qualify, probabilistically removes dominant eligible choices while retaining the weakest plausible alternative before renormalization.

  • Results

    XTC improves the diversity-repetition Pareto frontier across four model families, with creative-task Distinct-2 gains of 11–15% and repeat-trigram reductions of 27–47%.

  • Takeaways & Limitations

    XTC is a lightweight, compositional decoding intervention that adds diversity gains to temperature and repetition penalties while remaining robust across model families and quantization levels.

  • Takeaways & Limitations

    XTC can hurt exactness-sensitive tasks, with code pass rates degrading significantly above ρ = 0.15 and structured extraction bounded at ρ = 0.20.

Abstract

from arXiv · show

Standard decoding rules for autoregressive language models promote diversity by rescaling the full next-token distribution or truncating its low-probability tail. These strategies overlook a common regime of open-ended generation in which several continuations are plausible but too much probability mass remains concentrated on the most generic choice. We introduce XTC (Exclude Top Choices), a lightweight head-aware decoding operator that targets this regime directly. XTC identifies tokens whose probabilities exceed an absolute plausibility threshold $τ$: when at least two qualify, it removes the dominant eligible choices with probability $ρ$ and retains only the weakest plausible alternative before renormalization. Across 60 experiments on Gemma 3 27B Q4, Gemma 3 12B Q6, and DeepSeek R1 14B Q6, with scaling validation on Llama 3.3 70B Q4, XTC improves the diversity-repetition Pareto frontier. On creative generation, Distinct-2 increases by 11--15% and repeat trigrams decrease by 27--47% across the four models. Combined with temperature scaling, gains reach 38% in Distinct-2 and 71% in repeat-trigram reduction over baseline. A blinded Amazon Mechanical Turk study with 150 Master raters yields a 62.3% creativity preference for XTC ($p<10^{-4}$) without reduced fluency, while a GPT-4o control judge reproduces the Anthropic-judge direction on every measure. On IFEval with Llama 3.3 70B Q4, XTC preserves prompt-level strict accuracy within 1.7 percentage points of baseline while recovering most of the diversity gain; a temperature setting matched on Distinct-2 reduces IFEval by 8.8 points. The effect is additive with temperature and repetition penalties, robust across quantization levels and model families, and consistent across twelve prompt genres. XTC has been adopted by llama.cpp, ExLlamaV2, and text-generation-webui.

1 Introduction

Open-ended decoding can fail when several plausible continuations exist but probability concentrates on the generic choice, a regime that tail truncation and global flattening do not target. XTC addresses this regime with a sparse, head-aware operator that improves diversity while preserving quality within identified boundaries.

  • XTC targets head ambiguity, where several plausible continuations coexist but the safest continuation is overweighted.
  • XTC identifies tokens above an absolute plausibility threshold τ, then selectively excludes dominant eligible choices and renormalizes the distribution.
  • XTC is sparse in time, head-aware, compositional with existing samplers, and conditionally beneficial when head multiplicity is present.
  • 60 experiments across four model families evaluate creative diversity, degeneration, code exactness, sampler composition, parameter interactions, and robustness.
  • Distinct-2 gains of 11–15% across four models improve the diversity-repetition Pareto frontier on creative tasks.
  • Combined with temperature and repetition penalties, XTC reaches 38% higher Distinct-2 and 71% lower repeat-trigram rates than baseline.

2 Background and related work

Prior decoding methods primarily truncate low-probability tokens, rescale distributions, contrast alternatives, or steer generation with auxiliary mechanisms. XTC instead targets plausible head tokens, motivated by evidence that language-model-assisted writing can reduce population-level diversity.

  • Existing decoding methods include tail truncation, temperature scaling, contrastive approaches, discriminator-based steering, and training-time diversity interventions.
  • Diversity evaluation uses multiple complementary metric families, alongside LLM-based judges for open-ended text.
  • LM-assisted writing can reduce content diversity across creative writing, ideation, expression, and reasoning, motivating decoding-time interventions.
  • Position of XTC: XTC requires no retraining, beams, or auxiliary models and operates on a single next-token distribution after upstream transformations.
  • Position of XTC: Unlike locally typical sampling, XTC removes individually plausible high-probability tokens rather than atypical low-probability tokens.

3 The XTC decoding rule

XTC transforms a next-token distribution only when at least two tokens clear an absolute plausibility floor, probabilistically retaining the least probable eligible token and renormalizing the support. It can be inserted after upstream sampling transformations, while protected tokens and unchanged pass-through behavior are part of the implementation context.

  • XTC operates on a next-token distribution that may already include temperature, penalties, or truncation.
  • The operator can be read as discarding dominant choices when the model is undecided between strong alternatives and retaining the underdog.
  • The sampling step collects tokens with probability at least τ and leaves the distribution unchanged when fewer than two are eligible.
  • Figure 1 illustrates eligibility, exclusion of dominant head tokens, preservation of the non-eligible tail, and renormalization into qt.
  • With intervention probability ρ, XTC removes eligible dominant choices, keeps the lowest-probability eligible token, and rescales surviving probabilities to sum to one.

4 Experiments

Across models, prompts, and evaluation axes, XTC improves creative diversity and repetition metrics while preserving quality and instruction-following more effectively than several baseline samplers.

  • Evaluation setup: 60 experiments span Gemma 3 27B q4, Gemma 3 12B q6, DeepSeek R1 Qwen 14B q6, and Llama 3.3 70B q4 scaling validation.The evaluation covers creative diversity, code exactness, quality retention, long-form repetition, and twelve creative prompt genres.
  • Open-ended diversity: XTC at ρ=1.0 and τ=0.1 achieves the best score on every creative metric, with significant Distinct-2 and repeat-trigram improvements.XTC wins 24 of 24 prompts on Distinct-2 and 22 of 24 on repeat trigram rate; the effect generalizes across all twelve genres.
  • Open-ended diversity: XTC conditions dominate the diversity-versus-repetition Pareto frontier, and T=1.3 plus XTC dominates T=1.3 alone across four metrics.XTC alone also outperforms tail-shaping baselines, while min-p plus XTC is reported as best overall.
  • Sampler composition: Pairing XTC Medium with temperature, top-p, or repetition penalties improves Distinct-2 and reduces Self-BLEU-4 in every tested pairing.The T=1.3 plus XTC composition attains the best score on every metric in a ten-metric headline.
  • Quality evaluations: Cross-vendor judges and human raters report greater creativity or diversity with quality preserved or statistically null overall-quality differences.The AMT study reports XTC preferred for creativity, while Opus and GPT-4o agree on repetition improvement and null overall quality at 27B and 70B evaluations.

5 Conclusion

XTC is a head-aware decoding operator that activates only when multiple tokens clear an absolute plausibility threshold, then removes dominant eligible choices and renormalizes. Its structural properties distinguish it from entropy-based controls, while experiments report improved diversity-repetition trade-offs and broad compositionality.

  • Empirical conclusion: Across 60 experiments and four model families, XTC improves the diversity-repetition Pareto frontier and composes additively with temperature and repetition penalties.The reported evaluation spans creative generation, long-form degeneration, code exactness, and instruction following.
  • Operator behavior: When active, XTC removes dominant eligible tokens while retaining the weakest eligible token and renormalizes the surviving distribution.The removed set is the eligible set excluding its minimum-probability token.
  • Operator behavior: XTC activates when at least two tokens exceed an absolute threshold and otherwise returns the input distribution unchanged.Activation also requires the Bernoulli intervention draw to fire.
  • Structural properties: XTC preserves relative probability odds among surviving tokens after its sparse support edit.This separates support removal from distortion of preferences among retained tokens.
  • Structural properties: The transformed distribution uniquely minimizes KL(q ∥ p_t) among distributions supported on the surviving tokens.Renormalization therefore has a restricted-support information-projection interpretation.
  • Structural properties: XTC responds to head multiplicity rather than entropy, activating for several individually plausible local branches even when global dispersion is modest.A diffuse tail can yield high entropy without activation if only one token clears the threshold.

B Proofs

The proofs establish XTC’s no-op behavior, relative-odds preservation, removed-mass characterization, threshold monotonicity, and KL-projection formulation. Together, these results clarify when the operator acts and how its intervention strength and transformed distribution are determined.

  • Propositions and proofs: If fewer than two eligible tokens exist or the intervention draw fails, XTC leaves the input distribution unchanged.This proves the no-op criterion directly from the operator’s activation checks.
  • Propositions and proofs: When XTC activates, every surviving token is divided by the same normalization factor, preserving relative odds.The ratio qt(v)/qt(w) therefore equals pt(v)/pt(w) for surviving tokens.
  • Propositions and proofs: The conditional removed mass equals the eligible mass minus the minimum eligible probability, yielding expected removed mass ρM_t(τ).The operator removes every eligible token except the weakest eligible token.
  • Propositions and proofs: Increasing τ can only shrink the eligible set and weakly decrease both the removable token count and removable mass.The proof follows from set inclusion E_t(τ2) ⊆ E_t(τ1) whenever τ2 ≥ τ1.
  • Propositions and proofs: Among distributions supported on the surviving tokens, the renormalized distribution uniquely minimizes KL divergence from the original distribution.Lagrange multipliers give q(v) proportional to p_t(v) on the surviving support, with uniqueness from strict convexity.

C.1 Limitations

XTC’s benefits are bounded by task exactness, parameterization, formatting, model coverage, and broader deployment risks. The paper therefore frames XTC as a task-gated local decoding rule rather than a universal solution.

  • Scope and task boundaries: XTC can hurt exactness-critical tasks when the highest-probability eligible token is required.The paper identifies code, extraction, constrained formatting, brittle mathematical reasoning, and safety-critical instruction following as vulnerable settings.
  • Parameterization: The threshold τ and activation probability ρ are task- and model-dependent rather than universal settings.Absolute-probability thresholds can behave differently across model scales, tokenizers, and sampler stacks.
  • Structured generation: Formatting and termination tokens entering the eligible set can destabilize structured outputs.The paper recommends task gating or protecting end-of-sequence, newline, indentation, and schema-critical delimiter tokens.
  • Empirical scope: The evaluation covers quantized open-weight models from 12B to 70B, leaving broader architectural and language generality untested.The paper calls for evaluation on mixture-of-experts and state-space models, plus non-English languages.
  • What XTC cannot solve: XTC redirects choices among plausible tokens but cannot replace model quality, calibration, or training-time anti-degeneration methods.It also cannot address corpus-level homogenization originating in training data overlap or RLHF reward hacking.
  • Broader impacts: Increasing output variety may also make harmful generations more varied, requiring task gating, safety filters, and harmful-use evaluation.The paper highlights spam, deception, and disinformation as relevant misuse risks.

D Cross-model generalization

XTC generalizes across four model families and quantization settings, improving diversity while reducing repetition. Its strongest Distinct-2 gain increases with model scale, although repetition gains face a ceiling from low baseline repetition.

  • Repetition ceiling: Llama 3.3 70B’s repeat-trigram confidence interval just touches zero because its baseline repetition is already low.Its baseline repeat trigram rate is 0.040, compared with 0.082 for Gemma 3 27B.
  • Cross-model results: Distinct-2 increases 11–15% and repeat trigram rate decreases 27–47% across four model families.Every model’s Distinct-2 confidence interval excludes zero; repeat-trigram improvements are significant for three families.
  • Scaling: +15.1% is the largest Distinct-2 improvement, observed for Llama 3.3 70B q4.This extends the effect to a third architecture family and a parameter count 2.6× larger than Gemma 3 27B.
  • Scaling: Distinct-2 gains rise monotonically from +11.4% at 12B to +15.1% at 70B under the strongest XTC conditions.The plotted points are Gemma 3 12B q6, Gemma 3 27B q4, and Llama 3.3 70B q4.
  • Quantization robustness: Gemma 3 27B shows no qualitative divergence between q4 and q6 at any operating point.The quantization comparison supports robustness across these two quantization levels.

E Design ablation

The design ablation supports XTC’s specific keep-weakest-eligible rule rather than generic head exclusion. Retaining the least probable eligible token produces the strongest diversity and repetition outcomes under matched parameters.

  • Parameter roles: XTC’s τ controls eligible alternatives while ρ controls how often intervention occurs.Higher τ makes XTC more conservative, and the operator can follow temperature, repetition penalties, and tail truncation.
  • Ablation outcome: Keep Least achieves the highest Distinct-2, lowest Self-BLEU-4, and largest repeat-trigram reduction on Gemma 3 27B q4.The comparison uses matched parameters ρ=1.0 and τ=0.1.
  • Mechanism: Drop Top-1 Only provides only partial improvement, indicating that effective head exclusion removes multiple dominant tokens.The ablation compares Keep Least, Keep Second, Keep Random, and Drop Top-1 Only.

F Composition with existing samplers

XTC composes with existing samplers rather than replacing them. Across pairings and factorial settings, it improves diversity and repetition metrics, with the strongest combined condition leading the reported comparison.

  • Additivity: Full factorials across temperatures and repetition penalties show approximately additive gains across three model families.The factorial grid covers three temperature levels and three repetition-penalty levels.
  • Combined settings: The composition T=1.3 + XTC achieves the best score on every metric in the ten-metric comparison.Table 2 reports this result for Gemma 3 27B q4.
  • Combined settings: XTC alone at ρ=0.75 outperforms T=1.3 alone on five of ten metrics.This comparison is reported in the extended Gemma 3 27B q4 metric table.
  • Sampler pairings: Adding XTC Medium improves Distinct-2 and reduces Self-BLEU-4 for every tested base sampler pairing.The pairings include baseline, temperature 1.1, top-p 0.95, and repetition penalty 1.05, with no degradation observed.
  • Mechanism: Temperature flattens the distribution globally, whereas XTC selectively removes dominant head tokens.The paper presents these interventions as complementary aspects of the distribution.

G Sampler composition: interaction across models

Across three model families, XTC combines additively with temperature and repetition penalties, improving diversity while reducing repetition. Heatmap and synergy analyses show no meaningful interference, saturation, or super-additive interaction.

  • Temperature × XTC: At all tested temperatures and across all three models, XTC Medium and Strong improve Distinct-2 while reducing Self-BLEU-4 and repeat trigram rate.The factorial study used 24 prompts and five seeds per condition.
  • Repetition penalty × XTC: Adding XTC to a repetition-penalty baseline further improves every metric on every model, with approximately additive effects.The interaction shows no evidence of saturation or interference between the mechanisms.
  • Interaction structure: Distinct-3 heatmaps show diagonal improvement across both temperature × XTC and repetition-penalty × XTC grids, consistent with additive composition.Improvement increases independently along both axes.
  • Synergy analysis: Joint improvement remains near the sum of individual effects, with no model showing super-additive or sub-additive interaction above noise.The synergy analysis therefore supports additive rather than synergistic composition.

H.1 Statistical significance and win/loss analysis

Across comparator, significance, robustness, mechanism, genre, and quality analyses, XTC consistently improves diversity and repetition metrics while retaining structured-output quality. Stronger evidence comes from significant paired improvements, broad prompt coverage, and favorable comparisons with temperature and tail-shaping baselines, although some combinations can reduce fluency.

  • Statistical significance: XTC Medium and Strong significantly improve Distinct-2, while repeat-trigram reductions are significant at every XTC strength.The forest plot reports p<0.001 for Distinct-2 and p≤0.002 for repeat trigram rate.
  • Win/loss analysis: XTC Strong wins on Distinct-2 for 24/24 prompts and on repeat trigram rate for 22/24 prompts versus baseline.These paired win/loss counts come from 24 Gemma 3 27B q4 prompts.
  • Tail-shaping comparators: XTC alone outperforms top-p, eta sampling, and min-p on all four reported metrics, while min-p + XTC is best in every column.The comparison uses a 24-prompt Gemma 3 27B q4 creative pool.
  • Instruction following: XTC Medium loses 0.13 IFEval points per Distinct-2 percentage gained, versus 0.63 for matched-Distinct-2 temperature scaling.The table describes this as a ∼5× ratio in favor of XTC.
  • Temperature comparison: T=1.3 + XTC achieves the highest score on all four metrics against strong comparator conditions.The reported composition uses ρ=0.75 and τ=0.05.
  • Robustness across prompts: Positive diversity gains occur across all 12 prompt genres, and XTC Strong achieves the lowest repetition across all 12 genres.Additional analyses cover extended diversity, embedding geometry, long-form repetition, and structured-quality retention.
Loading 2608.22758v1…