PaperlayerView exact revision on arXiv

Source-linked AI summary

Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model

Jacqueline He, Jonathan Hayase, Wen-tau Yih, Sewoong Oh, Luke Zettlemoyer, Pang Wei Koh

arXiv:2602.07120v2cs.CL

TL;DR

Language models can reproduce memorized copyrighted text, creating risks for creators and developers. The paper introduces Anchored Decoding, which constrains a risky model near a permissively trained safe model with an adaptive information budget, and reports strong risk–utility trade-offs across six model pairs. AnchoredByte Decoding extends the approach across vocabularies, while the guarantees remain bounded by the study’s assumptions and do not constitute legal certification.

  • Problem

    Language models may reproduce copyrighted training data verbatim, raising creator consent, compensation, and developer compliance concerns.

  • Method

    Anchored Decoding fuses risky and safe models under adaptively allocated divergence budgets, with TinyComma 1.8B and byte-level decoding enabling broader model compatibility.

  • Results

    Across six model pairs, Anchored and AnchoredByte Decoding preserve near-original fluency and factuality while closing up to 75% of the measurable copying gap at modest inference overhead.

  • Takeaways & Limitations

    The method offers a training-free, user-controllable safeguard for retrofitting high-utility language models when a permissively trained safe reference is available.

  • Takeaways & Limitations

    The guarantees and empirical results depend on the study’s assumptions and evaluation protocol, and K-NAF is not a legal certification of noninfringement.

Abstract

from arXiv · show

Language models (LMs) tend to memorize portions of their training data and emit verbatim spans. When the underlying sources are sensitive or copyright-protected, such reproduction raises issues of consent and compensation for creators and compliance risks for developers. We propose Anchored Decoding, a plug-and-play inference-time method for suppressing verbatim copying: it enables decoding from any risky LM trained on mixed-license data by keeping generation in bounded proximity to a permissively trained safe LM. Anchored Decoding adaptively allocates a user-chosen information budget over the generation trajectory and enforces per-step constraints that yield a sequence-level guarantee, enabling a tunable risk-utility trade-off. To make Anchored Decoding practically useful, we introduce a new permissively trained safe model (TinyComma 1.8B), as well as Anchored$_{\mathrm{Byte}}$ Decoding, a byte-level variant of our method that enables cross-vocabulary fusion via the ByteSampler framework (Hayase et al., 2025). Across six model pairs on long-form metrics for copying risk and utility, Anchored and Anchored$_{\mathrm{Byte}}$ Decoding define a new Pareto frontier, preserving near-original fluency and factuality while closing up to 75% of the measurable copying gap between the risky baseline and a safe reference, at a modest inference overhead.

1. Introduction

Anchored Decoding is a training-free inference method that fuses risky and safe language models under a user-chosen divergence budget. It extends to mismatched vocabularies through TinyComma 1.8B and byte-level decoding, achieving strong risk–utility trade-offs in long-form evaluation.

  • Core method: Anchored Decoding interpolates risky and safe next-token distributions using local divergence budgets that compose into a sequence-level K-NAF guarantee.The method adaptively allocates a user-chosen budget over generation, including prompt-dependent prefix debt and adaptive banking.
  • Core method: Training-free decoding produces fluent, plausible continuations while avoiding verbatim reproduction and retrofits off-the-shelf language models with exposed logits.It requires access to a permissively trained safe anchor but does not require retraining or original pre-training data.
  • Vocabulary compatibility: TinyComma 1.8B is trained exclusively on 169.5B openly licensed Common Pile tokens and uses the Llama 3.1 tokenizer for direct compatibility.This model addresses token-level vocabulary alignment between permissively trained safe models and Llama-family risky models.
  • Vocabulary compatibility: AnchoredByte Decoding operates on next-byte distributions through ByteSampler, bypassing tokenizer mismatch and broadening compatible model pairs.The byte-level analogue retains K-NAF compliance while operating across tokenizers.
  • Results: Across six model pairs, Anchored and AnchoredByte Decoding achieve Pareto-optimal risk–utility trade-offs, closing up to 75% of the measurable copying gap.They retain near-original fluency and factuality with modest inference overhead, including 1.1x for TinyComma 1.8B paired with Llama 3.1 70B.
  • Results: KL divergence between risky and safe models identifies when generation enters a copyright-sensitive regime and provides a general signal beyond copyright mitigation.The framework is described as agnostic to tokenizer, modality, and domain when a trusted reference distribution is available.

2. Preliminaries

The paper models generation as autoregressive distributions over token sequences and frames safety as bounded divergence from a permissively trained reference model. The target distribution should retain risky-model utility while staying close to the safe model.

  • Autoregressive setup: A token-level autoregressive language model defines a distribution over variable-length sequences from a prompt, with tokens drawn from a fixed vocabulary and EOS marking termination.The sequence probability factors into conditional next-token probabilities through the generation horizon.
  • Autoregressive setup: The safe model is trained only on permissively licensed text, while the more capable risky model may reproduce copyrighted data from broader training.The models are assumed to share vocabulary support under standard softmax parameterizations.
  • Safety criterion: The central objective is to approximate the risky model for utility while strictly satisfying the safety constraint relative to the safe model.This formulation balances copyright safety from the safe model against utility from the risky model.
  • Safety criterion: Global K-NAF bounds total divergence between the generated sequence distribution and the safe model for every input and generation length up to Tmax.The paper primarily uses KL divergence as the divergence function.

3. ANCHORED DECODING

Anchored Decoding approximates an intractable sequence-level constrained optimization with per-step fusion of risky and safe distributions. Prefix debt and adaptive banking allocate the global safety budget while preserving the formal guarantee, including in byte space.

  • 3.1. A Tractable Token-Level Approximation: The method seeks a distribution close to the risky model while remaining subject to the global safety constraint relative to the safe model.Direct sequence-level optimization is computationally intractable because it searches over exponentially many token sequences.
  • 3.1. A Tractable Token-Level Approximation: Per-step constrained objectives replace direct sequence-level optimization, with KL chain-rule decomposition enabling local budgets whose sum remains bounded.Each local next-token distribution stays within its budget relative to the safe model while drawing close to the risky model.
  • 3.1. A Tractable Token-Level Approximation: Theorem 3.1 establishes that local constraints yield the global K-NAF guarantee when per-step budgets satisfy the global budget condition.A constant cap k satisfies the condition whenever k Tmax ≤ K.
  • 3.2. Solving for a fused distribution p∗: The optimal local fused distribution is a weighted geometric mean of the risky and safe next-token distributions.The mixing weight is determined through a one-dimensional root-finding problem for the KL constraint.
  • 3.3. Prefix debt δinit(x): Prefix debt conservatively reduces the available budget when prompt prefixes show strong risky–safe likelihood-ratio outliers associated with memorization.The offset can make early-step budgets effectively zero for memorization-triggering prompts while preserving the safety guarantee.
  • 3.4. An adaptive budgeting strategy: Adaptive banking rolls unused budget from low-risk steps forward for later high-risk spikes instead of applying the same allowance at every step.The complete method sets K := k Tmax, applies prefix debt, and allocates the remaining budget adaptively.
  • 3.6. ANCHOREDByte DECODING: ANCHOREDByte Decoding addresses shared-vocabulary limitations by operating on byte-level distributions induced from token probabilities.It remains K-NAF compliant with K = kBmax, while requiring BPE tokenizers that map tokens to UTF-8 byte strings.

4. Experiments

The experiments evaluate copying risk alongside fluency and factuality, using normalized copying reduction relative to a safe reference rather than zero overlap. They compare Anchored Decoding with single-model and joint-model mitigation baselines across multiple model pairs.

  • Evaluation metrics: Normalized copying reduction measures the fraction of the gap between risky and safe reference performance closed by a setting.The aggregate NCR is averaged across six metrics, treating the safe reference as the safety gold standard rather than requiring zero overlap.
  • Evaluation metrics: The high-protection regime is defined as NCR ≥ 75%, isolating settings with substantial rather than marginal copying-risk reduction.The threshold is acknowledged to be task-specific but is chosen to identify dominant mitigation effects.
  • Evaluation metrics: Utility evaluation covers fluency on BOOKS continuations and factuality on BIOS biography prompts.Fluency uses Prometheus-v2’s five-point rubric, while factuality uses FActScore claim precision from independently fact-checked atomic claims.
  • Baselines and models: Baselines include system prompting, MEMFREE n-gram blocking, Reversed Context Aware Decoding, CP-FUSE, and TOKENSWAP.CP-FUSE assumes disjoint copyrighted training shards, whereas TOKENSWAP manually replaces selected common-token probabilities.
  • Baselines and models: Risky models include Llama 3.1 70B, Qwen 2.5 72B, and Llama 4 Scout 17Bx16E, selected for measurable verbatim reproduction.The experiments use base, non-instruction-tuned models to isolate memorization effects in next-token distributions.
  • Baselines and models: The evaluation sweeps the method’s k parameter across values from 0.1 to 20 with Tmax=200 and Bmax=800 bytes.The prefix-debt window is set to n=5.

5. Results

Across token- and byte-level decoding, Anchored Decoding methods trace strong risk–utility frontiers and retain the best utility among configurations reaching high protection. Ablations show that the KL-based objective, prefix debt, and adaptive budgeting each improve the trade-off, while overhead remains modest.

  • 5.1. Risk–utility trade-offs: Anchored and AnchoredByte Decoding trace the Pareto frontier across model pairs at both token and byte granularities, with statistical significance over three seeds.Table 1 evaluates the best factuality/fluency among configurations satisfying NCR≥75%.
  • 5.2. ANCHORED DECODING ablations: The KL-based ANCHORED DECODING formulation is more Pareto-optimal than NoOpt and ColdStart, while ANCHORED DECODING∞ improves fluency trade-offs but worsens factuality.The ∞-Rényi variant provides a worst-case rather than average-case divergence guarantee.
  • 5.2. ANCHORED DECODING ablations: Top-n prefix-debt aggregation consistently outperforms averaging all prefix likelihood ratios, and removing prefix debt further degrades the trade-off.The results support treating prefix debt as a tail statistic.
  • 5.2. ANCHORED DECODING ablations: Adaptive budgeting yields a strictly better trade-off than both a constant per-step cap and a holistic lump-sum budget.Adaptive budgeting accounts for realized spending in prior steps, unlike Fixed and Global.
  • 5.3. Efficiency: ≈1.1× slower throughput: token-level ANCHORED DECODING incurs modest wall-clock overhead, while the added safe-model forward pass increases arithmetic compute by only ≈2.6%.The benchmark uses the {TinyComma 1.8B, Llama 3.1 70B} pair and compares throughput with standalone risky-model decoding.

6. Analysis

The analysis finds that divergence and prefix log-likelihood ratios distinguish copyright-sensitive generation, while copying events concentrate early and motivate prefix-dependent intervention.

  • 6.2. KL is a Useful Memorization Diagnostic: Per-step KL divergence sharply separates Copyright prefixes from Creative and Factual prefixes, especially in the extreme right tail.Under a shared pooled 90th-percentile threshold, Copyright has the greatest fraction of steps with large divergence.
  • 6.2. KL is a Useful Memorization Diagnostic: Per-step KL(pr ∥ ps) provides a diagnostic signal for identifying when risky-model generation enters a copyright-sensitive regime.The diagnostic relies on risky-model distributions becoming sharply concentrated on continuations that the safe model considers unlikely.
  • 6.3. Copying Risk is Front-loaded in Early Generation: Copying events tend to occur early in generation across token-level and byte-level decoding.The study examines LCS and ACS under both any-overlap and substantial-overlap regimes, with overlap concentrating near the beginning.
  • 6.3. Copying Risk is Front-loaded in Early Generation: Prefix debt operationalizes front-loaded risk by increasing reliance on ps early when the prompt shows asymmetric memorization evidence.The initial budget is debited according to how strongly the prefix favors pr over ps.
  • 6.2. KL is a Useful Memorization Diagnostic: Copyright prefixes exhibit heavier right tails in prefix LLRs and right-shifted prefix-debt values than Creative and Factual prefixes.The pattern persists into extreme quantiles, including the 99.5 percentile, supporting a selective cold start.

7. Discussion

Anchored Decoding offers a controllable, training-free safety mechanism that can retrofit existing language models and extend across tokenizer configurations. The discussion reports a risk–utility trade-off supported by model contrast and byte-level decoding.

  • Flexibility: Anchored Decoding exposes a sequence-level safety budget K that lets practitioners choose a formal risk threshold.The method provides a controllable risk–utility trade-off rather than a fixed operating point.
  • Practicality: Anchored Decoding requires neither retraining nor access to original pre-training data during inference, enabling retroactive safeguarding of existing models.The method is presented as practical for high-utility models with infringement leakage.
  • Tokenizer agnosticism: AnchoredByte Decoding removes the shared-vocabulary requirement and works across model pairs with mismatched tokenizers.The byte-level variant extends the strategy beyond directly compatible tokenizers.
  • Risk–utility trade-off: The method yields a strong risk–utility trade-off by using risky–safe model contrast as a signal of likely copyrighted regimes.The discussion connects this signal to selective control of generation while retaining general capabilities.

8. Impact Statement

The paper addresses copyright risks from memorization in web-scale language models and positions Anchored Decoding as an inference-time complement to other safeguards. It also highlights practical constraints and related methodological distinctions.

  • Motivation: Web-scraped training corpora may contain copyrighted material, creating risks of infringement, reduced creator market value, and inadequate consent or compensation mechanisms.The impact statement frames these concerns as harms associated with substantial similarity between outputs and protected training examples.
  • Intervention scope: Anchored Decoding operates at inference time, avoiding the expense of retraining while remaining orthogonal to pre-training, post-training, and post-generation procedures.The authors leave integration with those procedures to future work.
  • Related work: Unlike CP-FUSE, Anchored Decoding does not require disjoint training shards and exposes a user-adjustable risk–utility control knob.Its pairing instead assumes copyright-infringing data may be present in pr but not ps.
  • Related work: Unlike TOKENSWAP’s model-size heuristic, Anchored Decoding uses an explicit safe–risky pairing and distribution-level fusion with a per-step divergence budget.This design addresses mismatched assumptions about memorization and constrains generation relative to the safe model.
  • Tokenizer compatibility: Byte-level adaptation removes the shared-tokenization bottleneck and supports decoding with mismatched vocabularies.The approach is implemented through ByteSampler-based integration.

A.3. Limitations and Future Work

The authors qualify Anchored Decoding’s guarantees by limiting them to the study’s assumptions and evaluation protocol. They identify residual risk, local optimization, imperfect risk proxies, provenance requirements, and unaddressed threat models, while outlining broader applications.

  • Probabilistic risk: Sampling does not eliminate protected-span reproduction, because Anchored Decoding inherits the safe model’s non-zero baseline reproduction risk.The method bounds infringement risk under K-NAF rather than making reproduction impossible.
  • Local approximation: The per-step local optimization may differ from the global optimum of the sequence-level constrained objective.This approximation is adopted for computationally efficient autoregressive decoding.
  • Asymmetric memorization as an imperfect proxy: Asymmetric memorization can suppress rare non-copyrighted facts that the risky model knows but the safe model lacks.The utility cost depends on the model pair and may decrease when the safe reference is closer in capability.
  • Data provenance and latent leakage: The method requires prior identification of a safe model with verified copyright-free or openly licensed training data provenance.Latent protected fragments in ostensibly open data can leave residual reproduction risk.
  • Future work: The primary threat model concerns parametric training-data memorization and does not cover copyrighted text explicitly supplied in the prompt.Future directions include applying the approach to policy compliance, code safety, privacy redaction, and other domains.

B.1.1. A TOKEN-LEVEL APPROXIMATION

The token-level approximation constrains each fused next-token distribution relative to the safe model, and these local constraints compose into a sequence-level K-NAF safety guarantee. The method uses a closed-form weighted fusion, with adaptive budgeting and byte-level extension preserving analogous guarantees.

  • Safety guarantee: Per-step budgets whose sum is at most K yield a global K-NAF guarantee for every continuation up to length Tmax.The proof decomposes sequence-level KL divergence into an expectation over local conditional divergences.
  • Token-level approximation: The optimization has an interior optimum on the common support, assigning positive probability to every supported token.A boundary solution can be improved by transferring an infinitesimal mass to a zero-probability supported token.
  • Token-level approximation: The optimal token distribution is a weighted geometric mean of the risky and safe next-token distributions.The weight is determined by the dual variable associated with the KL constraint, with normalization constant Z.
  • Adaptive budgeting: Adaptive banking reallocates unspent low-risk budget to later high-risk steps while incorporating prompt-dependent prefix debt.The prefix adjustment reduces the initial budget when the prompt appears memorized, and banking preserves the global budget bound.
  • Byte-level extension: Byte-level decoding retains the safety guarantee and can steer away from memorized strings at the exact character where divergence occurs.ByteSampler induces autoregressive byte distributions, enabling finer-grained control than multi-byte token-level decoding.
  • Worst-case divergence: The ∞-Rényi variant provides a worst-case alternative, with local pointwise ratio bounds composing into a global K-NAF guarantee.The sequence-level objective is computationally intractable for autoregressive decoding, motivating the local formulation.

C.1. Pretraining TinyComma 1.8B

TinyComma 1.8B is a permissively trained compact safe model designed for tokenizer compatibility with Llama 3.1. It performs strongly among open models of its size and supports token- and byte-level Anchored Decoding experiments.

  • Model construction: TinyComma 1.8B is trained exclusively on permissively licensed Common Pile data and uses the Llama 3 family tokenizer.The model is trained for 169.5B tokens, including general pretraining and a quality-focused cooldown stage.
  • Evaluation: TinyComma 1.8B is the most performant open model in its size range on the reported standard language evaluation tasks.It outperforms other safe models of similar size but underperforms the larger, more extensively trained Comma 7B models.
  • Prefix-debt analysis: Prefix debt improves the Anchored Decoding trade-off for every tested setting with n > 0, while curves are largely insensitive to the chosen n.The sweep uses the TinyComma 1.8B and Llama 3.1 70B token-level pair.
  • Prefix-debt analysis: Higher prefix debt correlates with stronger copyright-copying indicators across all tested memorization-window choices.Figure 9 groups overlap-based copying indicators by prefix-debt decile for the TinyComma–Llama 3.1 70B pair.
  • Baseline details: MEMFREE and RCAD require inference-time blocklists constructed from retrieved contexts, whereas CP-FUSE assumes comparable models trained on mutually exclusive data shards.These baseline requirements differ from Anchored Decoding’s setup and motivate the reported comparisons.

D.5. Efficiency

ANCHORED DECODING adds two-model inference and budget-management costs, with forward passes dominating arithmetic overhead. Byte-level decoding and prefix-debt computation introduce additional practical latency, while adaptive budgets track localized risk spikes and long-tail factuality remains a concern.

  • Arithmetic cost: ANCHORED DECODING runs one forward pass of each model per generated token, estimated at 2(Nr + Ns) FLOPs/token.Prefix debt reuses the two-model prefill, while logit fusion is dominated by vocabulary-sized reductions.
  • Arithmetic cost: For billion-parameter LMs, the two-model forward-pass term dominates the arithmetic FLOP count.The amortized estimate includes the two-model prefill contribution over the prompt.
  • Wall-clock efficiency: Byte-level decoding has higher TTFB and worse TPS than token-level decoding because its implementation is less optimized and large-model sharding adds communication overhead.The 70B model must be sharded across GPUs, slowing throughput.
  • Wall-clock efficiency: Prefix debt raises byte-level TTFB from 186.3 ms to 3566.8 ms by computing log probabilities and BPE tree decompositions at every prompt byte.This adds O(L) computation with a substantially larger constant factor than standard prefill, even with caching.
  • Adaptive budgeting and factuality: Adaptive budgeting banks allowance during low-risk steps and releases it during memorization spikes, while factual precision improves with k but long-tail entities improve more slowly.At low k, the decoder tracks the safe model closely; byte-level risk appears as sharper localized spikes.

E.3. ANCHORED DECODING on Other Settings

The method extends beyond base-model literary evaluations to instruction-tuned models, news, downstream utility tasks, and non-literal copying. These experiments report favorable risk-utility tradeoffs, while non-literal copying remains domain- and task-dependent.

  • Other domains and utility metrics: On a news domain with an alternative pairwise win-rate utility metric, ANCHORED DECODING remains effective.The experiment uses 560 New York Times articles from NewsSpan.
  • General-purpose utility: At k = 1.5, ANCHORED DECODING performs comparably to Llama 3.1 70B on TruthfulQA, CNN/DailyMail summarization, and HumanEval.The result indicates preserved utility in the evaluated high-protection regime.
  • Non-literal copying: The non-literal copying experiment measures character overlap above three matched names and reports non-verbatim fluency in open-ended story generation.Prefix debt is disabled because it is primarily designed for literal copying and can hurt fluency in this setting.
  • Instruction-tuned models: ANCHORED DECODING defines the Pareto frontier for an instruction-tuned model pair, achieving the best tradeoff between fluency and non-literal copying mitigation.The evaluation uses Llama 3.1 70B Instruct with Comma 1.7B.
  • Non-literal copying: Non-literal copying may require task-aware evaluation because overlap is typically a non-issue for factual news content, and exhaustive evaluation is deferred.The stated domain boundary concerns transformations and literary settings.

E.4. Full Tradeoff Results

The full tradeoff results cover six model pairs, multiple domains, and token- and byte-level decoding. They include examples, adaptive-budget diagnostics, factuality stratification, downstream tasks, and wall-clock comparisons.

  • Tradeoff results: Full results are reported for six risky-safe model pairs spanning Llama 3.1, Qwen 2.5, and Llama 4 Scout risky models.The safe models are TinyComma 1.8B or Comma 7B, with results averaged over three seeds.
  • Examples and domains: The examples cover BOOKS, BIOS, CREATIVE, and byte-level literary generations, including prompts for biography and prefix completion.The tables identify the source works and heldout Creative-domain construction.
  • Factuality: Factual precision rises with k across entity-frequency buckets, but Rare and Very Rare entities improve more slowly and plateau at lower precision.Frequent entities recover quickly, while the safe model remains uniformly low across buckets.
  • Downstream tasks: Downstream evaluation reports ANCHORED DECODING results alongside risky and safe models on TruthfulQA, CNN/DailyMail, and HumanEval.The table covers question answering, summarization, and code generation.
Loading 2602.07120v2…