PaperlayerView exact revision on arXiv

Source-linked AI summary

Copy-Paste Language Models

Nathan Godey, Yoav Artzi

arXiv:2610.19247v1cs.CLcs.AI

TL;DR

Copying is common in context, but the conventional LM head can make it difficult to learn. CPLMs add a lightweight pointer-based head alongside the LM head, improving training and downstream performance and enabling sliding-window models to match full attention with less memory.

  • Problem

    Context copying is common, but the conventional low-rank LM head constrains expressivity and discards training signal, raising the question of how to model copying more efficiently.

    Source: §1 · p. 1

  • Method

    CPLMs add a Copy-Paste head that points to past context positions, with a less-compressed prediction and gradient pathway alongside the conventional LM head.

    Source: §1 · p. 2

  • Results

    Across tested backbones, CPLMs improve training speed, perplexity, downstream performance, and long-context results; 1.7B FinePDF perplexity improves by 0.5–1.5 points.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Sliding-window CPLMs match full-attention models with up to 12× less memory per token, while the CP head’s copy keys offer a memory-efficient retrieval option.

    Source: §1 · p. 2

    Because CPLMs copy from context more easily, they may reproduce sensitive or copyrighted input content more faithfully.

    Source: ETHICS STATEMENT · p. 11

Abstract

from arXiv · show

Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head. CPLMs improve training efficiency, downstream performance, and offer new operating points in long-context and retrieval setups. We validate our approach by pretraining models from 135M to 1.7B parameters, showing improvements across all scales and backbones with negligible training and inference overheads. We explain the success of CPLMs by showing that they partially mitigate the LM head gradient bottleneck through an alternative gradient path.

1 INTRODUCTION

Although context repetition makes copying a common LM operation, the conventional output head constrains copying. CPLMs add a pointer-based Copy-Paste head, improving training and model performance while offering different long-context memory trade-offs.

1 INTRODUCTION

  • About 60% of FineWeb-Edu tokens already occur in their preceding 4k-token context, yet the narrow LM head can constrain distributions and discard training signal.The introduction connects this bottleneck to vanilla Transformers failing a synthetic copying task at large vocabulary sizes.Source: §1 · p. 1
  • CPLMs add a Copy-Paste head that points to past context positions, with the LM head gating copying and a sink allowing fallback to the LM prediction.The added pathway is less compressed and partially mitigates the LM head gradient bottleneck; its parameter and compute overheads are 0.5% and 3% at 1.7B scale.Source: §1 · p. 2
  • CPLMs reach vanilla models’ final loss up to 2× faster and improve perplexity at every tested scale and backbone, with 0.5–1.5-point gains on 1.7B FinePDFs.They also improve supervised fine-tuning by 1–5 average accuracy points across math, code, and instruction following.Source: §1 · p. 2
  • Sliding-window CPLMs match full-attention and hybrid models with up to 12× less memory per token and retrieve better from contexts longer than training contexts.The CP head can read documents through copy keys alone, offering a memory-efficient alternative for retrieval-augmented generation.Source: §1 · p. 2
  • CPLMs improve the prior NanoGPT speedrun record by 11.3%, the largest architecture-centric speed-up and second largest overall since December 2024.The passage attributes these speedrun results to CPLMs but does not provide further experimental details here.Source: §1 · p. 2

2 METHOD

The method addresses the LM head’s vocabulary-space and gradient bottlenecks with a second prediction pathway over context positions. A gate and sink mix that copy distribution with the conventional LM prediction.

2.1 BACKGROUND & NOTATION

  • Because the LM head projects from a D-dimensional representation, it cannot express every distribution over a much larger vocabulary; its backward pass also compresses vocabulary-sized logit gradients.The method adds a second prediction and gradient pathway that reduces these bottlenecks.Source: §2.1 · pp. 2–3

2.2 COPY-PASTE LANGUAGE MODELS

  • The Copy-Paste head provides a second next-token pathway without a fixed vocabulary target space or prediction anchored to a static weight matrix.It scores past context positions and marginalizes positions emitting the same token to form a distribution supported only on tokens in context.Source: §2.2 · p. 3
  • The LM head’s special <copy> probability gates the CP distribution, while a sink lets the CP head defer to the LM prediction when no context position is suitable.The effective gate combines opening the copy pathway with the probability of not falling back to the sink.Source: §2.2 · p. 3
  • CPLMs use standard next-token cross-entropy on the mixed prediction, with no auxiliary loss or copy supervision.The model learns when and where to copy; the <copy> token acts as a gate rather than an actual prediction target.Source: §2.2 · p. 3
  • For w = 256, the FineWeb-Edu validation-loss curves for Full and Hybrid attention are essentially superposed before learning-rate decay.Figure 2 distinguishes backbone attention setups by color and plots loss against training tokens.Source: Figure 2 · p. 4

3 EXPERIMENTS & RESULTS

Across pretraining and downstream evaluations, CPLMs improve efficiency and performance across model setups. Their gains extend to long-context use and retrieval, including settings where the backbone has limited document access.

3.1 PRETRAINING

  • 1.25× faster in Full and Hybrid setups and 2× faster in Sliding Window at 1.7B, CPLMs reach vanilla models’ final loss sooner.The 1.7B Sliding Window CPLM also matches the Full LM with 12× less memory at 4k tokens.Source: §3.1 · p. 4
  • +0.4–0.9 accuracy points over vanilla models on standard benchmarks, with the largest gains on LAMBADA (+2.5–4.0 points).Token-level NLL also decreases for tokens that receive near-zero <copy> probability.Source: §3.1 · pp. 4–5
  • Across window sizes, CPLMs lower perplexity by 0.5–2 points on FineWeb-Edu and up to 15 points on other datasets, improving the memory–performance frontier.A small-window CPLM generally matches a full-attention vanilla model, while using less memory.Source: §3.1 · p. 5, Figure 3 · p. 5
  • At 16k tokens, CPLM perplexity reaches 6.5–7 as context grows, while vanilla models worsen after roughly 1k positions.The Sliding Window CPLM copies tokens more than 2,000 positions back despite a 256-token window.Source: §3.1 · pp. 5–6

3.2 SUPERVISED FINE-TUNING (SFT)

  • CPLMs outperform vanilla models by 1.2–5.1 average-score points after SFT, except on MBPP for the Hybrid model.On mathematical tasks, gains average +3.9 points for Full and +7.9 for Sliding Window models.Source: §3.2 · p. 6

3.3 LONG-CONTEXT ABILITIES

  • +7.5 average points on synthetic long-context tasks and +4.1 on LongBench: base CPLMs outperform vanilla counterparts.For single-needle retrieval, the Sliding Window CPLM scores 99.4 versus 14.3 for Vanilla across the depth/length grid, with a 256-token window.Source: §3.3 · p. 7
  • CPLMs outperform Vanilla on single-needle retrieval across architectures, including contexts up to 64k tokens.The Sliding Window CPLM’s gain is 14.3 → 99.4 averaged over the depth/length grid.Source: §3.3 · p. 7

3.4 EFFICIENT RETRIEVAL

  • With documents hidden from the backbone, CPLMs retain 90–97% of accuracy at five distractors and reach 34–38% EM with 500, using a 50× smaller cache than Full.The last-layer control, with twice the CP-cache memory, remains at 14% EM.Source: §3.4 · p. 7

4 ANALYSIS

The CP head provides an alternative gradient route that loses less signal than the LM head, and CPLM backbones copy effectively with fewer induction heads. The experiments link these findings to bottlenecked setups and copying behavior.

4.1 CPLMS MITIGATE THE LM HEAD BOTTLENECK

  • Up to 85% of gradient norm is conserved by the CP-head keys pathway at early positions, while LM heads lose over 99%.The CP head’s logits span context positions rather than vocabulary items, giving its keys a smaller shape when context length is much less than vocabulary size.Source: §4.1 · p. 8
  • At 512k vocabulary, vanilla models reach 34% accuracy on SpamLang, while CPLMs solve the task at every tested vocabulary size.Blocking CP-head gradients to the backbone removes most of the benefits; byte-level CPLMs do not improve on vanilla counterparts.Source: §4.1 · p. 8

4.2 CPLM BACKBONES LEARN FEWER INDUCTION HEADS

  • At T=2048, Full vanilla has NLL 13.8 versus a uniform-guess ln |V| of 10.8, while Full and Hybrid CPLMs predict repeated tokens almost perfectly at every tested length.At T=128, CPLM backbones have 4 induction heads versus 13–15 for vanilla models.Source: §4.2 · p. 9
  • CPLM backbones usually concentrate induction in one sharp head; the CP head takes over copying work vanilla models spread across many heads.The probe uses a random sequence repeated once and counts heads whose induction score exceeds 0.3.Source: §4.2 · pp. 8–9

5 RELATED WORK

The related work situates CPLMs among softmax-bottleneck remedies, pointer and copy mechanisms, and external-memory approaches. It distinguishes the CP head by its training-time gradient pathway and learned token-level memory.

Softmax Bottleneck and Mitigations

  • Prior softmax-bottleneck remedies target forward expressivity, whereas the CP head also addresses the backward bottleneck with a second gradient pathway.The cited remedies include mixtures of LM heads and element-wise nonlinearities.Source: §5 · p. 9

Pointer and Copy Mechanisms

  • Earlier pointer-copy systems combined pointers with vocabulary prediction through switches, gates, shared normalization, or sentinels, mostly in recurrent or encoder-decoder models.Decoder-only pointers were often considered redundant because self-attention can retrieve previous tokens.Source: §5 · pp. 9–10
  • Unlike the cited CPR approach, CPLMs use <copy>-<sink> gating rather than forced copying; unlike local-memory matching, the CP head points to past tokens with learned query and key projections.The CP head stores one vector per token in a dedicated learned key space.Source: §5 · p. 10

6 CONCLUSION

CPLMs add a lightweight pointer over past positions to the LM head, providing a less-compressed prediction and gradient pathway. They improve training and model performance across tested backbones, with benefits and limits tied to copying tokens already in context.

6 CONCLUSION

  • CPLMs train faster and improve perplexity, downstream performance, supervised fine-tuning, and long-context results across every tested backbone.The CP head adds negligible cost, and sliding-window models can match full attention with a fraction of the memory.Source: §6 · p. 10
  • Sliding-window CPLMs match full-attention performance with a much smaller memory footprint.The CP head adds a second, less-compressed prediction and gradient pathway to the LM head.Source: §6 · p. 10
  • The CP head relaxes the LM-head bottleneck only for tokens present in context, and experiments extend only to 1.7B parameters.The paper identifies span copying, external corpora, and larger-scale studies as next steps.Source: §6 · p. 10

AI USE STATEMENT

The authors describe their use of generative AI and report that models were trained and evaluated on public data and benchmarks. They also identify deployment and reproduction risks, and specify the context-scaling choices used for different backbones.

AI USE STATEMENT

  • Generative AI assisted literature search, drafting, editing, proofreading, formatting, and implementation of standard evaluation and plotting scripts; authors reviewed the assisted work.The authors state that AI tools did not generate research ideas, design experiments, or produce results, and that reported numbers come from their own runs.Source: AI USE STATEMENT · pp. 10–11

ETHICS STATEMENT

  • The small research models are not safety-aligned and should not be deployed in user-facing applications without further evaluation.The authors also note possible reproduction of web-data biases or factual errors and more faithful reproduction of sensitive or copyrighted input content.Source: ETHICS STATEMENT · p. 11

A PRETRAINING HYPERPARAMETERS

  • Full-attention and hybrid models use YaRN with ×4 scaling from 4,096 tokens; sliding-window models retain plain RoPE because they never see positions beyond 256 tokens.For full and hybrid models, YaRN is applied at inference for base models and during fine-tuning for SFT models.Source: A PRETRAINING HYPERPARAMETERS · p. 18, Table 4 · p. 18

B LATENCY AND MEMORY

The cost study finds small training and inference overheads for CPLMs. Additional experiments report copy-related gains across seeds, dimensions, scale, sink settings, and recurrent backbones, while noting a parameter-count confound for cross-mixer comparisons.

B LATENCY AND MEMORY

C.1 MULTI-SEED ROBUSTNESS

  • CPLMs lower perplexity for every backbone, with gains roughly 15–40× the seed standard deviation and one-sided paired-test p < 10^-3 in all three cases.The comparison repeats 135M/20B runs with three seeds.Source: §C.1 · pp. 19–20, Table 7 · p. 20

C.2 COPY DIMENSION

  • Perplexity stays within 0.02 points across copy dimensions 64–2,048; dimension 64 stores 9× fewer values per token than default D = 576.Reducing window size to reach the same memory budget increases perplexity by up to 0.2 points.Source: §C.2 · p. 20, Figure 8 · p. 20

C.3 360M SCALE

  • At 360M, CPLMs improve over vanilla models for every backbone, and the sliding-window CPLM outperforms vanilla Full and Hybrid models.The full and hybrid training curves are essentially superposed.Source: §C.3 · p. 20, Figure 9 · p. 21

C.4 SINK AND WINDOW SIZE

  • The sink lowers perplexity by about 0.3 points at a 32-token window and 0.07 points at 256 tokens; its benefit vanishes at windows of 512 or more and with full attention.The authors hypothesize that the sink is more valuable with small windows because the backbone has less information about whether the next token is in context.Source: §C.4 · p. 20, Figure 10 · p. 21

C.5 GATED DELTANET PRETRAINING

  • For GDN and Hybrid-GDN, CPLMs lower perplexity on every dataset, by about 0.5 points in-domain and 1–6 points out-of-distribution.The fully recurrent GDN CPLM outperforms vanilla Hybrid-GDN on out-of-distribution data while using several times less memory; these mixer comparisons are not parameter-matched because GDN models carry about 20% more parameters.Source: §C.5 · p. 21

C.6 CP HEAD ABLATIONS

The ablations compare ways to combine a context pointer with the LM head, and the paper also tests whether an inference-time neural cache can reproduce CPLM gains. The copy-gated mixture performs comparably to pointer alternatives, while the tuned cache barely changes vanilla-model perplexity.

C.6 CP HEAD ABLATIONS

  • The copy-gated mixture with a sink matches or slightly improves on other pointer variants; partitions remain behind pointer variants, and margins between pointer methods are small.The authors use the gate and sink for exact two-stage sampling and to expose copying decisions for analysis, rather than for performance gains.Source: §C.6 · p. 23, Table 8 · p. 23

C.7 COMPARISON WITH A NEURAL CACHE

  • The tuned neural cache barely changes vanilla-model perplexity, while CPLMs lower it by 0.11 (Full), 0.14 (Hybrid), and 0.41 (SW).The best cache mixture weight is γ ≤ 0.002 for all three backbones; the authors attribute CPLM gains to training the copy mechanism jointly with the LM.Source: §C.7 · p. 24, Table 9 · p. 24

D SFT EVALUATION

The section reports supervised fine-tuning evaluations across language, reasoning, coding, and instruction-following tasks, with GSM8K pass@k comparisons across model backbones.

D SFT EVALUATION

  • SFT sequences reach 16k tokens; evaluations use the fine-tuning chat template except MATH, which uses its standard plain prompt.Full and Hybrid use YaRN RoPE scaling (factor 4), while SW-256 uses standard RoPE.Source: D SFT EVALUATION · p. 24
  • MATH evaluation covers 861 level 1–2 problems from four subjects, using Minerva 4-shot prompts and pass@4 over eight samples.The paper excludes harder levels as out of reach for models of this size.Source: D SFT EVALUATION · p. 24
  • The section also evaluates 5-shot GSM8K, zero-shot HumanEval and MBPP, and zero-shot IFEval using the stated sampling and scoring protocols.GSM8K reports pass@4 over eight samples; HumanEval and MBPP report pass@10 over 20 samples, while IFEval averages prompt- and instruction-level accuracy under strict and loose criteria.Source: D SFT EVALUATION · p. 24
  • CPLMs outperform vanilla models at every GSM8K pass@k from 1 to 8, with SW gains reaching 7.7 points at pass@8.At pass@1, gains range from 1.6 to 3.6 points; the Full CPLM stays around 4 points across k.Source: D SFT EVALUATION · p. 25, Figure 12 · p. 25

E NEEDLE-IN-A-HAYSTACK

The needle-in-a-haystack evaluation tests retrieval across context lengths and needle depths. CPLM gains are strongest when the backbone cannot attend across the full context.

E NEEDLE-IN-A-HAYSTACK

  • With CPLM, SW-256 reaches 98–100% accuracy in almost every cell through 64k, whereas vanilla finds needles only in its last 256 tokens.Vanilla accuracy is stuck at 1/8 of depths for contexts of at least 2k; CPLM has only small drops at depth 0 and 1k.Source: E NEEDLE-IN-A-HAYSTACK · p. 26
  • Hybrid CPLM retains 60–84% accuracy per length from 16k to 64k, while vanilla Hybrid loses retrieval for earlier needles beyond 8k.The shallowest needles are hardest for Hybrid CPLM; vanilla average accuracy per length falls from 64 at 8k to 12 at 64k.Source: E NEEDLE-IN-A-HAYSTACK · p. 26
  • Full CPLM improves retrieval at 8k from 48 to 84, but neither Full model retrieves successfully from 16k onward.CPLM recovers many of vanilla’s weak shallow-to-middle depth rows at 8k, but does not shift the failure point.Source: E NEEDLE-IN-A-HAYSTACK · p. 26
  • The results indicate CPLM helps most when backbone attention cannot cover the full context, preserving long-range retrieval beyond its window or training length.The setup inserts a random seven-digit needle at varying relative depths across 1k–64k prompts, with 200 examples per depth-length cell.Source: E NEEDLE-IN-A-HAYSTACK · pp. 25–26

F LONG-CONTEXT EVALUATION

Long-context evaluations show CPLM gains across several backbones and retrieval settings, alongside improved prediction and document copying. The results also examine language transfer and the gradient-path explanation for these gains.

F LONG-CONTEXT EVALUATION

  • CPLMs improve TREC by +16.0, +29.0, and +24.5 points for Full, SW, and Hybrid base models, respectively; abstractive-task gains vary by backbone.On the three summarization tasks, SW and Hybrid improve on every task by +3.6 to +6.2 points, while Full trails vanilla on GovReport and QMSum.Source: F LONG-CONTEXT EVALUATION · p. 26, Table 10 · p. 27
  • After SFT, CPLMs raise average LongBench scores by +0.6, +2.7, and +1.9 points for Full, SW, and Hybrid, respectively.Gains are smaller and less uniform across tasks; every backbone improves on GovReport and MultiNews, while SW loses ground on SAMSum and code completion.Source: F LONG-CONTEXT EVALUATION · p. 26, Table 11 · p. 27
  • SW CPLM RULER accuracy rises from 10.2 to 36.1 at 4k and from 5.0 to 18.9 at 64k, while both SW models remain near the BABILong floor beyond 2k.Full models fail beyond 16k; Hybrid CPLM retains a RULER advantage at 32k.Source: F LONG-CONTEXT EVALUATION · p. 26, Table 12 · p. 28
  • At 4k, Full CPLM improves BABILong average accuracy from 18.6 to 32.6, while the hardest multi-key RULER tasks remain near the floor for all models.Across per-task results, CPLM gains vary by backbone and task; Hybrid is on par with vanilla on BABILong average and both SW models remain near the floor.Source: F LONG-CONTEXT EVALUATION · p. 26, Table 13 · p. 28, Table 14 · p. 29

G.1 COPY PROBABILITY AND PREDICTION QUALITY

  • CPLMs achieve lower NLL than vanilla in every copy-probability bin, including near-zero copy probability, with gains generally growing as copy probability rises.Top-1 gains are largest for likely-copied tokens, especially for SW; Full and Hybrid are on par with vanilla at intermediate copy probabilities.Source: §G.1 · pp. 27–28, Figure 14 · p. 27

G.2 COPY DISTANCE

  • CP-head copy distance grows almost linearly with document position, and SW CPLM copies from more than 2,000 tokens back at 16k despite its 256-token backbone window.The distance analysis uses 5,000 FinePDFs documents of 16k tokens.Source: §G.2 · pp. 28–29, Figure 15 · p. 28

G.3 OTHER LANGUAGES

  • For the 135M Full models, CPLM reduces perplexity by 14%–22% relative on non-English languages, with the largest relative gain on Mandarin Chinese.The models were pretrained only on English; Italian improves by 4.4 points and English FineWeb by 0.3 points.Source: §G.3 · p. 29, Figure 16 · p. 29

H WHERE DOES THE GAIN COME FROM?

  • The gain analysis tests the gradient-path account by stopping CP-head gradients into the backbone and by removing the LM-head bottleneck with byte-level models.The interventions use 135M Full-attention models; runs in this section use a single seed.Source: H WHERE DOES THE GAIN COME FROM? · p. 30

H.1 STOPPING THE COPY GRADIENT INTO THE BACKBONE

Stopping CP-head gradients into the backbone removes most CPLM gains, while byte-level models with a full-rank LM head show no meaningful gain. Together, the interventions support the proposed role of the LM-head gradient bottleneck.

H.1 STOPPING THE COPY GRADIENT INTO THE BACKBONE

  • The ablation detaches hidden states used for CP queries and keys, cutting that gradient path while leaving the LM-head path and CP-head projections trainable.The CP head’s standalone loss sends no gradient into the backbone with stop-gradient in place.Source: §H.1 · p. 30
  • Stopping CP-head gradients leaves a 0.008-nat (0.12-PPL) gain over vanilla, about 25% of the full CPLM’s 0.031-nat (0.49-PPL) improvement.The residual gain remains several times larger than the reported seed-to-seed standard deviation.Source: §H.1 · p. 30, Figure 17 · p. 30, Table 15 · p. 30
  • The stop-gradient model’s copy probability falls to 0.046 versus the CPLM’s stable 0.21, and its effective gate is 0.036 versus 0.091.Its sink probability is 0.22 versus 0.57 for CPLM; the paper reports the stop-gradient model mostly turns off the CP head.Source: Figure 17 · p. 30, §H.1 · p. 31
  • The authors attribute most CPLM gains to the CP gradient’s richer training signal to the backbone, rather than to the CP head as an additional output layer.They connect this to better context-feature representations, fewer induction heads, and improved NLL even on tokens not copied.Source: §H.1 · p. 31

H.2 BYTE-LEVEL LANGUAGE MODELING

  • The byte-level setting has vocabulary size 257 below hidden dimension 576, making the LM head full row rank; the authors say copying then adds nothing it cannot express.The byte models copy at least as much as token-level models without improving loss, consistent with the bottleneck account.Source: §H.2 · pp. 31–32
  • Byte-level CPLMs finish within 0.002 nats per byte of vanilla, despite maintaining high copy probability throughout training.The reported differences are +0.0010 with the sink and +0.0016 without it; the sink helps early, but neither variant is meaningfully ahead overall.Source: §H.2 · p. 31, Table 16 · p. 31, Figure 18 · p. 32

Related papers

Loading 2610.19247v1…