Source-linked AI summary
Mirror, Mirror on the Wall: Prompt Echoing in Small Instruct Language Models
Inez Okulska, Bartosz Naskręcki, Jan Piotrowski, Tomasz Steifer
TL;DR
Prompt echoing raises the question of whether echoes reflect training-data overlap or internal copying mechanisms. The study tests these explanations in small instruct models and finds that overlap is present but does not fully explain the behavior. Once an echo begins, internal copying mechanisms primarily drive its continuation.
Problem
The study asks whether prompt echoing in small instruct models reflects training-data overlap or internal copying mechanisms.
Source: §1.1 · p. 3Method
The authors scan public training data for prompt overlap, force echo prefixes on non-echogenic prompts, and ablate candidate induction heads.
Source: §1.1 · p. 3, §2 · p. 5Results
Echo-prone prompts show partial training-data overlap, but forced-prefix and ablation experiments support internal copying as the primary driver once echoing starts.
Source: §2 · p. 5Takeaways & Limitations
Prompt echoing appears to involve interacting mechanisms: memorization influences it, but internal copying primarily sustains it after initiation.
Source: §2 · p. 5Overlap analysis covers only two models, and short prefix matches do not establish verbatim memorization or training-data leakage.
Source: Limitations · p. 6
Abstract
from arXiv · showhide
Prompt echoing is a recognized failure mode of instruct language models, in which a model instead of generating a response, mirrors the provided prompt, even though it did not receive a specific instruction to do so. Is this phenomenon a sign of the model leaking the content of its training dataset, or is it rather caused by a misaligned behavior of the internal induction/copying mechanisms? We investigate prompt echoing small language models from different families (Gemma, Llama, Qwen, SmolLM and OLMo) and show that echoing prompts are likely to have partial overlap with the training dataset but the phenomenon is primarily driven by the model's induction heads.
1 Experimental setup
The experiments use echoing and non-echoing prompt sets to test when small instruct models echo and whether overlap or internal copying mechanisms contribute. They vary prompt construction, decoding, forced echo prefixes, and attention-head interventions.
1.1 Data preparation.
- Dataset A contains 384 self-generated prompts screened for echoing, while Dataset B contains 384 prompts built from ten-step recursive response-to-prompt chains.Dataset A used 64 accepted prompts from each of six source models; Dataset B used 64 chains per model.Source: §1.1 · pp. 2–3
- Dataset C provides 128 reference prompts accepted only when none of the screened completions produced a prefix echo.Source: §1.1 · p. 3
- Experiment 1 evaluates six instruct-model families and selected quantized models on Datasets A and B, sampling each pair 20 times at T=1.0 and once at T=0.3 and T=0.0.Full echoes begin with the complete prompt; partial echoes begin with at least 40% of its tokens, using a strict prefix-only criterion.Source: §1.1 · p. 3, Table 1 · p. 4
- Experiment 2 checks whether the first five words of echo-prone prompts occur contiguously in public SmolTalk or Tulu-3 SFT data, treating this as a weak overlap test.Source: §1.1 · p. 3
- Experiment 3 forces 2, 3, or 5 initial prompt tokens on non-echogenic prompts before sampled continuation; Experiment 4 intervenes on ranked copying heads during echo evaluation.Experiment 4 compares no intervention, ablation of the top 8 or 16 copying heads, and ablation of 16 random heads; ablation zeros selected attention-output slices.Source: §1.1 · pp. 3–5
2 Results and interpretation
Across small instruct models, prompt echoing reflects both training-text overlap and internal copying mechanisms. The experiments find that copying mechanisms primarily drive continuation once an echo begins.
2 Results and interpretation
- Echoing occurs across all six model families, including at T=1.0; quantization raises SmolLM2 from 1.95% to 5.19% but lowers OLMo-2 1B from 0.61% to 0.38%.For SmolLM2 Q4_K_M, the any-echo rate reaches 15.36% at T=0.0.Source: §2 · p. 5
- Training-data matches occur for 32.63% of OLMo echo prompts versus 9.38% of non-echogenic prompts, and 26.33% versus 17.97% for SmolLM prompts.The paper says this pattern suggests memorization does not explain echoing well.Source: §2 · p. 5
- Forcing just 2 initial prompt tokens significantly increases partial and full echoes on prompts selected not to echo spontaneously.For Gemma 3n E4B IT, forcing 5 tokens raises the echo rate to almost 40%, even at high temperature.Source: §2 · p. 5
- Once initialized, prompt echoing is primarily driven by internal copying; candidate induction-head ablations reduce echo more than random-head ablations, with larger drops after ablating 16 rather than 8 heads.The conclusion characterizes memorization as influential but insufficient to explain echoing fully.Source: §2 · p. 5
Limitations
The study measures a narrow, prefix-based form of prompt echoing on diagnostic prompt distributions, not deployment prevalence. Its scope and analyses are limited, and the strict definition excludes several related behaviors.
Limitations
- Echo rates measure prefix-based prompt echoing on diagnostic prompt distributions, not deployment prevalence.The strict token-prefix definition excludes copying later prompt spans, paraphrasing, quoting within otherwise normal answers, and diffuse repetition.Source: Limitations · pp. 5–6
- Training-data overlap analysis covers only two models, and short prefix matches do not establish verbatim memorization or training-data leakage.Source: Limitations · p. 6
- Head ablations test causal involvement of high prefix-matching-score heads, but are coarse and do not fully identify the underlying circuit.Source: Limitations · p. 6
- Experiments cover only a limited set of English, one-shot prompts and small instruct models, so conclusions may not transfer to larger models, multilingual settings, long contexts, or multi-turn interactions.Source: Limitations · p. 6