PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Tip of the Tongue: First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs

Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim

arXiv:2610.23400v1cs.CL

TL;DR

Closed-book factual errors are often attributed to missing knowledge, though models may encode answers they fail to produce. The paper probes these errors by cueing the gold answer’s first token, finding that many are recoverable first-token recall failures. The cue recovers errors across models and languages, but supplying it requires the gold answer, so the probe is diagnostic rather than deployable.

  • Problem

    Although models encode factual knowledge, they still make closed-book errors, and whether supplying the correct answer start resolves them has not been tested.

    Source: §1 · pp. 1–2

  • Method

    First-Token Cueing prefills the response with the gold answer’s first token and tests whether the model can complete it.

    Source: §1 · p. 2

  • Results

    Supplying the first token corrects a large share of multi-token answer errors across 16 open models and improves accuracy in every tested language.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Many closed-book errors are recoverable first-token recall failures, locating a class of errors at the first decoding step.

    Source: §1 · p. 2, §7 · p. 8

    Because the cue uses the gold answer, First-Token Cueing is a diagnostic rather than a deployable decoding method.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models often answer factual questions incorrectly in closed-book settings, and such errors are usually attributed to missing knowledge. Yet models often encode answers they fail to produce. We show that many closed-book errors are first-token recall failures: the model commits to a wrong first token, but once given the right one it completes the answer and often recognizes it as correct. We study this with First-Token Cueing, a behavioral probe that prefills the response with the first token of the gold answer. Model uncertainty peaks at the first answer token, where it already predicts correctness, and among gold prefixes of any length the first token alone accounts for the largest share of recovered errors. Across 16 open models, this one token corrects between a fifth and a half of the errors on multi-token answers in English QA and improves accuracy in all seven languages of a multilingual benchmark. Most of the errors it corrects are also solved by the model's own sampled reasoning, a criterion for encoded knowledge, and models recognize correct starts they do not produce: they continue a correct start more confidently than an incorrect start of the same answer type and prior probability, yet rarely find it without the gold answer. These results locate a recoverable class of factual errors at the first decoding step, where inference-time methods may gain the most.

1 Introduction

Closed-book factual errors are often treated as missing knowledge, although models may encode answers they fail to produce; where these failures arise during generation remains unclear. The paper probes this question with a gold-first-token cue and argues that many errors are recoverable first-token recall failures.

1 Introduction

  • First-Token Cueing supplies the gold answer’s first token and tests whether the model can complete answers it initially fails to produce.The human tip-of-the-tongue analogy motivates the probe, but the paper does not claim a shared mechanism.Source: §1 · p. 2
  • Many closed-book errors are first-token recall failures: models commit to a wrong start, then often complete and recognize the answer when given its correct start.The paper reports that the cue corrects a substantial share of errors across 16 open models and seven languages.Source: §1 · p. 2
  • Uncertainty about correctness concentrates at the first answer token, which accounts for the largest share of errors recovered by gold prefixes.Source: §1 · p. 2
  • Models can recognize correct answer starts they fail to produce, but rarely find those starts on their own.With correct starts outside their top candidates, they continue more confidently than incorrect starts matched for answer type and prior probability.Source: §1 · p. 2

2 Related Work

Prior work examines hidden knowledge, the role of early tokens in generation, and human tip-of-the-tongue states. This paper connects these lines through a behavioral probe of whether a first-token cue recovers closed-book answers.

2.1 Recall Failures and Hidden Knowledge in LLMs

  • Probes and inference-time interventions have revealed correct answers that models fail to produce; this paper instead tests recall behavior by supplying only the answer’s first token.A crossed-cue control compares correct starts with incorrect starts of the same answer type and prior probability.Source: §2.1 · p. 2

2.2 The First Token in Autoregressive Generation

  • Prior studies treat first-token uncertainty or branching as a signal or search strategy, while this work tests whether intervening on the first token resolves factual errors.Generic output prefixes carry no answer information and do not help in the paper’s open-ended setting.Source: §2.2 · p. 3

2.3 Tip-of-the-Tongue States in Human Memory

  • The paper adopts human onset cueing as a behavioral analogy, while noting that a token can span several characters or a whole word.Feeling-of-knowing judgments provide a human counterpart to the recognition tested in the paper.Source: §2.3 · p. 3

3 Uncertainty Concentrates at the First Token

The section tests whether correctness uncertainty concentrates at the first answer token. It finds that first-token entropy predicts correctness nearly as well as whole-answer entropy, with later positions carrying a weaker signal.

3 Uncertainty Concentrates at the First Token

  • The analysis measures next-token entropy at answer positions and compares the first three positions with the whole-answer average using AUROC and PRR.PRR compares accuracy gained by rejecting the most uncertain answers with an oracle; 1 denotes oracle performance and 0 random performance.Source: §3 · p. 3
  • First-token entropy alone separates correct from incorrect answers nearly as well as the whole-answer average, while second- and third-token entropy do much worse.Entropy is highest at the first token and drops sharply afterward.Source: §3 · p. 3, Figure 2 · p. 3
  • For all 16 models, first-token entropy predicts baseline correctness with AUROC between about 0.6 and 0.85 on the English benchmarks.The first-step distribution also predicts cue recovery: the cue fixes about two-thirds of errors when the gold first token ranked second to fifth, and under a third when it ranked below the top thousand.Source: §3 · p. 3

4 First-Token Cueing

First-Token Cueing prefills a response with the gold answer’s first token and lets the model continue greedily. The method uses the model’s tokenization and is an oracle diagnostic rather than a deployable decoding method.

4 First-Token Cueing

  • First-Token Cueing appends the gold answer’s first token to the prompt, then greedily generates the continuation for evaluation.A k-token cue instead prefills the first k gold tokens; the baseline uses the same prompt without a cue.Source: §4 · p. 4
  • The cue is a token because it is the model’s first-step commitment unit, and the model’s own tokenization keeps the input in its expected form.A single-character prefill can derail the answer; aliases are used only for grading.Source: §4 · p. 4
  • Because the cue comes from the gold answer, First-Token Cueing is an oracle intervention that probes recoverable errors but cannot be deployed directly.Cue correction alone does not establish that the fact is encoded, so the paper tests recall failures separately.Source: §4 · p. 4

5 The First Token Recovers Many Errors

First-Token Cueing recovers many closed-book QA errors across English and multilingual benchmarks, with the largest share of prefix-recovered errors fixed by the first token alone. Its gains vary with language and tokenizer, but remain positive across the evaluated answer lengths and languages.

5.1 Experimental Setup

  • The study evaluates 16 instruction-tuned models from Gemma 3, Llama 3.1/3.2, and Qwen 2.5 on three English QA benchmarks and seven-language MKQA.Headline results use answers of at least two gold tokens, excluding about a tenth of English questions and fewer MKQA questions; responses are greedily decoded and judge-graded against gold answers and aliases.Source: §5.1 · p. 4

5.2 English QA

  • On multi-token English answers, First-Token Cueing corrects between a fifth and a half of Baseline errors, and every model improves on the three benchmarks.Accuracy improves for every model; the cue’s identity matters, since random or plausible-but-incorrect tokens lower accuracy and a generic prefix leaves it unchanged.Source: §5.2 · p. 4, Figure 3 · p. 5
  • Across model scale, the cue corrects 40–50% of Baseline errors in every model except three with the lowest Baseline accuracy.Absolute gains are roughly 15–25 points for the smallest family members and 10–13 points for the largest; the share of errors recovered stays similar despite scale reducing errors.Source: §5.2 · pp. 4–5
  • Cueing improves accuracy at every gold-answer length, with gains shrinking as answers grow but staying positive for answers of five or more tokens.Across the English benchmarks, gains also grow with first-token character length under all three tokenizers and remain positive for single-character tokens.Source: §5.2 · p. 5, §5.3 · p. 6

5.3 Multilingual QA

  • Cueing improves accuracy for every model in all seven MKQA languages, with the smallest gains in Arabic and Russian and the largest in English and Spanish.Gains roughly track Baseline accuracy, with Russian the clearest exception; in non-Latin scripts, first tokens are often single characters while answers span more tokens, so the cue covers less of the answer.Source: §5.3 · pp. 5–6, Figure 4 · p. 5, Figure 5 · p. 6

5.4 The First Token Accounts for the Largest Share of Prefix Recovery

  • For Llama-3.1 8B, the first gold token alone recovers more Baseline errors than all longer prefixes combined, including among long answers.The first token is the largest recovery group at every answer length except for the three models with the lowest Baseline accuracy, which need longer prefixes for long answers.Source: §5.4 · p. 6, Figure 6 · p. 6

6 Models Recognize Correct Starts They Do Not Produce

The paper tests whether errors recovered by First-Token Cueing reflect answers models can reach or recognize, rather than only information supplied by the cue. Sampled reasoning often recovers the same errors, and matched-start tests show models favor correct starts in confidence even though they rarely find them unaided.

6.1 Oracles over the Model’s Own Outputs

  • On NOT-IN errors, First-Token Cueing recovers about two-fifths, versus about a quarter for an oracle choosing among the model’s top-p first-token candidates.About two-fifths of Baseline errors in Llama-3.1 8B and 70B are IN, where the model proposed a prefix of the correct first token and the cue recovers about half; on NOT-IN errors, cueing’s excess over the oracle persists even with a single candidate.Source: §6.1 · pp. 6–7, Table 1 · p. 7
  • Sampled chain-of-thought pass@16 recovers 77% and 62% of cue-recoverable errors for Llama-3.1 8B and 70B, respectively, supporting that many are encoded but unrecalled.Pass@16 exceeds cueing by a few points from 1B upward in Gemma-3, while direct-answer and greedy chain-of-thought baselines remain below cueing; pass@k shows reachability, not selection ability.Source: §6.1 · p. 7, Figure 7 · p. 7

6.2 Crossed-Cue Control

  • The crossed-cue control pairs NOT-IN questions of the same answer type and similar start-token probability, assigning each question’s correct start to the other question as an incorrect cue.Either cue often changes the Baseline answer, and incorrect cues almost never yield a correct answer, so the test compares continuation confidence rather than answer changes.Source: §6.2 · pp. 7–8
  • With answer type and start-token prior matched, models continue the correct start more confidently on about 60–67% of questions, well above chance.The preference weakens to about chance when the incorrect start is far more probable; choosing by confidence recovers about one-third to two-thirds of the gain from the correct cue alone.Source: Figure 8 · p. 7, §6.2 · p. 8, Table 2 · p. 8
  • Models often recognize a correct start they did not consider, but for Llama-3.1 8B, searching 36 model-scored first characters gains about as much as beam search despite two-thirds of questions having a correct candidate.The correct start is continued more confidently than most alternatives but usually less confidently than the model’s own start.Source: §6.2 · p. 8

7 Conclusion

Across 16 open models, many closed-book errors are first-token recall failures: models often complete and recognize answers once given the correct start. Cueing recovers many multi-token-answer errors, and most corrected errors are also solved by sampled reasoning.

7 Conclusion

  • First-token cueing corrects between a fifth and a half of multi-token-answer errors across 16 open models, in every tested benchmark and language.The recoveries are the largest share obtained by any gold-prefix length.Source: §7 · p. 8
  • Most cue-corrected errors are also solved by the model’s own sampled reasoning, while models continue correct starts more confidently than comparable incorrect starts.They rarely find the correct start without the gold answer.Source: §7 · p. 8
  • Inference-time methods may benefit from focusing computation on the first answer token, but recovering cueing’s gains without the gold answer remains open.The authors suggest external evidence, such as retrieval, to help choose among starts.Source: §7 · p. 8

Limitations

The study probes short-answer factual QA, where the gold answer and its first token are well defined. Extending the probe to long-form generation is proposed as future work.

Limitations

  • The study is limited to short-answer factual QA, where the gold answer and its first token provide a well-defined probe target.Long-form generation would require treating the first token of each factual claim as the counterpart.Source: Limitations · p. 9
  • First-Token Cueing is a diagnostic, not a decoding method, because it uses the gold answer to measure errors resolved by its first token.Recall failures are identified behaviorally through cue recovery and its overlap with reasoning-based recovery, rather than from internal states.Source: Limitations · p. 9
  • Whether retrieval or another source can provide the first token and recover the gains without the gold answer remains future work.The proposed long-form extension would apply cueing to the first token of each factual claim.Source: Limitations · p. 9

Ethical Considerations

The paper reports using public benchmarks and open-weight models without collecting personal data. Its appendix describes dataset handling, model inference, decoding and prompting conditions, and response judging.

Ethical Considerations

  • The authors use public benchmarks under their licenses and publicly released open-weight models, collect no personal data, and acknowledge AI assistance for code debugging and manuscript readability.The AI assistant was Claude Opus 5.5; the authors retain responsibility for the final content.Source: Ethical Considerations · p. 9

A.1 Datasets

  • For NQ-Open and WebQuestions, the first listed answer is gold and remaining answers are aliases; MKQA samples with empty target-language answers are filtered out.TriviaQA and MKQA use their provided alias lists directly.Source: §A.1 · p. 12

A.2 Models and Inference Backend

  • The evaluation uses 16 open-weight instruction-tuned models from the Gemma-3, Qwen-2.5, and Llama families.Models are served with vLLM; prompts use each model’s native chat template.Source: §A.2 · p. 12

A.3 Decoding Hyperparameters

  • The six decoding conditions vary prompt type and decoding policy; baseline and cueing share greedy direct-answer decoding and differ only by the prepended gold first token.Sampling conditions use temperature 0.7, top-p 0.95, and 16 generations per question; CoT conditions use a 1,024-token budget.Source: §A.3 · p. 12, Table 4 · p. 13

A.4 Prompt Templates

  • Direct-answer prompts use five demonstrations, while CoT prompts are zero-shot and require a final answer in a specified phrase.The short answer is extracted from that phrase; if parsing fails, the full output is sent to the judge.Source: §A.4 · p. 12

A.5 LLM-as-Judge

  • A single judge model, gpt-oss-120b, evaluates responses against gold answers and aliases, accepting semantically equivalent answers and certain extra content.The judge’s verdict is parsed by prioritizing an explicit “the answer is” true/false phrase, then variant keywords, then the final true/false occurrence.Source: §A.5 · pp. 12–13

A.6 Judge Agreement with Author-Checked Labels

An author-checked sample finds that the judge agrees with labels on most responses but rejects correct responses disproportionately, including cued responses. Reweighting estimates a larger average cueing gain under labels than under the judge.

A.6 Judge Agreement with Author-Checked Labels

  • +23.1 points under the labels versus +20.1 points under the judge: reweighted average cueing gain across languages.The estimated gain under labels has a 95% interval of +11.8 to +29.9 points.Source: Table 5 · p. 14, §A.6 · p. 14
  • 93.0% of judge decisions agree with author-checked labels (95% Wilson interval 88.6–95.8; Cohen’s κ = 0.86).Of 14 disagreements, 13 are correct responses the judge rejects.Source: §A.6 · p. 14
  • Agreement is 95.0% on Baseline responses and 91.0% on cued responses; the judge accepts no incorrect cued response and rejects nine correct cued responses.The nine rejections include all three correct cued responses with extra entities.Source: §A.6 · p. 14
  • Per-language gain estimates are noisy because each verdict stratum contains only 12 or 13 responses, so one flipped label shifts a cell by about seven points.The judge underestimates accuracy by 6.3 points for Baseline and 9.3 points for cueing, on average across languages.Source: §A.6 · p. 14

B Complete First-Token Cueing Results

First-Token Cueing improves closed-book QA across models, tasks, and answer lengths, with gains concentrated in early prefixes. Random-token controls show that improvement depends on matching the gold answer.

B Complete First-Token Cueing Results

B.1 Per-Model Accuracy by Gold Answer Length

  • Cueing beats Baseline in all 80 model-by-length cells, with gains generally shrinking as answer length grows.Gains range from +12pp to +70pp for one-token answers and +2pp to +12pp for answers of five or more tokens.Source: §B.1 · p. 15, Figure 9 · p. 16

B.2 Per-Model Minimum-k Distribution

  • The first-token prefix is the largest recovered-error group in 13 of 16 models, while 81–85% of errors in the largest models are recoverable by some prefix.In three lowest-accuracy models, longer prefixes can be the largest group; longer-prefix recovery alone is not evidence of recall.Source: §B.2 · pp. 16–17, Figure 10 · p. 17

C Random-Token Control

  • Random-token prefilling lowers accuracy in all 48 model-task cells, by 22.7pp on average, indicating that the gold-token match matters.The control uses the same conditioning context, decoding policy, and evaluation protocol as First-Token Cueing.Source: C Random-Token Control · p. 17, Table 8 · p. 18

D Lexical Metrics: Exact Match and Token-Level F1

Additional analyses find that cueing gains persist across lexical metrics and languages, depend on cue content, and overlap with failures recoverable by the model’s own reasoning. The analyses also examine cue strength, matched-start recognition, and gold-free search.

D Lexical Metrics: Exact Match and Token-Level F1

E.1 Oracle under Exact Match

  • Under strict exact match, cueing remains below the oracle on IN questions but exceeds it on NOT-IN questions, even after excluding single-token answers.On NOT-IN questions with one candidate, cueing still exceeds the oracle by +5.3 and +1.9pp.Source: §E.1 · pp. 18–19, Table 11 · p. 19, §E.2 · p. 19

E.2 Candidate-Set Size; E.3 Prefix Ambiguity

  • On NOT-IN questions, cueing’s excess over the oracle persists for single-candidate cues and peaks at intermediate prefix ambiguity rather than unique prefixes.Prefixes shared by 2–20 answers exceed the oracle by +10 to +16pp; unique prefixes exceed it by +8pp.Source: §E.2 · p. 19, §E.3 · p. 19, Table 12 · p. 19

E.4 Cue Controls

  • A plausible same-type competitor of the gold answer lowers judge accuracy 7–15pp below Baseline, while the gold token raises it 9–23pp on the same questions, showing cue content matters.On open-ended QA, generic output prefilling yields no meaningful gain (+0.8 / +0.7 / −0.7pp), while the gold first token gains +13.0 / +20.8 / +12.7pp under the same judge.Source: §E.4 · p. 20

E.5 Relation to Hidden Knowledge: Thinking-Recoverable Failures

  • Cueing recovers about 60% of thinking-recoverable failures versus 27–34% of other failures, showing overlap with failures recovered by sampled reasoning.For multi-token answers, cueing recovery is 1.8–2.3× higher for thinking-recoverable failures.Source: §E.5 · p. 21, Table 17 · p. 21

F Tokenizer and Script Effects on Cue Strength

G Crossed-Cue Control: Details

  • In the crossed-cue control, correct starts are preferred over incorrect starts matched in answer type and prior probability; labels were not human-verified.The answer-type labels include broad THING and exclude 64 questions labeled OTHER.Source: G Crossed-Cue Control: Details · pp. 21–22

H Gold-Free Search over First Characters

  • Gold-free first-character search gains +2.1pp by total log-probability, slightly less than beam search at +3.0pp, although correct candidates appear for 64.2% of questions.Continuation-confidence selection fixes some errors when the greedy first character is wrong but reduces accuracy by 4.4pp overall.Source: H Gold-Free Search over First Characters · p. 23, Table 22 · p. 23

J Results on All Questions

Including single-token answers, First-Token Cueing improves accuracy across English QA and all seven MKQA languages. Gains appear in every reported model-task cell.

J Results on All Questions

  • Cueing improves accuracy over Baseline in all 160 model-task cells when single-token answers are included.For single-token answers, the cue is the complete gold answer.Source: J Results on All Questions · p. 23, Table 25 · p. 25
  • The all-question English QA and MKQA results are shown as Baseline-versus-Cueing comparisons with absolute and relative gains annotated.Figure 12 covers English open-domain QA; Figure 13 covers MKQA.Source: Figure 12 · p. 24, Figure 13 · p. 24
  • All 112 model-language cells show positive cueing-minus-Baseline accuracy gaps across 16 models and seven MKQA languages.Figure 13 reports baseline accuracy in parentheses for each cell.Source: Figure 13 · p. 24

Related papers

Loading 2610.23400v1…