PaperlayerView exact revision on arXiv

Source-linked AI summary

The Model Knows When to Stop: Training-Free Early Stopping for Long-Context Reading

Muath Alyobi, Mohamed Eltahir, Almoayyad Abuljdail, Riyadh Almutawa, Tanveer Hussain, Naeemullah Khan

arXiv:2609.34590v1cs.CLcs.LG

TL;DR

Long-context readers can waste computation by continuing after sufficient evidence has been collected, while existing stopping approaches learn a signal or ask the model to self-report. ACS instead measures the frozen model’s answer state and stops when it is confident and stable. It is the only stopping policy reported to match or exceed full-reading accuracy on LongBench-v2 for both frontier models, with low premature-stop rates across five models on S-NIAH.

  • Problem

    Chunked processing handles inputs that exceed a model’s context window, but can incur substantial cumulative cost after sufficient evidence has been collected.

    Source: §1 · pp. 1–2

  • Method

    ACS probes a frozen model’s answer state after each chunk and stops when confidence and stability criteria are both satisfied, without training a stopping component.

    Source: §1 · p. 2, §3.3 · p. 4, §3.2 · p. 4

  • Results

    ACS is the only stopping policy that matches or exceeds full-reading accuracy on LongBench-v2 for both frontier models, and its S-NIAH premature-stop rate ranges from 0% to 12% across five models versus 8.4% to 45.6% for the verbalized gate.

    Source: §6 · pp. 9–10

  • Takeaways & Limitations

    The results show that a useful stopping signal can come from a frozen model’s evolving output-side answer state without training a separate controller or accessing internal activations.

    Source: §6 · p. 10

    ACS requires token log probabilities, which some inference endpoints do not expose, and savings can be small when confidence and stability occur only late in the context.

    Source: §7 · p. 10

Abstract

from arXiv · show

Language models often process long inputs sequentially in chunks, but continuing to read after sufficient evidence has been acquired wastes computation. Existing stopping mechanisms either learn sufficiency from internal activations or train an exit gate, while a simpler alternative asks the model whether it has read enough. We introduce Answer-Convergence Stopping (ACS), a training-free stopping rule that measures rather than asks. After each chunk, it probes the frozen model's current answer state and stops when that state is both confident and stable. The rule requires only output-side generation and token log probabilities, has no trained components, and uses one shared configuration across models and benchmarks. Because a stopping policy can save computation simply by stopping too early, we evaluate the stopping decision itself using evidence position where available. On the full LongBench-v2 with two frontier models, ACS is the only stopping policy that matches or exceeds full-reading accuracy. Furthermore, across 250 S-NIAH questions, the premature stopping rate for ACS across five models from two families ranges from 0% to 12%, compared to 8.4% to 45.6% for the verbalized gate. Taken together, ACS reveals that by properly utilizing the output signals of frozen models, we can achieve favorable behaviors like adaptive stopping without the need for additional training.

1 INTRODUCTION

Chunked reading makes long inputs tractable, but it can keep consuming computation after sufficient evidence has been collected. The paper introduces ACS, which measures answer-state confidence and stability, and evaluates stopping against evidence position as well as accuracy and cost.

1 INTRODUCTION

  • Sequential chunk processing is motivated by long inputs that exceed a model’s context window and can incur cumulative cost after sufficient evidence is collected.Incremental processing carries information forward rather than presenting the full context in one call.Source: §1 · pp. 1–2
  • Prior approaches include learned activation-based sufficiency classifiers and a trained exit gate, while DCC also studies prompting models to judge context sufficiency.DCC reports that self-prompting improves with model scale, while activation-based probes remain stronger across evaluated models.Source: §1 · p. 2
  • ACS measures the frozen model’s evolving answer state and stops only when its confidence and stability criteria are both met.The confidence criterion measures commitment, while stability checks that the belief has stopped changing; no stopping component is trained.Source: §1 · p. 2
  • The paper evaluates stopping decisions against known evidence positions on needle benchmarks, reporting premature-stop rate, over-read, accuracy regret, and savings capture.This distinguishes evidence-aligned savings from savings obtained by truncating reading before the evidence.Source: §1 · p. 2
  • On the full LongBench-v2, ACS is the only stopping policy that matches or exceeds full-reading accuracy on both frontier models.On S-NIAH, ACS stops before the evidence on 1.6% of questions with Qwen3-14B, versus 8.4% for the verbalized gate; its premature-stop rate ranges from 0.0% to 12.0% across five models, versus 8.4% to 45.6%.Source: §1 · p. 2
  • The contributions include a training-free rule for option-constrained and open-ended answers and a shared confidence-stability formulation evaluated across settings.The evaluation spans model families, scales, benchmarks, and answer formats, with a shared conservative default and separately studied accuracy-efficiency operating points.Source: §1 · p. 2

2 RELATED WORK

Prior work makes long inputs tractable through sequential processing, decomposition, or navigation, and explores learned or prompted stopping signals. The paper identifies whether a frozen model’s evolving answer state can itself provide a training-free stopping signal as an open question.

Reading in pieces

Knowing when to stop

  • DCC uses a trained sufficiency classifier and GRU-Mem trains an exit gate; neither establishes whether an evolving answer state can provide a training-free signal for sequential reading.DCC’s classifier uses labels derived from ground-truth answer locations, while GRU-Mem trains exit behavior with an explicit reinforcement-learning reward.Source: §2 · p. 3

Stopping on other axes

  • Prior work also uses confidence, uncertainty, and answer-space measurements to adapt computation along axes other than sequential input reading.Examples include exiting transformer depth, stopping sampling, triggering retrieval, and allocating compute across video frames.Source: §2 · p. 3

Synthesis

  • The paper frames the unresolved question as whether a frozen model’s evolving answer state can terminate sequential long-context reading without training.The preceding lines of work address long-input processing, learned sufficiency or exit behavior, or other forms of adaptive computation.Source: §2 · p. 3

3 ANSWER-CONVERGENCE STOPPING (ACS)

ACS processes a document in chunks, probes the frozen model’s answer state after each step, and stops at the first sufficiently confident and stable state. Its evaluation measures whether stops align with evidence position, not just whether they reduce reading cost.

3.1 SETUP

  • The reader folds document chunks into capped running notes, then uses a separate probe’s prediction as the final answer at the selected stopping step.ACS makes a note-update call and a probe call at each processed step; reported costs include all calls, and the serving stack is assumed to support output generation and token log probabilities.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 THE MEASURED SIGNAL

  • After each fold step, the probe produces an answer state and confidence; option-constrained questions use an option posterior, while open-ended questions use a greedy draft and its token probabilities.Changes are Jensen–Shannon divergence for posteriors and 1 − F1 for drafts; empty or abstaining drafts cannot trigger a stop.Source: §3.2 · p. 4

3.3 THE STOPPING RULE

  • ACS stops at the first step where confidence meets θ, mean answer-state change meets ε, and the answer is eligible to trigger stopping; if no step qualifies, it reads the full document.The stability statistic averages change over the last min(t, w) answer states; empty or abstaining open-ended drafts are ineligible.Source: §3.3 · pp. 4–5

3.4 EVALUATING THE STOPPING DECISION

  • On needle benchmarks, known answer offsets identify the evidence-containing chunk, enabling evaluation of premature stops, over-read, accuracy regret, and savings capture.Oracle stopping is the earliest evidence-aligned stop; premature stops contribute zero captured savings.Source: §3.4 · p. 5

4 EXPERIMENTS

Experiments compare ACS with full reading, asked gates, and random stopping across long-context benchmarks, while also evaluating whether stopping aligns with evidence position. ACS matches or exceeds full-reading accuracy on LongBench-v2 for both frontier models and shows lower premature-stop rates across evaluated models.

4.1 SETUP

  • The evaluation covers LongBench-v2, S-NIAH, RULER-HotpotQA, and BrowseComp-Plus, comparing stopping rules on the same reader, chunking, and note-update procedure.Direct baselines include full reading, random stopping, a verbalized confidence gate, and an END/CONTINUE gate; DCC and GRU-Mem are discussed as related methods rather than direct baselines.Source: §4.1 · pp. 5–6

4.2 ACS MATCHES FULL READING AT HALF THE COST

  • On LongBench-v2, ACS is the only stopping method matching or exceeding full-reading accuracy on both frontier models.On Qwen3.5, accuracy rises from 0.535 to 0.541 with 50% fewer tokens and 53.1% less elapsed time; on Kimi, the shared setting saves 3% of tokens and 14.1% of time. Table 1 also reports the post-hoc Kimi setting at θ = 0.92: accuracy rises from 0.517 to 0.535 with 49% fewer tokens and 52.6% less elapsed time.Source: §4.2 · p. 6, Table 1 · p. 6
  • At similar token budgets on Qwen3.5 and Kimi, random stopping loses 6.1 and 3.6 accuracy points, while ACS gains 0.6 and 1.8 points.On Qwen3.5, the verbalized gate saves 30.2% of tokens with near-full accuracy, while END saves 65% but loses 2.8 accuracy points.Source: §4.2 · p. 6
  • ACS saves tokens across all six LongBench-v2 domains, with the largest reductions on code repositories, reaching 62% on Qwen3.5 and 67% on Kimi.Reported token and time savings include the calls required by each stopping policy, accounting for ACS's probing overhead.Source: §4.2 · p. 6
  • On S-NIAH, ACS matches full-reading accuracy on both models while saving 27–28% of tokens and about 29% of elapsed time.On RULER-HotpotQA, ACS trails full reading by 0.8 points on Qwen3.5 and 0.2 points on Kimi; on BrowseComp-Plus, it trades 1.2 accuracy points for 41% fewer tokens and 45.2% less elapsed time on Qwen3.5, while matching full reading on Kimi with 12% token and 19.3% time savings.Source: §4.2 · p. 7

4.3 IT STOPS ONCE THE EVIDENCE HAS BEEN READ

  • On S-NIAH, ACS stops before evidence on 1.6% of questions versus 8.4% for the verbalized gate, while capturing 60.7% of oracle-available savings.Across five models, premature stopping ranges from 0.0% to 12.0% for ACS and 8.4% to 45.6% for the verbalized gate; ACS over-reads 1.51 chunks on average. The verbalized and END gates capture more oracle savings, 92.0% and 95.2%, but have higher premature-stop rates and accuracy regret.Source: §4.3 · pp. 7–8, Table 2 · p. 7
  • At frontier scale, ACS matches full-reading accuracy on Qwen3.5 and Kimi K2.5, scoring 1.000 and 0.996 while capturing 63.9% and 65.8% of oracle-available savings.Across five models from Qwen and Gemma families, ACS's premature-stop rate remains 0–12%, compared with 8.4–45.6% for the verbalized gate under the shared θ = 0.995 setting.Source: §4.3 · p. 8, Table 3 · p. 8

5 ABLATION STUDY

The ablations show that stability materially improves stopping safety and accuracy, while the confidence threshold controls a model-dependent safety–efficiency trade-off. Tested window and fold settings have comparatively limited or workload-dependent effects.

5 ABLATION STUDY

  • Removing stability raises mean premature-stop rate from 3.7% to 10.4% and lowers mean accuracy from 0.957 to 0.890 across five models.On Gemma-3-27B, accuracy falls from 0.876 with stability to 0.520 without it.Source: §5 · p. 9
  • On Qwen3.5, premature stopping falls from 30.4% at θ = 0.80 to zero at θ ≥0.99; Kimi’s tested thresholds all retain at least its full-reading accuracy of 0.517.The shared θ = 0.995 gives Qwen3.5 its highest LongBench-v2 accuracy, 0.541, but only 3% token savings on Kimi.Source: §5 · p. 9, Table 6 · p. 9
  • On S-NIAH, every tested ε from 0.005 to 0.05 yields identical stopping decisions, while changing the LongBench-v2 stability window from w = 2 to w = 5 shifts accuracy by at most 0.8 points.The default w = 3 gives the highest Kimi accuracy and ties for the highest Qwen3.5 accuracy while retaining greater savings than larger windows.Source: §5 · p. 9
  • On S-NIAH, halving or doubling chunk size L or notes cap B changes accuracy and premature stopping only modestly, whereas LongBench-v2 is more sensitive to these fold settings.For LongBench-v2, L = 24K and B = 6K give the highest accuracy, 0.538.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents ACS as a training-free stopping rule based on a frozen model’s confident, stable answer state. It reports accuracy parity or gains against full reading on frontier-model LongBench-v2, alongside lower and more consistent premature stopping than the verbalized gate on S-NIAH.

6 CONCLUSION

  • On full LongBench-v2, ACS is the only stopping policy that matches or exceeds full-reading accuracy on both frontier models; on Qwen3.5 it uses 50% fewer tokens and 53.1% less elapsed time.On S-NIAH, Qwen3-14B has 1.6% premature stopping, and ACS remains within 0–12% across five models versus 8.4–45.6% for the verbalized gate.Source: §6 · p. 9
  • ACS uses the frozen model’s evolving output-side answer state without training a separate controller or accessing internal activations.The ablations show that confidence alone is insufficient: stability reduces premature stopping, and the confidence threshold exposes a model-dependent safety–efficiency trade-off.Source: §6 · p. 10

7 LIMITATIONS

ACS depends on token log probabilities and its savings depend on when the answer state becomes confident and stable. When that occurs late in the context, the method can approach full-reading cost.

7 LIMITATIONS

  • ACS requires token log probabilities, which some inference endpoints do not expose.This limits its availability on endpoints without token-level log probabilities.Source: §7 · p. 10
  • When the answer state becomes sufficiently confident and stable only late in the context, ACS continues reading and can approach full-reading cost.Substantial savings are therefore not guaranteed across workloads.Source: §7 · p. 10
  • The method’s efficiency trade-off is intended to preserve safety rather than assume substantial savings are always available.The paper frames late stopping as a workload-dependent boundary on efficiency gains.Source: §7 · p. 10

A IMPLEMENTATION DETAILS

The implementation folds documents into fixed-width chunks and maintains capped running notes, then probes the current answer after each chunk. Prompts are fixed across steps, models, and benchmarks, with separate probe and reader instructions for multiple-choice and open-ended tasks.

A IMPLEMENTATION DETAILS

  • The fold uses document-order chunks of L = 24,000 characters and running notes capped at B = 6,000 characters.The notes prompt directs the model to retain question-relevant information, preserve exact names and numbers, and stay within the cap.Source: A IMPLEMENTATION DETAILS · p. 12
  • Multiple-choice probes constrain generation to option letters and return top-k = 20 token log probabilities, while open-ended probes greedily decode at most 32 tokens.Answer labels are stripped before draft comparison, and fixed-list abstentions cannot trigger stopping.Source: A IMPLEMENTATION DETAILS · p. 12
  • Full reading probes after every chunk, while the verbalized and END/CONTINUE gates ask the model whether its notes suffice or enough information has been collected.Policies are compared over shared recorded reader trajectories, with each charged for its own required calls.Source: A IMPLEMENTATION DETAILS · p. 12
  • The main experiments use shared constants θ = 0.995, ε = 0.05, and w = 3, fixed across experiments.Source: A IMPLEMENTATION DETAILS · p. 12

B PROMPTS

  • Prompts remain fixed across steps, models, and benchmarks, with per-call placeholders for the question, options, notes, chunk, notes cap, and draft.The multiple-choice and open-ended fold prompts both instruct the model to update running notes from each new chunk.Source: B PROMPTS · p. 12
  • The verbalized gate asks for a 0–100 sufficiency confidence, while the END/CONTINUE prompt requests an explicit decision about whether the notes contain enough information.Source: B PROMPTS · p. 13

C LONGBENCH-V2 BY DOMAIN

Code repositories yield the largest LongBench-v2 token savings, while sensitivity analyses show that confidence-percentile normalization does not provide a general replacement for the shared conservative threshold.

C LONGBENCH-V2 BY DOMAIN

  • 62% on Qwen3.5 and 67% on Kimi are the largest LongBench-v2 savings, in code repositories averaging 150.2 chunks.Table 7 reports token savings by domain; the prose identifies code repositories as the longest domain and gives the savings values.Source: C LONGBENCH-V2 BY DOMAIN · p. 13, Table 7 · p. 14
  • The S-NIAH threshold sweep reports accuracy and mean token cost as θ varies; all tested ε values produce the same results.The table covers Qwen3-14B on 250 questions with w = 3.Source: Table 8 · p. 14

D SENSITIVITY

  • The best LongBench-v2 thresholds lie at the 54th and 55th percentiles of full-reading confidence, but the percentile mapping yields 30% premature stopping on Qwen3-32B.The paper concludes that confidence-percentile normalization does not replace the shared conservative threshold.Source: D SENSITIVITY · p. 14

E ROBUSTNESS OF THE SHARED OPERATING POINT

A shared conservative configuration was compared with model-specific cost-aware tuning on held-out S-NIAH questions. The shared setting retained equal or higher accuracy on every model and higher mean held-out accuracy.

E ROBUSTNESS OF THE SHARED OPERATING POINT

F ADDITIONAL RESULTS

Additional analyses account for probing costs and examine accuracy–cost trade-offs across thresholds, chunk settings, and stability windows. Results show that stricter thresholds can reduce premature stopping while lowering savings.

F ADDITIONAL RESULTS

  • 56.4% on Qwen3.5 and up to 55.2% on Kimi are counterfactual savings after removing intermediate probe calls; reported savings include probe calls.The counterfactual preserves stopping decisions and accuracy and retains the final probe used to produce the answer.Source: F ADDITIONAL RESULTS · p. 15, Table 10 · p. 15
  • 20.7% to 8.8% premature stopping and 0.532 to 0.608 accuracy accompany the stricter threshold for Qwen3-14B.Across the three models, increasing θ reduces pre-proxy stopping and moves accuracy toward full reading at the cost of lower savings.Source: F ADDITIONAL RESULTS · p. 15
  • On the LongBench-v2 subset, default L = 24K and B = 6K give the highest accuracy, while changing either exposes different accuracy–cost trade-offs.Table 12 covers Kimi K2.5 on a prespecified 80-question subset.Source: F ADDITIONAL RESULTS · p. 16, Table 12 · p. 16
  • Accuracy varies by at most 0.8 points across w = 2–5 on either model; w = 3 is best or tied for best accuracy and preserves more savings than larger windows.Table 13 reports the stability-window ablation on all 503 LongBench-v2 questions.Source: F ADDITIONAL RESULTS · p. 16, Table 13 · p. 16
  • A verbalized threshold of 0.995 attains or ties the best observed operating point for every model, yet premature stopping remains high on Qwen3-32B and Gemma-3-12B.Raising the verbalized threshold helps some models but has little or no effect on others.Source: F ADDITIONAL RESULTS · p. 16, Table 14 · p. 16
Loading 2609.34590v1…