Periscope: Extending Frozen Language Models Beyond Their Context Window

1 INTRODUCTION

• Single-pass reads incur quadratic attention cost, stop at the context window, and can lose accuracy well before reaching it.Prior studies report accuracy drops even when relevant passages are present and substantial degradation for models claiming 128k-token contexts.Source: §1 · p. 2
• Periscope addresses finite-set decisions by composing bounded-window answers with a maximum, which the paper says adds no approximation of its own.The motivation is that a decision often needs only one supporting passage, and a bounded window can be judged by a frozen model.Source: §1 · p. 2
• Periscope arranges chunks on a K×K grid and uses K local and K strided probes to score answers and map evidence locations.Each chunk receives the combined score of its local and strided spans; the map supports document ranking, direct answers, evidence location, and selection of a short read.Source: §1 · p. 2
• A 9k-token map-selected read matches the same model’s best LongBench v2 window read; InfiniteBench En.MC leads by 5 points, and BRIGHT tops six methods on NDCG@10 and MRR.The LongBench v2 comparison covers windows from 32k to 1M tokens; the selected read uses the K chunks ranked highest by the map.Source: §1 · p. 2

Limits of the single long read

• Long-context models can lose accuracy before their claimed window limit, including when retrieval is perfect or relevant content is present.The cited studies also report reduced use of middle-of-input content and greater difficulty when the evidence shares little vocabulary with the question.Source: §2 · p. 3

Extending the window

• Extending windows preserves quadratic attention and growing cache costs; sub-quadratic architectures still retain full-attention layers, require training, and read the whole text in one pass.The cited examples fit long contexts using multi-GPU systems, and accuracy can decline within the window.Source: §2 · p. 3

Reading less

• Retrieval and prompt compression shorten the pass, but a selector can omit what the reader needs, especially when relevance requires reasoning rather than surface matching.Dense and late-interaction retrievers score passages using representations computed before the query is seen.Source: §2 · p. 3

Language models as judges

• Language-model rerankers read relevance scores or orderings, but assume each document fits in one call and candidates have already been assembled.A single-answer-token judgment is described as cheap and reliable.Source: §2 · p. 3

Reading in pieces

• Passage scoring sees only its own piece, while sequential or parallel agent systems carry generated messages that can drop information needed by later chunks.The final answer in these systems has no stated relation to a read of the full text.Source: §2 · p. 3

Separable structure and answer-space probing

• Grid probing applies separable row-and-column structure to video frames in 2K calls for K^2 frames, but long text has no such geometry of its own.The cited approach combines answers into a map used to select frames for a second pass.Source: §2 · pp. 3–4
• A read beyond the window needs bounded, independent calls, a decision made by the reader, whole-text coverage, and evidence-location scores; Figure 2 illustrates a grid-based map peaking at the evidence chunk.In the depicted LongBench v2 example, the window read misses chunk 9, while Periscope’s map identifies it.Source: §2 · p. 4, Figure 2 · p. 4

3.1 THE PROBE

• Each probe scores answers from a bounded span at the first answer token, using abstention so unsupported spans score every option low rather than favoring one randomly.For relevance, the score is the log-odds of Yes against No.Source: §3.1 · p. 4

3.2 LOCAL AND STRIDED SPANS

• The text is split into chunks on a K×K grid; K local spans cover consecutive chunks and K strided spans sample every K-th chunk across the text.Every chunk appears in exactly one span of each kind and is read twice; each probe holds about √sc tokens.Source: §3.2 · pp. 4–5

3.3 COMPOSITION, MAP, AND READOUTS

• For each answer, Periscope adds its best local and strided scores, so both a local span and a text-spanning sample must support a high score.The score is built from the strongest span in each family.Source: §3.3 · p. 5
• Each chunk’s local and strided span scores form an evidence map at no further call cost; its peak identifies the chunk whose spans produced the answer.The map’s largest cell equals the answer score.Source: §3.3 · p. 5
• The same probes support fixed readouts for document ranking, direct answers, evidence localization, or a selected read of the K highest-scoring chunks.The selected read joins those chunks in original order and answers in one additional call, for 2K + 1 probes total.Source: §3.3 · p. 5

3.4 REACH, COST, AND MEMORY

• A window of W tokens can reach about W^2/c tokens; at c=500, 32k and 128k windows reach 2M and 33M tokens, respectively.This bounds reachable text length, not read accuracy.Source: §3.4 · p. 5
• Probe attention cost grows as s^1.5 rather than s^2 for a single read, while each call caches only its probe, so long-read memory is bounded by one window.The probes do twice the per-token work because each chunk is read twice; the long read’s memory need is set by model weights rather than text length.Source: §3.4 · p. 6

4.2 BENCHMARKS & BASELINES

• Question answering uses LongBench v2 and InfiniteBench; retrieval uses BRIGHT’s long-document corpora, with window, chunk-selection, and retrieval baselines.All arms use frozen models and the same GPU; chunks are 500 tokens for QA and 100 for retrieval.Source: §4.2 · p. 6, §4.1 · p. 6

4.3 THE MAP REPLACES THE READ

• On LongBench v2, reading the map’s top K chunks (9k tokens) scores 41.9, matching the 4B model’s best window read across all 503 questions.The best window read peaks at 131k; every other tested window scores lower.Source: §4.3 · p. 6, Table 1 · p. 7
• Map-selected chunks beat embedding, random, and first-chunk selection by 6, 9, and 11 points on every LongBench split, respectively.On 298 contexts fitting 131k, the selected read matches whole-context answers on 71% of questions, versus 64% for embedding selection.Source: §4.3 · p. 7
• At 262k, the 27B window read scores 55.9 overall, 3.2 points above either Periscope read; Gemma’s selected read scores 43.7 versus 42.9 for its window read.On LongBench’s long split, the 27B direct readout scores 57.4, matching the 262k read and exceeding the 131k read by 7.4 points.Source: §4.3 · p. 8

4.4 A SECOND BENCHMARK

• On InfiniteBench En.MC, the map-selected read scores 82.1, 5 points above the best window read at 262k.The map leads by 14 points on the 150 novels that do not fit 131k; the 131k read leads on the 79 that do.Source: §4.4 · p. 8, Table 2 · p. 8

4.5 IS THE MAP RIGHT?

• On BRIGHT, Periscope ranks the long-document corpora with the best NDCG@10 and MRR among six methods.Against the task-trained embedder, it leads MRR and Recall@1 but trails by half a point on Recall@10.Source: §4.5 · p. 8, Table 3 · p. 8

4.6 COST

• At 1M tokens, a probe batch takes 108 s versus 457 s for one read; at 4.5M, a 27B probe caches 3.1 GB versus 296 GB for a single read.The measured compute crossover is 119k tokens for the 4B and 278k for the 27B.Source: Figure 3 · p. 9, §4.6 · p. 9

5 ABLATIONS

• On BRIGHT, adding strided probes raises NDCG@10 from .551 for local spans to .637; on LongBench v2, they add 0.8.Chunk-alone MaxP and local spans both score .551 on BRIGHT, so the added score comes with the strided probes’ whole-document sample.Source: §5 · p. 9
• On LongBench v2, Chain of Agents scores 10 points below the selected read and takes three times as long.Its K+1 calls run in sequence, with each message written before the next span is read.Source: §5 · p. 9

6 CONCLUSION AND LIMITATIONS

• Periscope uses 2K bounded local and strided probes to reach W^2/c tokens from a W-token window, with the cache of one probe.The method achieves comparable performance to a full read.Source: §6 · p. 10
• The method covers decisions over a finite answer set; evidence from distant spans is joined only in the selected read.It trails the window read on multi-document QA.Source: §6 · p. 10
• Below the crossover, Periscope costs more than a single read; future work includes open-ended readouts and distilling probe scores and maps into cheaper retrievers and rankers.Source: §6 · p. 10

A.1 IMPLEMENTATION

• Each probe is one user turn; relevance and question prompts place the text first, and both read out a single answer token.The relevance prompt asks whether a document is relevant; the question prompt presents options A–D and an unsure option E.Source: §A.1 · p. 14
• Calls use max_tokens=1, read top-20 answer log-probabilities, and disable thinking mode where available.The selected read and window read use the same prompt without the unsure option; no question is defaulted to a letter.Source: §A.1 · p. 14
• Sampling uses temperature 1.0 with nucleus and top-k truncation disabled so answer tokens remain in the returned distribution.Temperature rescales log-odds uniformly and does not change rankings.Source: §A.1 · p. 14
• Texts are split into c-token chunks and arranged in a padded K×K grid; empty spans are skipped and chunks are joined with spaces.Strided spans therefore contain discontinuities at chunk boundaries, and scores are used without cross-span calibration.Source: §A.1 · p. 14
• Models run on one A100 80GB GPU with vLLM, and the 2K probes are issued as one batch.Every arm of a model, including the window read, runs against the same instance; the window arm reads the longest prefix that fits.Source: §A.1 · p. 14
• Chunk selection reads the top K 500-token chunks by cosine similarity in original order; random selection uses a fixed seed.The Chain of Agents baseline summarizes the K local spans in sequence before a manager answers from the final summary.Source: §A.1 · p. 14

A.2 COST

• For Qwen3-4B-Instruct-2507 at c=100, the FLOPs crossover is 65.9k tokens, or 15.7k against local probes alone.The reported coefficients are a=7.3×10^9 FLOPs per token and b=1.8×10^5 for attention.Source: §A.2 · p. 15
• At 1M tokens, probes take 108 s versus 457 s for one read; at 262k, they take 0.65 times the single-read time.On the measured A100 80GB setup, probes cost more time through 131k tokens and less from 262k onward.Source: §A.2 · p. 15
• At c=500, compute crossover is 119k tokens for the 4B and 278k for the 27B; at 1M, probe compute is 0.23 and 0.44 times single-read compute, respectively.Below crossover probes cost more; at 262k their ratios are 0.63 and 1.03.Source: §A.2 · p. 15
• 296 GB of cache would be needed for a 27B single read of the 4.5M-token LongBench v2 context, versus 3.1 GB for a probe.At this context, one probe holds at most 47.6k tokens; on the 80GB GPU, a 27B single read past 356k tokens cannot run.Source: §A.2 · p. 15
• On BRIGHT, most documents are below crossover and Periscope costs 2.6 to 27 times the whole-read FLOPs by domain, with a median of 5.Whole reads fail on 0.7 to 26.1% of documents by domain at the 16k window; the reported window depends on budget and hardware.Source: §A.2 · p. 16

A.3 PER-DOMAIN RETRIEVAL RESULTS

• Periscope leads the window read on NDCG@10 and MRR in five domains, but trails in Earth science and Robotics.The latter domains’ documents that fit the 16k window carry most of the relevance.Source: §A.3 · p. 16, Table 5 · p. 16
• Against the embedder, Periscope leads MRR in five domains, Recall@1 in six, and Recall@10 in three.The table reports per-domain NDCG@10, MRR, and Recall@10 on the full corpus, with 50 queries per domain.Source: §A.3 · p. 16, Table 5 · p. 16

• Single-document QA and dialogue lead or tie at both scales; multi-document QA trails the window read by 6 points for 4B and 9 for 27B.Structured data and code beat the window read for 27B on both; in-context learning trails by 5 points for 4B but leads by 5 for 27B.Source: §A.4 · p. 16, Table 6 · p. 17
• Table 8 reports accuracy by context-length bin for both models, with edges at the 131k and 262k windows.A window read beyond its window uses the longest prefix that fits.Source: Table 8 · p. 17
• Table 9 compares each arm’s answers and option log-probabilities with whole reads on 298 LongBench v2 contexts read whole.Figure 4 plots direct-readout log-probabilities against the whole-read values for those contexts.Source: Table 9 · p. 18, Figure 4 · p. 18
• Eleven questions at 4K chunks were excluded from accuracy because their reads did not fit the window; none of the Table 1 arms had this issue.Accuracy was computed over the 492 answered questions.Source: §A.4 · p. 18
• At 1,000-token chunks, the medium-split selected-read score falls from 41.9 to 38.6.Doubling chunk size halves probe count and doubles probe length; the score difference is close to run-to-run variation.Source: §A.4 · p. 18

• The study marks evidence chunks in 100 OBLIQ Congress documents, with a second annotation pass confirming 53; documents contain 10–30 chunks of 50 tokens.Top-1 tests whether the map’s peak is annotated, while MRR uses the first annotated chunk’s rank.Source: B CHUNK-LEVEL LOCALIZATION, WHERE THE… · p. 18
• In the flat-map example, selection misses while direct readout still answers correctly; other examples peak on answer-bearing chunks in long documents.The figure marks local spans as rows, strided spans as columns, selected chunks as dots, and the peak with an outline.Source: Figure 5 · p. 19
• 2.5 Top-1 points and 7 MRR points above the judge’s whole read, at a fifth of its tokens, on 100 annotated OBLIQ documents.The map also scores 17 points above chance on Top-1.Source: B CHUNK-LEVEL LOCALIZATION, WHERE THE… · p. 19, Table 10 · p. 19
• At 500–1,500 tokens, documents fit in one probe, so strided spans add no reach and the map ties LocalOnly and the embedder.Below the length of a call, the map is a retriever among retrievers.Source: B CHUNK-LEVEL LOCALIZATION, WHERE THE… · p. 19