PaperlayerView exact revision on arXiv

Source-linked AI summary

Masking Stale Observations Helps Search Agents -- Until It Doesn't: A Regime Map and Its Mechanism

Haoxiang Zhang, Qixin Xu, Zhuofeng Li, Lei Zhang, Pengcheng Jiang, Yu Zhang, Julian McAuley

arXiv:2606.00408v1cs.CLcs.AIcs.IR

TL;DR

It remains unclear when masking stale observations benefits search agents managing accumulated context. The paper uses minimal observation masking as a diagnostic and finds that gains plateau under weak retrieval, peak in a sweet spot, and collapse for saturated models.

  • Problem

    It remains unclear when observation masking benefits agentic search as agents manage accumulated context.

    Source: §1 · p. 2

  • Method

    The study uses minimal observation masking as a diagnostic to investigate when stale observations matter to agent performance.

    Source: §2.2 · p. 4

  • Results

    Masking gains follow a non-monotonic pattern: a retriever-bottleneck plateau, a +11.7 pts sweet spot, and a model-saturated collapse that recurs on live-web benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Context management should be calibrated jointly with retriever recall and model capacity rather than applied by default.

    Source: §7 · p. 10

    The regime framework is descriptive rather than predictive.

    Source: LIMITATIONS · p. 11

Abstract

from arXiv · show

Long-horizon search agents accumulate large amounts of retrieved content across many tool calls, making context-budget efficiency increasingly important. A minimal intervention is to mask stale observations from the context as the trajectory progresses, but it remains unclear when this form of context management helps and why. We study observation masking through a systematic sweep over various agent backbones (4B to 284B parameters) and three retrievers on offline and live-web agentic search benchmarks. We find that the accuracy gain from masking follows an asymmetric inverted-U shape when plotted against the model's accuracy without context management: a plateau under weak retrievers, a peak when a strong retriever meets a mid-capacity model, and a sharp collapse when the model is saturated. This pattern reflects the interaction between retriever recall and the model's implicit filtering capacity, rather than either factor in isolation. Mechanistically, masking implements a token-for-turn trade-off: it removes observations the model has largely stopped attending to and pages the agent rarely re-opens. The added turns help when they convert failures into successes, but they fail when masking removes evidence the model would otherwise have used. We therefore reframe context management as a regime-dependent intervention and provide a holistic perspective for analyzing context use in agentic deep search. We release our scaffold and trajectories here (https://github.com/i-DeepSearch/observation-masking) to support future research.

1 INTRODUCTION

Masking stale observations produces regime-dependent gains, peaking at intermediate proficiency and collapsing when models saturate. Attention and page-opening patterns help explain this behavior, while retrieval quality constrains advanced agents.

1 INTRODUCTION

  • The study addresses a missing systematic account of when context management helps as growing search trajectories interact with increasingly capable backbones.Trajectories can reach tens of thousands of tokens dominated by observations. Capable models may already ignore stale content, potentially making context management redundant in some regimes.Source: §1 · p. 2
  • 4B–284B-parameter backbones and sparse-to-specialized-dense retrievers are evaluated across offline and live-web search while the scaffold, masking rule, prompts, and evaluation protocol remain fixed.A unified parallel-tool-call scaffold supplies a competitive baseline for measuring context-management gains.Source: §1 · p. 2
  • +6–7 pts, +11.7 pts, and ≤0 pts characterize offline CM gains in retriever-bottleneck, sweet-spot, and model-saturated regimes, respectively.Gains vary non-monotonically with baseline proficiency; the model-saturated collapse also appears in live-web environments.Source: §1 · p. 2
  • Observation attention decays sharply beyond recent turns, while reasoning attention and page-opening behavior follow U-shaped patterns that neglect the trajectory’s middle.Observations receive substantially less attention than reasoning. Masking removes neglected middle content but can backfire once advanced models move beyond these constraints.Source: §1 · p. 2
  • Regression probes identify context signal-to-noise optimization as a bottleneck for advanced agents, favoring higher-quality retrieval over aggressive masking.Better retrievers increase the upper bound of actionable signal, whereas masking risks removing critical evidence when models saturate.Source: §1 · p. 2
  • Observation masking replaces older tool outputs with placeholders while retaining reasoning and tool calls; masked pages remain available for reopening.The most recent K observations remain visible. Performance depends on whether evidence remains necessary, has already entered active reasoning, or can be retrieved again.Source: §1 · p. 2, Figure 2 · p. 3

2 METHODOLOGY

Observation masking replaces older observation content while preserving tool-call structure and errors, providing a minimal diagnostic of when that content matters. A persistent page pool keeps historical pages accessible even after their observations are masked.

2.1 PRELIMINARY

  • Agents produce reasoning–action–observation trajectories with parallel tool calls and a final answer action; policies condition each next step on the rendered trajectory.Context grows linearly with turns and is dominated by observation tokens.Source: §2.1 · p. 3

2.2 OBSERVATION MASKING

  • Staleness means falling outside the retention window, not proven irrelevance: an older observation may become useful again when the agent changes its plan.The heuristic assumes observations mainly inform the next reasoning and action step and are seldom reread; this assumption is tested empirically.Source: §2.2 · p. 3
  • Observation masking replaces older observation content with fixed placeholders while retaining tool-call structure and exempting tool-call errors.The retention window K preserves the most recent observations; error observations remain available for adaptive debugging. Placeholders preserve which tool was called and its arguments.Source: §2.2 · p. 4
  • Masking serves as a diagnostic instrument, adding neither tokens nor model calls rather than competing with more sophisticated context-management methods.Its purpose is to isolate when stale-observation content matters to agent performance.Source: §2.2 · p. 4

2.3 SCAFFOLD

  • A persistent page pool preserves historical pages independently of masked context, while search, open, and find separate exploration, reading, and localization.The append-only pool deduplicates URLs and assigns stable cursors, allowing historical pages to be referenced without repeating a search.Source: §2.3 · p. 4

3 EXPERIMENT SETUP

The evaluation spans four search benchmarks, offline and live-web retrieval, and open-weight agents with 4B to 284B parameters. A stronger no-context-management scaffold provides a conservative basis for measuring masking’s contribution.

3.1 BENCHMARKS

3.2 BASELINES

  • 4B to 284B is the parameter range of evaluated open-weight tool-calling agents; all except GPT-OSS support parallel tool calls.Backbones include Qwen (QwenTeam, 2026a;b), GPT-OSS (Agarwal et al., 2025), OpenResearcher (Li et al., 2026a), DeepSeek-V4-Flash-Max, and Tongyi-DeepResearch (Team et al., 2025b). Experiments use a 500-turn limit, observation retention window K = 5, and GPT-5-mini as the LLM judge.Source: §3.2 · p. 5
  • Three offline retrievers span sparse retrieval, mid-tier dense embeddings, and reasoning-tuned agentic retrieval; live-web benchmarks use an online retriever.BrowseComp-Plus uses BM25, Qwen3(-Emb)-8B (Zhang et al., 2025b), and AgentIR-4B (Chen et al., 2026), respectively. Live-web benchmarks use Serper API (Serper API, 2026).Source: §3.2 · p. 5

3.3 SCAFFOLD RELIABILITY

  • +11.2 points (GPT-OSS-20B), +12.4 points (GPT-OSS-120B), and +12.6 points (Tongyi-DeepResearch-30B-A3B) are No-CM accuracy gains over matched publicly reported BrowseComp-Plus baselines.A weaker scaffold could inflate context management’s apparent benefit by making noisy-context pruning helpful against scaffold deficiencies; the stronger baseline measures masking’s conservative contribution.Source: Figure 3 · p. 5, §3.3 · p. 5

4 MAIN RESULTS

Observation masking helps most when strong retrieval supplies evidence that a moderately accurate model cannot filter; gains plateau under weak retrieval and collapse for saturated models. These regimes depend on retrieval, model filtering capacity, and task—not parameter count alone.

Three regimes

  • +6.2 to +6.6 points is masking’s low plateau under BM25 across backbones, with recall never exceeding 0.55.Too little answer-supporting evidence reaches the context for masking to amplify; retrieval, rather than model capacity, caps accuracy.Source: §4 · p. 5
  • +11.7 points is the peak masking gain on BrowseComp-Plus, achieved by Qwen3.5-35B-A3B+AgentIR between retriever-limited and model-saturated regimes.This pairing combines retriever recall of 0.88 with moderate No-CM accuracy of 62.9%: supporting evidence is available, but the model cannot filter surrounding noise.Source: §4 · pp. 5–6, Table 1 · p. 6
  • −1.1 points is masking’s gain for Tongyi-DeepResearch-30B-A3B, illustrating the collapse once No-CM accuracy passes 70%.Tongyi has the table’s highest No-CM accuracy (80.7%) and recall (0.93), yet masking harms it. CM also adds 68.7 tool calls per query for GPT-OSS-120B and 57.7 for DeepSeek-V4-Flash-Max.Source: §4 · p. 6

Mismatch, not size

  • +11.7 points for Qwen3.5-35B-A3B versus +3.7 points for Qwen3.6-35B-A3B with AgentIR shows that identical architecture and parameter count do not determine masking’s regime.The authors attribute the gap to training-dependent attention dynamics, examined through localization behavior.Source: §4 · p. 6

Live web amplifies the collapse

  • −4.8 points on GAIA contrasts with +8.0 points on xBench for GPT-OSS-120B, showing that masking’s effect depends on task.The same model gains +0.1 points on BrowseComp-Plus; its xBench No-CM accuracy is 70.0%, placing it in the middle band. Noisier, less controllable live-web retrieval makes masking more likely to remove evidence capable readers would otherwise use.Source: §4 · p. 6, Table 1 · p. 6

5 RESEARCH FINDINGS

Observation masking saves rolling tokens on rescued queries but incurs costly failures when needed evidence is removed, so net gains depend on the rescue–harm balance. Attention and page-reopening analyses explain why stale observations are often expendable, while scaffold ablations favor retaining tool errors and exact URLs.

5.1 THE TRADE-OFF: WHERE COMES THE GAIN AND WHAT DOES IT COST?

  • +11.7 pts for Qwen3.5-35B-A3B versus +0.1 pts for GPT-OSS-120B accompany roughly 3:1 versus 1:1 ratios of fixed to broken queries.Fixed queries consume fewer rolling tokens, whereas broken queries drive token growth through additional searching, page reopening, and inference turns. Rolling input tokens sum context lengths across all turns, measuring total input cost rather than final context size.Source: §5.1 · pp. 6–7, Figure 4 · p. 7

5.2 REGRESSION PROBE: CM HELPS WHEN SIGNAL IS SPARSE

  • 0.74 CM-rescue AUC accompanies only +0.1 net gain for GPT-OSS-120B+AgentIR: predictable rescues do not guarantee useful masking.A balanced linear probe uses No-CM prefix complexity and fitted evidence density; saturated settings rescue recognizable high-complexity, low-SNR cases, whereas high-gain settings improve diffuse borderline trajectories. AgentIR evidence density declines as pages accumulate, while BM25 remains low and retriever-limited.Source: Figure 5 · p. 7, §5.2 · p. 8
  • +7.0% versus +2.0% accuracy gains distinguish Qwen3.5-9B and DeepSeek-V4-Flash-Max on xBench despite similar rescue separability, with AUC ≈0.70–0.71.Without gold documents, final-answer citation lines provide proxy signal and uncited observed lines provide proxy noise; this measures evidence surfacing, not factual correctness. DeepSeek already has higher proxy SNR, reinforcing that baseline capacity and rescue–harm balance govern net utility.Source: Figure 5 · p. 7, §5.2 · p. 8

5.3.1 ATTENTION EVIDENCE: REASONING OVER OBSERVATIONS

  • 53.7% of per-step attention goes to past reasoning versus 25.6% to tool observations, supporting masking of observations the model largely stops reading.Observations receive 65% of their attention within the most recent 10% of past turns and 80% by midpoint; reasoning reaches only 68% by midpoint. Reasoning attention rebounds on the oldest turns, while stale middle observations receive little attention.Source: §5.3.1 · pp. 8–9, Figure 6 · p. 9
  • Greater attention to mid-trajectory reasoning accompanies larger masking gains, while emphasis on the latest observation accompanies smaller gains.Among the three examined settings, 9B-AgentIR has the strongest reasoning attention and largest masking gain. Conversely, 3.6-35B-AgentIR emphasizes the latest observation, which masking preserves, and gains less.Source: §5.3.1 · p. 9, Figure 6 · p. 9

5.3.2 WHERE DO AGENTS RE-OPEN?

  • Agents rarely reopen middle pages; masking strengthens reopening of the newest and earliest pages, with larger gains accompanying more first-page revisits.Together, limited attention and infrequent reopening define stale middle-trajectory observations without downstream metrics. Masking reclaims largely abandoned context but removes optional evidence that can still be needed, explaining occasional failures.Source: §5.3.2 · pp. 9–10, Figure 7 · p. 9

5.4 ABLATION STUDIES

  • 18.6% to 22.6% on 4B and 20.4% to 24.6% on 9B are the open-error-rate increases caused by removing error retention under masking.Replacing exact URLs with blurred page titles raises open error rates to 20.8% on 4B and 26.2% on 9B. Retained errors support recovery from malformed calls, while exact URLs avoid reconstructing target addresses.Source: §5.4 · p. 10

6 RELATED WORK

Long-horizon agentic search accumulates retrieval noise that aggravates long-context degradation. This work uses a minimal diagnostic setup to study context-management utility without heavy architectural overhead.

Long-horizon Agentic Search

  • Long-horizon search requires state across dozens of tool-use turns, while retrieval noise amplifies long-context degradation and lost-in-the-middle effects (Liu et al., 2024).Retrievers support exploration and evidence localization but also generate observation-heavy contexts, making sparse evidence harder to recover.Source: §6 · p. 10

Context Management for Autonomous Agents

  • AgentFold (Ye et al., 2025) uses non-native tool-calling loops, while ReSum (Wu et al., 2025) requires rolling summarization that increases computation and can reduce KV-cache reuse.This paper instead uses a minimal diagnostic setup to investigate how observation dynamics shape context-management utility without heavy architectural overhead.Source: §6 · p. 10
  • Context-management research builds on working memory inspired by MemGPT (Packer et al., 2023), using static truncation, heuristic eviction, and online compression.Static-truncation approaches include Liu et al., 2025; Zeng et al., 2026; Moonshot AI, 2025.Source: §6 · p. 10

7 CONCLUSION

Observation masking is a regime-dependent intervention, not a default: its utility depends on baseline system capability. It works best when retrieval recall outpaces the model’s intrinsic noise-filtering capacity.

7 CONCLUSION

  • Observation masking thrives in an intermediate sweet spot where enhanced retrieval recall outpaces the model’s intrinsic noise-filtering capacity.Its utility depends on the system’s baseline capability.Source: §7 · p. 10
  • Weak retrievers leave masking without evidence to expose, while saturated models risk losing critical information they could otherwise exploit.These failure modes limit masking’s usefulness outside the intermediate regime.Source: §7 · p. 10
  • Context management must be jointly calibrated with retriever recall and model capacity rather than treated as an isolated default.The authors propose their multi-regime framework as a guide for evaluating broader context-management strategies and designing future agentic systems.Source: §7 · p. 10

LIMITATIONS

The findings are robust across benchmarks, but the study isolates a minimal masking policy and excludes frontier proprietary models. Its regime framework remains descriptive rather than predictive.

LIMITATIONS

  • The minimal turn-based observation masking policy leaves open whether adaptive policies could preserve model-retriever mismatch gains while avoiding model-saturated collapse.Untested alternatives include learned, attention-guided, and semantic-adaptive policies.Source: LIMITATIONS · p. 11
  • The regime framework is descriptive rather than predictive, and its boundaries require testing across different scaffolds, downstream tasks, and retrieval interfaces.Live-web replication suggests the pattern is not specific to BrowseComp-Plus, but transfer to tasks such as SWE code localization remains untested.Source: LIMITATIONS · p. 11
  • 4B to 284B open-weight backbones are evaluated, but computation, inference, and attention analysis constraints leave frontier proprietary models outside the study.Model coverage is broad but not exhaustive.Source: LIMITATIONS · p. 11

ETHICAL CONSIDERATIONS … B EVALUATION DETAILS

Long-horizon search accumulates observation-heavy context, motivating context management while raising trade-offs over evidence retention and computational cost. Efficiency gains also introduce misuse and reliability risks, making human oversight and domain-specific validation essential.

ETHICAL CONSIDERATIONS

  • High benchmark performance does not guarantee high-stakes reliability: context management cannot fix backbone hallucinations or reasoning failures, so human oversight and domain-specific validation remain essential.The framework is intended to augment human-in-the-loop analysis rather than replace it, particularly in medicine, law, and finance.Source: ETHICAL CONSIDERATIONS · p. 11
  • Context management can suppress ethical disclaimers or edge-case safety warnings, potentially reinforcing model biases through the context that remains.The remaining context may include both explored evidence and the model’s self-generated material.Source: ETHICAL CONSIDERATIONS · p. 11
  • Cheaper long-horizon web research can broaden access while also enabling lower-cost automated generation of low-quality or misleading content.Adaptive history compression improves efficiency for budget-constrained models, creating both accessibility benefits and potential misuse.Source: ETHICAL CONSIDERATIONS · p. 11

A MORE RELATED WORK

  • Accumulated tool outputs, errors, artifacts, and environmental states make context windows increasingly costly and difficult-to-use working memory over dozens or hundreds of interaction turns.Agents must preserve coherent task state, revise plans using feedback, and connect intermediate observations to later decisions.Source: A MORE RELATED WORK · p. 16
  • Observation-heavy search histories risk context dilution and lost-in-the-middle failures (Liu et al., 2024), motivating tests of when masking stale observations preserves downstream evidence use.Repeated search, page opening, and evidence localization append snippets, pages, spans, and tool feedback to the trajectory.Source: A MORE RELATED WORK · p. 16
  • Static pruning and compaction shorten context but may permanently remove long-range dependencies.Post-generation trajectory retrieval instead parses or re-searches completed execution traces for critical evidence (Lee et al., 2026; Zhou et al., 2026).Source: A MORE RELATED WORK · p. 16
  • AgentFold (Ye et al., 2025) and ReSum (Wu et al., 2025) compress context online but introduce additional computational or tool-calling costs.AgentFold relies on non-native tool-calling loops; ReSum uses step-by-step rolling summarization that increases intermediate computation and can reduce KV-cache reuse.Source: A MORE RELATED WORK · p. 16

B.1 BENCHMARKS · B.2 RETRIEVERS • · B.3 COMPARED BASELINES •

The evaluation combines fixed-corpus and open-web benchmarks with sparse, dense, and reasoning-aware retrievers. Compared baselines include open-weight foundation models, reasoning models, and a specialized deep-research model.

B.1 BENCHMARKS

  • 830 BrowseComp-Plus (Chen et al., 2025c) examples evaluate multi-document retrieval and synthesis using a fixed corpus with human-verified supporting documents and hard negatives.The offline search engine uses the officially released corpus with its BM25 and Qwen3-Embedding-8B FAISS index, eliminating reliance on live web access.Source: §B.1 · pp. 16–17
  • 289 BrowseComp-ZH (Zhou et al., 2025) questions across 11 domains test Chinese-web multi-hop search and cross-source reconciliation.Each question is reverse-engineered from a short, objective, verifiable answer to ensure answer uniqueness and facilitate automatic evaluation.Source: §B.1 · p. 17
  • 103 text-only GAIA (Mialon et al., 2024) and 100 xbench-DeepSearch (Chen et al., 2025a) examples assess reasoning, tool use, and sustained multi-turn web research.GAIA uses a text-only subset, following Team et al. (2025b;a); xbench-DeepSearch uses its full set and tests question decomposition, iterative evidence gathering, and grounded synthesis.Source: §B.1 · p. 17

B.2 RETRIEVERS •

  • BM25 supplies lexical retrieval, while Qwen3-Embedding-8B (Zhang et al., 2025b) supplies primary dense retrieval on BrowseComp-Plus.BM25 uses lexical matching and term-frequency weighting; Qwen3-Embedding-8B is an open-weight embedding model.Source: §B.2 · p. 17
  • AgentIR-4B (Chen et al., 2026) jointly embeds reasoning traces and search queries to incorporate the agent’s intent and multi-turn context into retrieval.The experiments use its released BrowseComp-Plus index and append the current turn’s reasoning to the search query, following its reported best-performing configuration.Source: §B.2 · p. 17

B.3 COMPARED BASELINES •

  • Qwen3.5/3.6 models (Team, 2026; QwenTeam, 2026b) serve as competitive open-weight deep-research baselines with parallel browser-tool access.Their hybrid architecture combines Gated Delta Networks with sparse mixture-of-experts routing and supports up to 1M input tokens.Source: §B.3 · p. 17
  • GPT-OSS models (Agarwal et al., 2025) provide open-weight reasoning baselines evaluated through the same browser-tool interface.The family includes GPT-OSS-120B and GPT-OSS-20B with a 128K input context window, adjustable reasoning effort, and tool-use capabilities.Source: §B.3 · p. 17
  • DeepSeek-V4 and specialized OpenResearcher-30B-A3B (Li et al., 2026a) are evaluated as open-weight baselines with parallel browser-tool access.Both support a 1M input context window; DeepSeek-V4 targets long-horizon tool use. OpenResearcher is trained through a fully open trajectory-synthesis pipeline.Source: §B.3 · pp. 17–18

C EXPERIMENT SETTINGS

Experiments use size-dependent tensor parallelism, native model-specific tool templates, and fixed decoding settings. Human verification of sampled LLM judgments reports over 99.9% agreement.

Model serving

  • Eight NVIDIA H100 GPUs serve all backbone models, with tensor parallelism scaled to model size.TP=1 serves 4B/9B models; TP=2 serves 20B/30B/35B; TP=4 serves 120B; and TP=8 serves DeepSeek-V4-Flash-Max.Source: C EXPERIMENT SETTINGS · p. 18

Tool-call formatting

  • Models use their native chat and tool-calling templates rather than a shared synthetic format, avoiding penalties from template mismatch.Tool specifications and observations are injected through model-specific templates whenever available.Source: C EXPERIMENT SETTINGS · p. 18

Generation configuration and evaluation

  • 8192 output tokens per turn and temperature 1.0 are the default generation settings, held fixed across model–retriever configurations.Default search top-k is 15, and default open lines is 50. Human verification of a random 15% sample of LLM-judged experimental results achieved over 99.9% agreement.Source: C EXPERIMENT SETTINGS · p. 18

D REGRESSION PROBE: EXPERIMENTAL SETUP … D.6 LIVE-WEB XBENCH PROXY

The probe estimates SNR from No-CM trace-shape features, while a separate AUC measures the separability of its two-dimensional visualization rather than regression quality. Live-web xBench lacks gold-document qrels, so it uses a weaker citation-based proxy.

D.1 BROWSECOMP-PLUS TRACES AND LABELS

  • Five BrowseComp-Plus model–retriever settings supply gold-document probes, with all features and SNR targets computed exclusively from No-CM trajectories.Settings are GPT-OSS-120B+AgentIR, Qwen3.5-4B+AgentIR, Qwen3.5-9B+Qwen3-Emb-8B, DS-V4-Flash-Max+AgentIR, and Qwen3.5-4B+BM25. Gold URLs come from BrowseComp-Plus qrels using the recall evaluator’s exact-match URL normalization.Source: §D.1 · p. 18

D.2 PREFIX SAMPLING

  • At most 8 prefixes per trajectory are sampled, always retaining the final evidence state.All prefixes are retained when T ≤ 8; otherwise, 7 are sampled uniformly from the first T − 1 turns and the final prefix is added. This produces roughly 6.5k prefix examples per 830-query configuration.Source: §D.2 · pp. 18–19

D.3 INPUT FEATURES

  • Trace-shape features describe trajectory progression, URL accumulation, observation length, tool-call type, and trajectory budgets without using model hidden states.Inputs include normalized prefix position, log-transformed cumulative/new URL counts and observation lengths, URL growth rate, search/open indicators, final URL budget, and log-transformed trajectory turn count.Source: §D.3 · p. 19

D.4 REGRESSION MODEL AND TWO-DIMENSIONAL PROJECTION

  • Configuration-specific SNR probes use standardized trace features, second-degree polynomial interactions, and ridge regression with query-grouped cross-validation.Regularization α is cross-validated over {10^-4, …, 10^4}, and predictions are clipped to [0, 1]. Grouping by query prevents prefixes from the same query appearing in both training and test folds.Source: §D.4 · p. 19
  • Min–max-normalized fitted SNR supplies the vertical coordinate, while the first principal component serves as an unsupervised input-complexity proxy.The first principal component strongly correlates with trace scale, with dominant positive loadings on cumulative turns and log observation characters.Source: §D.4 · p. 19

D.5 AUC COMPUTATION

  • Scatter-plot AUC measures linear separability of the displayed coordinates, not SNR-regression quality.A balanced logistic probe uses query-grouped cross-validation; reported AUC macroaverages held-out-fold Mann–Whitney ROC-AUC values. CM-rescue positives are No-CM-wrong → CM-correct queries; BrowseComp-Plus negatives are all non-rescued prefixes, whereas xBench excludes degraded trajectories and uses unchanged prefixes.Source: §D.5 · p. 19

D.6 LIVE-WEB XBENCH PROXY

  • Live-web xBench uses a weaker citation proxy because its traces lack gold-document qrels needed for true gold-document SNR.The target is the fraction of cumulatively observed browser lines cited by the final No-CM answer: |L_q,t ∩ C_q| / max(|L_q,t|, 1). The same grouped regression and two-dimensional separability procedure is applied to this proxy.Source: §D.6 · p. 20

E ATTENTION ANALYSIS: EXPERIMENTAL SETUP … G MORE EXPERIMENT RESULTS

The attention analysis measures how browsing agents allocate attention across past turns and roles using matched trajectories from three model–retriever settings. The prompt templates define browsing operations, parallel tool calls, answer formatting, and correctness judging.

E.1 MODELS AND TRAJECTORIES

  • 150 matched BrowseComp-Plus (Chen et al., 2025c) trajectories per setting compare three model–retriever combinations, all using parallel tool calls.The combinations are Qwen3.5-4B/BM25, Qwen3.5-9B/AgentIR (Chen et al., 2026), and Qwen3.6-35B-A3B/AgentIR. Every fourth layer uses full attention: eight layers in the 4B/9B models and ten in the 35B model.Source: §E.1 · p. 20

E.2 CAPTURING QUERY/KEY TENSORS WITH HOOKED VLLM

  • A single chunked-prefill pass through hooked vLLM (Kwon et al., 2023) captures query/key tensors for attention computation at every reasoning-start token.Forward hooks (Ko & Chen, 2026) record every token position; causal masking prevents later tokens from affecting earlier queries and keys. The 35B-A3B run uses two H100 GPUs, with head shards concatenated offline.Source: §E.2 · p. 20

E ATTENTION ANALYSIS: EXPERIMENTAL SETUP / E.3 PER-TURN ATTENTION SCORES

  • Attention at each reasoning-start token is averaged across layers and heads, then summed over preceding tokens by interaction turn and role.Chat-template markers identify system, user, reasoning, tool-call, and observation segments. Each turn groups reasoning with its following tool-call/observation pair; the first also includes the initial system and user segments.Source: §E.3 · pp. 20–21, E ATTENTION ANALYSIS: EXPERIMENTAL SETUP · p. 20

E.4 PER-TRAJECTORY HEATMAP

  • Per-trajectory heatmaps separate reasoning attention into a blue lower triangle and transposed observation attention into an orange upper triangle.The diagonal is masked and self-attention excluded from color-limit selection because current-step attention dominates off-diagonal values. The illustrated example is a 50-turn 9B-AgentIR trajectory.Source: §E.4 · p. 21

E.5 AGGREGATED DECAY CURVES

  • Aggregated decay curves report mean role-specific attention and cumulative attention share across normalized past-turn positions.Across 150 trajectories per setting, positions enter 20 equal-width bins, using only reasoning steps with at least five past turns. Cumulative shares remain below 100% because system, user, and current-self attention consume additional attention mass.Source: §E.5 · p. 21

F PROMPT TEMPLATES / F.3 TOOL PROMPTS / F.4 PARALLEL TOOL-CALL INSTRUCTION / Parallel Tool-Call Instruction

  • The tool scaffold combines search, opening links or URLs, scrolling, and exact-pattern finding with model-specific instructions for parallel calls.Independent calls can appear as consecutive tool-call blocks in one response. The scaffold recognizes either multiple function calls in one assistant turn or a Tongyi DeepResearch search containing multiple queries as parallel.Source: §F.3 · pp. 22–23, §F.4 · p. 23, Parallel Tool-Call Instruction · p. 23

F PROMPT TEMPLATES / System Prompt / F.5 LLM-AS-JUDGE PROMPT

  • The agent prompt requests interleaved search and reasoning, cited explanations, exact answers, and confidence; the judge checks final-answer agreement rather than solving questions.The judge permits a small numerical margin of error but rejects inconsistency, ambiguity, or non-equivalence. It extracts reported confidence and defaults to 100 when confidence is absent.Source: System Prompt · p. 22, §F.5 · p. 23

G.1 MAIN RESULT ANALYSIS · H CASE STUDY

Masking helps most when retrieval supplies evidence that the model cannot reliably separate from noise; benefits shrink under retrieval bottlenecks or model saturation and can become negative. Masking adds tool calls across regimes, but those costs do not consistently improve accuracy.

G.1 MAIN RESULT ANALYSIS

  • +6.3, +6.6, and +6.2 masking gains for Qwen3.5-4B, Qwen3.5-9B, and GPT-OSS-20B under BM25 form a near-constant plateau despite differing model capabilities.Recall never exceeds 0.54 and absolute accuracy stays below 39%, making retrieval—not model capability—the limiting factor.Source: §G.1 · pp. 23–24
  • +11.7 is the peak accuracy gain over No-CM, achieved by Qwen3.5-35B-A3B+AgentIR within a non-monotonic, three-regime pattern.The mismatch regime combines recall in the high 0.7–0.9 range with No-CM accuracy of 45–65%. Once No-CM accuracy crosses approximately 70%, gains collapse, reaching +0.1 for GPT-OSS-120B and −1.1 for Tongyi-DeepResearch.Source: §G.1 · p. 24
  • Retriever–model mismatch, rather than parameter count, determines the regime: similarly sized backbones show substantially different masking gains.Qwen3.5-35B-A3B gains +11.7 versus Qwen3.6-35B-A3B’s +3.7 despite sharing architecture and parameter count and differing only in training recipe. At approximately 30B scale, OpenResearcher-30B-A3B gains +2.6 while Tongyi-DeepResearch loses −1.1, supporting the importance of training for selective context use.Source: §G.1 · p. 24
  • 68.7 additional calls per query buy GPT-OSS-120B only +0.1 accuracy gain, illustrating masking’s universal call cost but regime-dependent benefit.Every populated configuration adds tool calls, ranging from 20.3 for Qwen3.5-9B+AgentIR to 91.6 for GPT-OSS-20B+Qwen3-Emb-8B. Clearing stale observations enables requerying and rereading, increasing recall rather than shortening trajectories.Source: §G.1 · p. 24
  • Across three additional benchmarks, weaker Qwen backbones retain masking benefits while DS-V4-Flash-Max remains saturated.Qwen3.5-9B gains +4.8 on GAIA-text and +7.0 on xBench, while Qwen3.5-4B gains +3.4 on BrowseComp-ZH. This reproduces the regime ordering beyond BrowseComp-Plus.Source: §G.1 · p. 24
  • +8.0 on xBench versus −4.8 on GAIA-text for GPT-OSS-120B shows that masking’s regime depends on task-specific base accuracy, not model identity alone.Its 70.0% xBench base accuracy places it back in the mismatch band, whereas higher GAIA-text base accuracy leads to over-pruning. The GAIA-text loss is the largest negative effect observed, consistent with masking removing useful evidence from capable readers.Source: §G.1 · p. 24
  • +1.9 on GAIA-text contrasts with +6–+11 across BrowseComp-Plus retrievers for Qwen3.5-4B, exposing the limits of masking under noisy live-web retrieval.The authors attribute the smaller gain to noisier, harder-to-use context and the 4B reader’s inability to exploit surfaced evidence. Weak retrieval quality and weak reader capability both push configurations toward low-gain regimes.Source: §G.1 · p. 24

H.1 CASE: PARALLEL TOOL USE ON BROWSECOMP-PLUS … H.5 CASE: BROWSECOMP-PLUS — CM IMPROVES (CM OFF WRONG →CM ON CORRECT)

These cases examine parallel browser use and observation masking in extended search investigations, including BrowseComp-Plus. They illustrate how masking can sustain citation gathering and exploration within context limits, but can also turn correct answers wrong by removing decisive or corrective evidence.

H.1 CASE: PARALLEL TOOL USE ON BROWSECOMP-PLUS

  • Recognizing that the 2001 book and 2012 archaeological publication were separate documents redirected the search after unsuccessful exact-phrase probes.The agent followed Jill Hamel’s charitable-trust trusteeship to the Hereweka–Harbour Cone Trust, then located Angela Middleton’s 2012 paper.Source: §H.1 · p. 26
  • 404 hectares was the final answer for William Larnach’s consolidated holding at his death in 1898.Qwen3.5-9B with Qwen-3-Embed-8B used browser.search, browser.open, and browser.find, including parallel calls. Targeted document views verified Larnach’s biographical context.Source: §H.1 · pp. 25, 27, 28

H.2 CASE: TONGYI-DEEPRESEARCH-30B-A3B — PARALLEL SEARCH IN ARRAY QUERY

  • DeepResearch-30B-A3B with AgentIR was judged correct while conducting a parallel-search investigation under observation masking.Five parallel tool calls in the first turn identified Brazuca as a candidate; 38 in the second pursued verification of its director’s background. Earlier results were subsequently replaced with archival placeholders.Source: §H.2 · pp. 29–30
  • The agent answered Brazuca with 100% confidence despite failing to obtain clean independent corroboration that its editor, Yorgos Georgopoulos, also edited the 2016 murder-solving film The Other Me.A 38-call parallel salvo committed the model to Brazuca, followed by 22 rounds checking subsidiary clues. Festival-page evidence confirmed the director’s background and identified Yorgos Georgopoulos as Brazuca’s editor.Source: §H.2 · pp. 31–33

H.3 CASE: XBENCH-DEEPSEARCH — GPT-OSS-120B

  • 62 browser results were auto-archived during GPT-OSS-120B’s correct xbench-DeepSearch run, sustaining extended citation gathering without context overflow.The model already had the answer—Wei Zheng had four sons—before spending roughly fifty further rounds securing exact supporting source lines.Source: §H.3 · pp. 34, 36

H.4 CASE: BROWSECOMP-PLUS — CM IMPAIRS (OFF CORRECT →CM ON WRONG)

  • GPT-OSS-120B with AgentIR answered correctly without masking but incorrectly with masking on BrowseComp-Plus after decisive date evidence was archived.Without masking, the agent preserved June 2023 as the relationship-end month. With masking, August 2023 announcement evidence dominated the remaining context, producing the wrong answer.Source: §H.4 · pp. 37–38
  • CM-off correctly answered June 2023 for the relationship-end question, whereas CM-on incorrectly answered August 2023.Both final answers reported 97% confidence.Source: §H.4 · pp. 39, 41
  • Auto-archiving removed unopened NTV snippets distinguishing the June breakup from its August announcement, leaving announcement evidence that the agent misinterpreted as the relationship-end date.Later confirmation searches were mostly masked; the final live breakup evidence came only from the Citizen Digital announcement article.Source: §H.4 · p. 40

H.5 CASE: BROWSECOMP-PLUS — CM IMPROVES (CM OFF WRONG →CM ON CORRECT)

  • Observation masking recovered the correct Sankomota / Frank Leepa answer, while the CM-off baseline fixated on Nass el Ghiwane and incorrectly answered Ali Benfarha.The CM-on run completed 36 rounds with 32 auto-archive events, versus 24 rounds without CM.Source: §H.5 · pp. 42, 45
  • Repeated retrieval of 2023 rather than 2003 deaths hindered the CM-off search, while bounded context let the masked run continue exploring non-English sources.The baseline pursued Nass el Ghiwane without verifying that its proposed guitarist satisfied the 2003-death clue.Source: §H.5 · pp. 42–44
  • 30 turns, spanning rounds 2–31, pursued incorrect band candidates while search results repeatedly favored 2023 deaths over the required 2003.Wrong leads included Osibisa, AC/DC, Pink Floyd, U2, and The Bricks; candidates conflicted with clues about instrument, death year, or formation date.Source: §H.5 · p. 45
  • 32 older browser results were archived, keeping context bounded so the agent could continue pivoting between candidate bands without overflow.Archiving occurred during the prolonged exploration preceding the Sankomota breakthrough.Source: §H.5 · p. 46
  • With context management enabled, the agent ultimately identified Frank Leepa, founder of Sankomota, as its final answer.Rounds 33–35 verified his role as guitarist and frontman and his death on 27 November 2003; by turn 36, the agent had settled on his identity.Source: §H.5 · pp. 46–47

Related papers

Loading 2606.00408v1…