PaperlayerView exact revision on arXiv

Source-linked AI summary

LongHarness: Advancing Agentic Reasoning over 100M-Token Corpora via Stateful Evidence Management

Songtao Huang, Zhijie Zhong, Xiaohan He, Shuaiyu Zhang, Zhiyin Yu, Li Kang, Heng Zhou, Yutao Fan, Xiangchao Yan, Bo Zhang, Shiyang Feng

arXiv:2610.13075v1cs.AI

TL;DR

Corpus-level reasoning must integrate evidence across collections that exceed context windows, but existing harnesses can leave evidence and evolving conclusions implicit in interaction history. LongHarness couples conclusion-guided exploration with evidence lifecycle management; on OfficeQA Pro, GLM-5.3-Flash beats Claude Code by 12.0 percentage points on V1 and Codex by 18.9 points on V2.

  • Problem

    Existing agent harnesses support incremental document access, but evidence and evolving conclusions can remain implicit in session history, leaving corpus-level reasoning with unresolved information gaps and disconnected evidence.

    Source: §1 · p. 2

  • Method

    LongHarness couples persistent source-linked evidence and conclusions, using unresolved conclusion gaps to guide exploration and accepted evidence to revise conclusions.

    Source: §1 · p. 2

  • Results

    12.0 percentage points over Claude Code on OfficeQA Pro V1 and 18.9 points over Codex on V2 with GLM-5.3-Flash.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Explicit information gaps focus retrieval on decision-relevant needs, while lifecycle management keeps evidence current and conclusions aligned with it.

    Source: §1 · p. 2

    Evaluation reports GLM-5.3-Flash variability across three runs; missing answers and agent termination failures count as incorrect.

    Source: §3.1 · p. 6

Abstract

from arXiv · show

Corpus-level long-context reasoning requires agents to acquire and integrate evidence across document collections that exceed their context windows. Existing agent harnesses enable incremental access to these corpora, but acquired evidence and evolving conclusions often remain entangled in a growing interaction history. This implicit state management can lead to two coupled failure modes: unbounded exploration, where retrieval expands the interaction history without resolving decision-relevant information gaps, and evidence-state drift, where previously acquired evidence or subsequent corrections fail to inform the current conclusion. We present LongHarness, a model-agnostic harness that makes the interaction between acquired evidence and evolving conclusions explicit. Its core mechanism, Evidence-Conclusion Coevolution, uses unresolved conclusions to guide exploration toward missing information, while newly acquired evidence strengthens or revises the conclusions that directed the search. LongHarness couples this process with Evidence Lifecycle Management, which keeps accumulated evidence current and traceable throughout the interaction. Together, these mechanisms focus retrieval on remaining information needs and keep conclusions aligned with current evidence. We evaluate LongHarness on OfficeQA Pro V1 and V2, AA-LCR, and LongBench v2 using GLM-5.3-Flash and Qwen3.8-Flash. Compared with direct reasoning using the same backbone, LongHarness improves GLM-5.3-Flash exact match on OfficeQA Pro from 15.8% to 60.2% on V1 and from 4.4% to 61.1% on V2. Against state-of-the-art agent harnesses, it surpasses Claude Code by 12.0 percentage points on V1 and Codex by 18.9 points on V2. With Qwen3.8-Flash, LongHarness reaches 61.7% on V1, outperforming Claude Code by 8.3 points and Codex by 12.0 points.

1 Introduction

Corpus-level reasoning must locate and integrate evidence across collections larger than a model’s context window, while keeping that evidence usable as conclusions evolve. LongHarness makes evidence and conclusions explicit, linking unresolved information needs to retrieval and new evidence to conclusion revision; its evaluations report gains over direct reasoning and agent harnesses.

1 Introduction

  • Corpus-level questions require agents to locate and integrate evidence scattered across collections that may exceed their context windows.Longer context windows expand available material, but models remain sensitive to evidence position and distractors.Source: §1 · p. 2
  • When conclusions remain implicit in session history or free-form notes, retrieval can continue without resolving information gaps and evidence can become disconnected from later reasoning.These failure modes can reinforce one another, making relevant evidence harder to recover.Source: §1 · p. 2
  • LongHarness maintains evidence and conclusions as coupled persistent states, using unresolved gaps to guide search and accepted evidence to strengthen or revise conclusions.Its EvidenceBank keeps source-linked statements active and traceable, helping align subsequent reasoning with current evidence.Source: §1 · pp. 2–3
  • The evaluation spans corpus-level reasoning over roughly 100M-token and doubled document collections, as well as provided-context benchmarks, using two backbones.OfficeQA Pro V2 doubles the V1 corpus and introduces more demanding compositional questions.Source: §1 · p. 2
  • 12.0 percentage points over Claude Code on OfficeQA Pro V1 and 18.9 points over Codex on V2 with GLM-5.3-Flash; Qwen3.8-Flash also outperforms both on V1.The evaluation covers OfficeQA Pro, AA-LCR, and LongBench v2 across two model backbones.Source: §1 · pp. 2–3

2 Method

LongHarness separates persistent evidence from evolving conclusions and connects them through adaptive evidence acquisition and revision. Its lifecycle controls and shared agent–environment interface keep evidence source-linked and active across interactions.

2.1 Overview

  • The policy acts on question, document map, persistent evidence, conclusions, and remaining budget; the harness validates action structure, source references, and budget constraints.The shared interface supports adaptive interleaving of evidence acquisition, maintenance, computation, and final submission.Source: §2.1 · p. 3

2.2 Evidence–Conclusion Coevolution

  • Evidence–Conclusion Coevolution uses conclusion gaps to direct searches and reading, then reassesses claims against active evidence and updates affected claims, references, and gaps.The agent identifies observations that could establish, qualify, or refute claims; before submission it checks whether remaining gaps could change the decision.Source: §2.2 · pp. 3–4, Figure 2 · p. 4

2.3 Evidence Lifecycle Management

  • Evidence Lifecycle Management admits source-linked statements, keeps them available across interactions, and explicitly revises or retires them as supporting evidence changes.Corrected statements become new active entries while prior entries are marked superseded; retracted statements leave the active state without replacement.Source: §2.3 · pp. 4–5

2.4 Agent–Environment Interaction

  • The shared action space lets the policy search, read, update evidence, compute, and submit answers with supporting references under controller-enforced protocol and resource constraints.Superseding or retracting evidence does not automatically rewrite dependent conclusions; the policy must revise affected claims after observing updates, and final references must resolve to active evidence and source passages.Source: §2.4 · p. 5

3 Experiments

Experiments compare LongHarness with direct reasoning and other agent harnesses across corpus-level and provided-context benchmarks, then examine component ablations and interaction trajectories. LongHarness outperforms competing harnesses on the reported benchmark settings, while the analyses probe evidence use and retrieval behavior.

3.1 Experimental Setup

  • Evaluation spans OfficeQA Pro, AA-LCR, and LongBench v2, comparing harnesses on matched GLM-5.3-Flash and Qwen3.8-Flash backbones.OfficeQA Pro V1 contains 697 documents totaling approximately 100M tokens, while V2 expands to 1,435 documents; the main comparison uses one run per harness and setting.Source: §3.1 · pp. 5–6

3.2 Main Results

  • LongHarness exceeds the strongest harness by 12.0 percentage points on OfficeQA Pro V1 and 18.9 points on V2 with GLM-5.3-Flash.Against direct reasoning, GLM-5.3-Flash exact match rises from 15.8% to 60.2% on V1 and from 4.4% to 61.1% on V2. On LongBench v2 Long, it exceeds the next-best harness by 6.5 points with GLM-5.3-Flash and 7.4 with Qwen3.8-Flash; on AA-LCR, the leads are 5.0 and 4.0 points.Source: Table 1 · p. 6, §3.2 · p. 6

3.4 Interaction Length and Trajectory Analysis

  • On V2, LongHarness records 87.8% retrieval, 87.3% conditional evidence adoption, and 79.7% conditional answer conversion; adoption and conversion remain close to V1.Editing and transformation rates are conditional on passing the preceding stage.Source: Table 3 · p. 7, §3.4 · p. 8

3.3 Ablation Study

  • On V1/V2, removing persistent states costs 4.5/7.8 exact-match points; append-only evidence yields 56.4%/51.1%, and every ablation loses more on V2.Removing Conclusion State feedback costs 5.3/9.4 points, removing explicit gap-guided retrieval instructions costs 4.5/7.8, and replacing structured conclusions with generic notes costs 6.8/7.8.Source: §3.3 · p. 7, Table 2 · p. 7

3.4 Interaction Length and Trajectory Analysis

  • LongHarness’s median turns rise from 22 to 30 between V1 and V2, versus 21 to 116 for Claude Code and 19 to 43 for Codex.In a Treasury example, LongHarness incorporates a recognized correction and returns 4962.46 million dollars, while Codex omits the correction.Source: Figure 3 · p. 8, Figure 4 · p. 8, §3.4 · p. 9

4 Related Work

Prior long-context work strengthens evidence acquisition and maintenance, while agent harnesses provide adaptive access and persistent storage. LongHarness instead couples evidence and conclusions at runtime across retrieval, editing, and transformation.

4.1 Long-Context Reasoning

  • LongHarness couples evidence and conclusions at runtime, linking retrieval, editing, and transformation without additional policy training.Prior work includes supervision of retrieval trajectories and memory edits, while LongHarness makes the evidence-conclusion transition explicit during interaction.Source: §4.1 · p. 9

4.2 Agentic Harnesses

  • Existing harnesses support adaptive document access and persistent storage, but LongHarness also maintains evolving conclusions and unresolved information needs to guide search.Newly acquired evidence informs revisions to conclusions and remaining gaps.Source: §4.2 · p. 9

5 Conclusion

LongHarness unifies evidence retrieval, editing, and transformation through coupled conclusion revision and evidence lifecycle management. Evaluations show gains across three benchmarks and two backbones, alongside complementary ablation findings and trajectory patterns.

5 Conclusion

  • 12.0 percentage points on OfficeQA Pro V1 and 18.9 points on V2 are LongHarness’s margins over Claude Code and Codex, respectively, with GLM-5.3-Flash.Evaluations on OfficeQA Pro, AA-LCR, and LongBench v2 show gains with two backbones.Source: §5 · p. 9
  • Evidence–Conclusion Coevolution couples exploration with conclusion revision, while Evidence Lifecycle Management keeps evidence current and traceable.The harness unifies evidence retrieval, editing, and transformation for corpus-level long-context reasoning.Source: §5 · p. 9
  • Ablations support complementary roles for persistent states, conclusion-guided retrieval, and evidence revision.Source: §5 · p. 9
  • Trajectory analysis finds similar conditional evidence-use rates across V1 and V2 and a smaller interaction-length increase than competing harnesses.Source: §5 · p. 10

A Benchmark Details

The benchmark suite covers external-corpus retrieval in OfficeQA Pro and reasoning over provided contexts in AA-LCR and LongBench v2. The settings vary in corpus scale, question demands, and scoring.

A Benchmark Details

  • The four evaluation settings cover external-corpus retrieval in OfficeQA Pro and provided-context reasoning in AA-LCR and LongBench v2.Source: A Benchmark Details · p. 13

OfficeQA Pro

  • OfficeQA Pro V1 has 133 questions over 697 Treasury documents totaling about 100M tokens; V2 has 90 questions over 1,435 documents and emphasizes composite numerical questions.Both versions expose the full corpus without question-specific gold documents; scores include exact match and accuracy within 5% relative error.Source: A Benchmark Details · p. 13

AA-LCR-1.1

  • AA-LCR-1.1 contains 100 questions across 30 document sets, averaging 99,325 tokens per set, and tests cross-document synthesis, temporal reasoning, conditional logic, or numerical computation.Each question supplies its relevant document set, and direct-reasoning scores use Artificial Analysis’s published official results.Source: A Benchmark Details · p. 13

LongBench v2

  • The evaluated LongBench v2 Long subset has 108 questions with contexts from about 132K to 1.38M words and measures exact accuracy on four-option answers.Its questions span code-repository understanding, long in-context learning, single- and multi-document QA, and structured-data understanding.Source: A Benchmark Details · p. 13

B Method Implementation Details

The implementation specifies an interaction interface, EvidenceBank operations, a persistent-state contract, and evaluation resource settings. Its controller validates actions and references while the policy manages evidence and conclusions.

B Method Implementation Details

B.1 Interaction Interface

  • Search returns candidate locations and snippets, while read returns source text and registered references; proposed EvidenceBank statements must cite references registered by read.The policy selects one operation per turn, and the controller validates its structure, references, and resource requirements.Source: §B.1 · pp. 13–14, Table 5 · p. 14

B.2 EvidenceBank Operations

  • EvidenceBank upsert admits validated source-linked statements, while revise replaces or retracts active entries and preserves prior versions in the revision record.Superseded and retracted entries cannot be cited in compute or final requests.Source: §B.2 · p. 14, Table 6 · p. 14

B.3 State Contract

  • Conclusion State stores established claims, supported hypotheses, and unresolved gaps linked to active EvidenceBank entries, separating inferences from source-linked statements.When evidence becomes inactive, logs flag affected references, but the runtime does not automatically rewrite those claims or block further search.Source: §B.3 · p. 14, Table 7 · p. 15

B.4 Runtime Settings

Runtime state combines the interaction history with explicit evidence, conclusion, and budget records. The interaction algorithm preserves state across turns, updates evidence from observations, and accepts a final answer only when it is confirmed.

B.4 Runtime Settings

  • The policy receives the full interaction history alongside an EvidenceBank, Conclusion State, and remaining resource budgets.Historical reasoning, actions, and observations are retained in full; the cumulative read budget counts source text returned by read operations.Source: §B.4 · p. 15
  • Each model request remains subject to the endpoint context limit, and a persistent request failure terminates the episode as incorrect.The client does not automatically truncate or compact interaction history.Source: §B.4 · p. 15

B.5 Interaction Algorithm

  • Conclusion updates merge supplied lists while preserving omitted lists, and an explicitly empty list clears its corresponding field.Algorithm 1 describes the interaction process; evidence updates are observed before conclusion revision, and a confirmed accepted submission returns the answer and resolved source references.Source: §B.5 · pp. 15–16

C.1 Benchmark Preparation

The benchmarks expose either external corpora or question-provided contexts, with agent-facing inputs withholding reference answers and gold evidence. LongHarness uses a shared evidence-grounded interaction policy, while harness interfaces and prompt instructions govern how agents search, retain evidence, revise conclusions, and submit answers.

C.1 Benchmark Preparation

  • OfficeQA Pro exposes its full V1 or V2 corpus, while AA-LCR and LongBench v2 provide question-specific document collections or contexts.Reference answers, gold evidence annotations, and question-specific document labels are withheld from agents.Source: §C.1 · p. 16
  • LongHarness preserves document structure, splits oversized paragraphs into blocks of at most 1,024 estimated tokens, and supports lexical or regex search followed by source-linked reads.Search uses BM25-style ranking; read operations return source text and registered source-span identifiers.Source: §C.1 · pp. 16–17

C.2 Harness Configuration

  • Harnesses use the same questions, available documents, and backbone model, but differ in interaction interfaces; workspace access is isolated and network access is disabled.Table 8 summarizes the available operations and resource limits for direct reasoning and agent harnesses.Source: §C.2 · p. 17, Table 8 · p. 17
  • The evidence-pipeline audit separates retrieval, evidence adoption, and answer conversion, with adoption and conversion rates conditioned on passing earlier stages.Correctness uses the benchmark scorer at tolerance 0; correct episodes count as passing the first two stages by protocol.Source: §C.2 · p. 17

C.3 LongHarness Prompt Template

  • EvidenceBank statements must be atomic and source-faithful, and invalid accepted statements can be replaced or withdrawn.The agent revises conclusions when evidence supports, qualifies, contradicts, or replaces earlier reasoning.Source: §C.3 · p. 18
  • For computation and final submission, the prompt requires source-linked operands, verification of direction, unit, and precision, and active EvidenceBank citations.The final answer is reconstructed from current established conclusions and their sources.Source: §C.3 · p. 18
  • The Conclusion State tracks established conclusions, supported hypotheses, and unresolved gaps; the agent searches for the gap most likely to change the answer.Continue only while resolving a remaining gap could change the result.Source: §C.3 · p. 18

D Repeated-Run Results

The repeated-run results report three GLM-5.3-Flash runs per harness and evaluation setting, with summary statistics derived from those scores.

D Repeated-Run Results

  • Three runs are reported for each harness and evaluation setting with GLM-5.3-Flash.Table 9 presents individual run scores as well as summary statistics.Source: D Repeated-Run Results · p. 19
  • The reported summary statistics are computed from the three scores for each harness and evaluation setting.Source: D Repeated-Run Results · p. 19
  • Standard deviation is the sample standard deviation, and Max. is the highest score across the three runs.The standard deviations are expressed in percentage points.Source: D Repeated-Run Results · p. 19

E Case Study

The case study contrasts Codex’s repeated backward reads and stale computation with LongHarness’s incorporation of corrections. LongHarness reaches the reference answer in 10 policy actions, while Codex’s answer misses exact match despite passing the 5% tolerance criterion.

E Case Study

  • Codex makes 103 consecutive backward reads after locating the table, then retains an earlier result after recognizing the corrected 4400 value.The trajectory illustrates unbounded exploration and evidence-state drift.Source: E Case Study · p. 19
  • Codex computes 4958.09 using 4100 for May 1942; replacing that operand yields 4962.46, so its answer fails exact match but passes the 5% tolerance criterion.The reported relative error is approximately 0.088%.Source: E Case Study · p. 19
  • LongHarness returns the reference answer 4962.46 in 10 policy actions after incorporating evidence and conclusion revisions.Further reading corroborates 4400 and corrects two other entries before computation.Source: E Case Study · p. 19

Related papers

Loading 2610.13075v1…