PaperlayerView exact revision on arXiv

Source-linked AI summary

AGORA: An Archive-Grounded Benchmark for Agentic Workplace Document Reasoning

Honglin Guo, Qi Zhang, Yu Zhang, Weijie Li, Rui Zheng, Zhikai Lei, Qiyuan Peng, Zhiheng Xi, Tao Gui, Qi Zhang

arXiv:2606.24526v1cs.CL

TL;DR

Workplace document reasoning requires agents to find sparse, inconsistent evidence across large archives, a gap existing benchmarks do not jointly test. AGORA addresses this with a cross-domain benchmark and agentic construction pipeline, finding that even the strongest of eight evaluated models achieves only 59.4% accuracy with substantial domain variation.

  • Problem

    Existing benchmarks do not jointly test archive-groundedness, agentic exploration, and cross-domain reasoning over messy workplace archives.

  • Method

    AGORA pairs 362 queries with eight workplace document collections and uses an agentic pipeline for task synthesis, leakage prevention, difficulty filtering, and human verification.

  • Results

    59.4% accuracy is the strongest model’s result, with per-domain analysis revealing systematic blind spots and rank inversions.

  • Takeaways & Limitations

    AGORA offers a rigorous, reproducible testbed for evaluating document-reasoning agents across workplace domains.

  • Takeaways & Limitations

    Absolute accuracies may shift under heavier agent frameworks because all models were evaluated in a single minimal bash-tool harness.

Abstract

from arXiv · show

Large language models are increasingly deployed as agents that reason over documents rather than answer from parametric knowledge. We study archive-grounded reasoning: locating sparse evidence across a large, messy collection of workplace files, reconciling inconsistent terminology, units, and time conventions, and computing an answer. Existing benchmarks address only parts of this setting and none jointly stresses archive-groundedness, agentic exploration, and cross-domain coverage. We introduce Agora, a benchmark pairing 362 questions with eight domain collections of 9,664 authentic documents and 372M tokens, far exceeding any model's context window, so agents must explore deliberately rather than scan exhaustively. Agora is built by an agentic pipeline combining cross-document task synthesis, leakage-preventing obfuscation, and difficulty filtering. Evaluating eight models, we find the task far from solved: even the strongest reaches only 59.4% accuracy, with notable variation across domains.

1 Introduction

AGORA benchmarks archive-grounded agentic document reasoning over large, heterogeneous workplace archives, requiring agents to locate sparse evidence, reconcile inconsistencies, compute answers, and verify results. Its 362-question evaluation spans eight domains and 9,664 documents, and the strongest of eight evaluated models achieves only 59.4% accuracy.

  • Motivation: Workplace agents must locate sparse evidence, reconcile inconsistent figures, dates, and assumptions, perform calculations, and produce accurate, verifiable answers.This motivates evaluating archive-grounded document reasoning rather than chat fluency alone.
  • Benchmark gap: Existing benchmarks miss key workplace-archive demands because they typically use homogeneous corpora, reading-comprehension formats, or open-web and simulated environments.These settings generally do not require heterogeneous file navigation and multi-step computation within a bounded internal archive.
  • Evaluation: 59.4% accuracy is achieved by the strongest of eight proprietary and open-weight models, leaving archive-grounded reasoning below the 60% threshold.All models are evaluated through mini-swe-agent with only a bash tool, keeping the agent interface fixed and simple.
  • AGORA: AGORA contains 362 questions over eight workplace collections totaling 9,664 real-world documents and 372M tokens, with unique numeric answers for deterministic verification.Collections exceed current model context windows, preventing exhaustive scanning and requiring planned exploration and evidence selection.

2 Related Work

Prior work has advanced tool-using agents for productivity tasks and multi-document question answering, but typically treats documents as manipulation targets or passages rather than as workplace archives serving as evidence.

  • Agentic productivity systems: Methodological advances established autonomous agents that interleave reasoning with tool invocation and enabled deployment across code repair and end-to-end office workflows.The cited foundations include Yao et al. (2023), Schick et al. (2023), Yang et al. (2024), and Wang et al. (2025).
  • Agentic productivity systems: SpreadsheetBench, MEBench, and OfficeBench evaluate cell-level reasoning, multi-step tool use, and task completion across diverse office software.These benchmarks ground evaluation in interactive environments or concrete office artifacts, with documents typically serving as manipulation targets.
  • Multi-document question answering: HotpotQA, 2WikiMultihopQA, MuSiQue, FRAMES, MultiHop-RAG, and M3SCIQA advance multi-document question answering across Wikipedia passages, news corpora, and workspace document archives.The listed benchmarks require composing facts across multiple passages, often with explicit supporting-fact supervision, and extend multi-hop reasoning beyond Wikipedia.

3 The AGORA Benchmark

AGORA is designed to measure archive-grounded, agentic document reasoning across diverse workplace collections through cross-document, multi-hop tasks with deterministic numeric evaluation. Its construction combines multi-domain document processing, evidence-grounded task synthesis, leakage suppression, and difficulty and validity filtering.

  • Design desiderata: AGORA requires answers to rely exclusively on a frozen source collection, making evidence collection-dependent and reproducible over time.The benchmark excludes open-web access and prevents the available evidence from drifting.
  • Verifiable evaluation: AGORA uses single numeric answers and specified output formats so responses can be checked automatically and deterministically without human or model-based judging.Verification normalizes superficial formatting differences against ground truth.
  • Task formulation: Tasks require deliberate exploration of sparsely distributed evidence across files, reconciliation of inconsistent terminology, units, and time conventions, and answer computation.Each query is cross-document and multi-hop, requiring the agent to locate bridging facts amid unrelated material and make multiple tool calls.
  • Construction pipeline: The construction pipeline collects and preprocesses documents, synthesizes tasks from cross-document evidence with reference reasoning paths and verification code, then applies quality control.Quality control discards parametric-knowledge-solvable tasks, removes tasks solved correctly by all three panel models, and retains only tasks whose reasoning chains are verified as valid, solvable, and unambiguous.
  • Benchmark composition: AGORA spans eight source collections, each representing one professional domain and containing authentic workplace documents converted into plain-text Markdown files.Each collection aggregates documents from several distinct data sources, including official reports, statistical yearbooks, and tabular records.

4 Experiments

Experiments evaluate eight models as archive-grounded agents in a constrained document environment. AGORA remains far from solved: the strongest model reaches only 59.39% overall accuracy, with substantial variation across domains that can reorder the leaderboard.

  • Models: Eight models are evaluated at temperature 1.0 with maximum supported reasoning effort, using official APIs except for locally served Qwen models.The evaluated systems are GPT-5.5, Gemini-3.1-Pro, Gemini-3.1-Flash-Lite, DeepSeek-V4-Flash, DeepSeek-V4-Pro, GLM-5.1, Qwen3.5-35B-A3B, and Qwen3.5-9B.
  • Evaluation setup: Agents use mini-swe-agent to explore mounted documents, run bash computations, and submit answers through an <answer> ... </answer> tag.Tasks run in isolated E2B sandboxes without internet access; episodes are capped at 200 turns and 3,600 seconds, and invalid or missing answer tags score incorrect.
  • Overall results: 59.39% is the strongest overall accuracy, achieved by Gemini-3.1-Pro; no model exceeds 60%.Because every task has a single verifiable numeric answer solvable from the mounted collection, the performance gap reflects a capability deficit rather than formatting artifacts.
  • Per-domain results: 41.03% is Gemini-3.1-Pro’s Finance accuracy, while GPT-5.5 reaches 38.00% on Business, showing that aggregate performance masks weak domains.Gemini-3.1-Pro tops five of eight domains but ranks fourth on Finance, so per-domain rankings diverge from the aggregate leaderboard.

5 Analysis

AGORA’s analysis finds that cross-domain performance does not transfer cleanly and that aggregate accuracy conceals domain imbalance. It also identifies five recurring failure modes and shows that long interaction trajectories are predominantly incorrect.

  • Cross-domain analysis: Cross-domain reasoning does not transfer reliably, making multi-domain coverage necessary to expose domain-specific blind spots and rank inversions.Models are centered on overall accuracy to isolate domain-specific residuals.
  • Cross-domain analysis: 30.97 points is the reported worst-domain spread, showing that aggregate strength and domain balance are distinct axes.A small spread among smaller models can instead reflect near-floor accuracy across all domains.
  • Failure modes: Wrong traces are categorized as Incomplete Inspection, Evidence Misidentification, Resource Exhaustion, Instruction Non-Following, or Hallucination.These categories cover skipped documents, wrong extracted values, exhausted budgets, ignored requirements, and fabricated or forgotten findings.
  • Exploration budget: Correct outcomes concentrate at low-to-moderate turn counts, whereas episodes extending deep into the interaction budget are predominantly incorrect.The distribution is stratified by final-answer turn and partitioned by outcome.

6 Conclusion

AGORA is an archive-grounded, cross-domain benchmark for agentic workplace document reasoning, pairing verifiable numeric queries with large workplace-document collections. Its construction jointly tests archive-groundedness, agentic exploration, and cross-domain coverage, and model evaluation shows the task remains far from solved.

  • AGORA pairs 362 verifiable numeric queries with eight domain collections containing 9,664 documents and 372M tokens.
  • An agentic pipeline combines cross-document task synthesis, leakage-preventing obfuscation, difficulty filtering, and human verification to build AGORA.
  • AGORA jointly stresses archive-groundedness, agentic exploration, and cross-domain coverage, while evaluation across eight models finds the task far from solved.

Limitations

AGORA prioritizes broad, carefully filtered workplace coverage over exhaustive domains and query volume, while acknowledging that benchmark difficulty and reported accuracies may change as pretraining and harness designs evolve.

  • 362 queries cover eight professional domains, but the collection is not exhaustive of all workplace settings.Queries pass multi-stage difficulty filtering, automated verification, and human annotation.
  • Closed-book filtering removes tasks solvable from parametric knowledge alone, but future pretraining may absorb similar documents and erode this guarantee.The authors propose refreshing the benchmark with new collections as model capabilities advance.
  • A single minimal bash-only harness isolates model capability from scaffolding engineering, so absolute accuracies may shift under heavier frameworks.The effect of harness design on real-world agent deployment remains for future study.

A Sandbox Environment

AGORA runs each task in a fixed, isolated sandbox with constrained compute, no network access, and a 3,600-second timeout. Its pinned environment provides document-parsing and scientific-computing tools, with standardized workspace paths and reproducible package versions.

  • Sandbox configuration: Each task runs in an isolated E2B2 sandbox with 2 vCPUs, 4 GB memory, no network access, and a 3,600-second timeout.The sandbox uses a fixed Docker template extending the official e2bdev/base image.
  • Sandbox configuration: The environment includes Python 3, command-line utilities, and pinned pandas, numpy, scipy, openpyxl, lxml, beautifulsoup4, pyyaml, pymupdf, pdfplumber, and tabulate libraries.These tools support the tabular computation and file parsing required by AGORA without allowing additional dependency installation.
  • Sandbox configuration: Documents reside at /workspace/documents, scratch files at /workspace/run, and pinned package versions keep evaluation runs stable.The standardized paths separate the document collection from working space.

B Heuristic for Information-Density Scoring

Agora uses a lightweight additive heuristic to prioritize chunks likely to support multi-hop computation, reducing the cost of agentic task synthesis. The score combines content, structure, and spreadsheet signals, applies hard exclusions, then retains up to 100 qualifying seed chunks per domain.

  • Scoring rationale: The preprocessing heuristic cheaply ranks chunks for downstream synthesis because each seed invokes an expensive full agent trajectory.The rule-based prefilter is preferred over LLM judgment to reduce wasted synthesis budget.
  • Scoring formula: Numeric density contributes 5·1000·|N(c)|/|c|, capped at 30, rewarding chunks rich in financial values, counts, or measurements.N(c) denotes the set of numeric tokens in chunk c.
  • Format-specific signals: Excel and CSV chunks receive additional rewards for rows, columns, numeric columns, time columns, and year span, favoring wide, typed, time-spanning sheets.The added terms are rows/100 capped at 15, cols capped at 10, 2·|numeric columns|, 8 for any time column, and 2·(yearmax − yearmin) capped at 10.
  • Format-specific signals: Markdown and PDF chunks gain points for detected tables, while Markdown lists also receive a smaller bonus because tabular structure supports cell-level lookup.Markdown adds 10 for a table and 3 for a list; PDF adds 12 when OCR output contains a table.
  • Selection: After hard exclusions and a minimum score floor of 10, each domain contributes the top 100 chunks as synthesis seeds.Chunks under 20 tokens and matching directory-listing patterns are excluded; the per-domain cap bounds synthesis and prevents richer corpora from monopolizing the pool.

C Prompts

The prompts define a sandboxed document-reasoning workflow, chunk-summary instructions, and a two-subagent quality-review protocol that returns structured JSON judgments. The reviewer compares blind and reasoning-guided solves to assess answerability, correctness, evidence, files, and deliverability.

  • System Prompt for Evaluation: The evaluation prompt restricts agents to a Linux sandbox, references domain documents under /workspace/documents/{{ domain}}/, and requires every reply to include a bash tool call.Final submission must use one bash command whose stdout contains exactly two lines, including COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT and a tagged answer.
  • Prompt for Chunk-Level Summary: The chunk-summary prompt requests the main topic, entities, time range, data type, and notable features in the input language, followed by 3–8 XML-formatted keyword tags.The required tags line uses the format <tags>tag1, tag2, tag3</tags>.
  • Solving Workflow: Launch Two Subagents: The quality reviewer launches two subagents: one solves blindly from the question, while the other receives the reference reasoning chain but not the reference answer.Both should assess question reasonableness, track accessed files, avoid reading all data, and complete before the reviewer compares their trajectories.
  • Quality Review: The reviewer evaluates clarity, ambiguity, answer and reasoning correctness, calculations, citations, evidence, file redundancy, missing information, external reliance, and delivery suitability.The output must be valid JSON containing is_usable, quality_score, ambiguity, answer_status, reasoning_status, evidence_status, file lists, issues, and review_summary.

D Responsible NLP Statements

AGORA’s data and model use follow original licenses and access conditions, with the dataset restricted to academic research. The paper also discusses risks from advancing autonomous document-archive reasoning and discloses limited use of language models as writing aids.

  • Data and model use: The evaluated models were used under their original licenses and intended-use terms, while AGORA’s derivative dataset is restricted to academic research and non-commercial use.The release follows the original research-access conditions.
  • Risks and responsible use: AGORA’s publicly available workplace-file collections are released strictly for research, and users should respect the original sources’ terms.The paper cautions against uses beyond academic evaluation.
  • Language-model disclosure: The authors used Claude Opus 4.7 and GPT-5.5 as writing aids for polishing, grammar, literature search, code completion, and presentation clarity.The stated use was limited to manuscript preparation assistance.

E Task Examples

The examples require agents to locate and reconcile evidence across multiple workplace documents, then compute precisely defined rates, comparisons, and normalized scores. Tasks span energy, housing, business, education, public health, and law, often imposing strict source-selection and output-format constraints.

  • Agriculture, Resources & Energy: Energy tasks combine regulator tables with three ministry documents, normalize physical and currency units, and report the regulator-minus-ministry mean growth gap in signed percentage points.Rates must use table-cell values and the result must be rounded to two decimals in `+x.xx pp` or `-x.xx pp` format.
  • Architecture, Construction, Real Estate & Facilities: Housing tasks identify Japan’s dwelling-related web-route series across categories and compare its largest compound annual change with an England housing-value change annualized by exact elapsed days.The quotient is rounded to three decimal places and returned with an `x` suffix.
  • Business, Management, Marketing & Sales: The connectivity-retailer task combines historical and present-day access, reach, usage, and demand-growth signals, yielding a premium of 2.463 pp/requirement.The result is the present-day-minus-historical premium per top-level security requirement entry, rounded to three decimals.
  • Education, Science & Academia: Education examples select a release span and qualifying Chinese statistical books, then estimate growth from the OLS slope of log cash received per university result-transfer deal.The requested output is exp(beta)-1, reported as one signed percentage rounded to three decimals.
  • Education, Science & Academia: Other examples compute a median forecast-profit CAGR for a role-defined stock group and a signed admissions count growth rate from carefully selected longitudinal observations.The admissions task uses the largest count as the base and the chronologically latest count as the endpoint.
Loading 2606.24526v1…