MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents

• Existing natural QA benchmarks usually contain simple single-passage questions, whereas complex expert-built datasets are costly and often small or limited to a handful of documents.Machine-generated complex benchmarks can instead produce contrived questions that do not reflect real-world users’ needs.Source: §1 · pp. 1–2
• MONACO contains 1,315 challenging multi-step questions requiring retrieval, filtering, and aggregation of dozens of intermediate facts from unstructured and structured sources, including paragraphs and tables.Each question has a manually annotated reasoning chain and requires 1 to 2,379 Wikipedia paragraphs and tables, averaging 43.3 unique pages per question.Source: §1 · p. 2
• MoNaCo elicits questions from target personas without fixed templates and uses a decomposed annotation pipeline to make large-scale answer collection feasible for non-expert workers.Question decomposition breaks difficult tasks into simpler intermediate annotations while preserving the broader multi-document objective.Source: §1 · p. 2
• MoNaCo supports evaluation of parametric knowledge, long-context reasoning, end-to-end retrieval-augmented QA, and multi-document retrieval.The reported experiments focus on the first three settings.Source: §1 · pp. 2–3
• 61.2% F1 was the top frontier-model score on MoNaCo, and only 38.7% of examples received a perfect score.Models suffered from low recall and hallucinated answers requiring aggregation across many facts.Source: §1 · p. 3

2.1 Eliciting Complex Natural Questions

• Persona-driven prompts and non-template writing encourage realistic, diverse, and time-consuming questions aligned with human information-seeking goals.Workers receive target personas and challenging reference questions to guide complexity without prescribing question forms.Source: §2.1 · pp. 3–4

2.2 Annotating Intermediate Answers

• QDMR decompositions turn each complex question into a sequence of intermediate steps that workers can answer separately.The design supports evidence distributed across documents, while each intermediate question is answered from a single page under the Wikipedia-focused annotation assumption.Source: §2.2 · p. 4

2.3 Deriving the Final Answers

• The execution engine derives follow-up questions by substituting answers from earlier decomposition steps into later steps.This lets later questions depend on multiple previously annotated answers.Source: §2.3 · p. 4
• 31 executable operators filter or aggregate intermediate answers to produce final answers from referenced prior steps.Operators are implemented as Python programs whose inputs are the answers to earlier steps.Source: §2.3 · p. 4

3.1 Question Writing and Decomposition

• Workers generated persona-focused questions and QDMR decompositions, enabling the execution engine to derive simpler intermediate questions automatically.Question writers produced challenging information-seeking questions, while expert workers decomposed them using QDMR syntax.Source: §3.1 · pp. 4–5, §3.2 · p. 5

3 Data Collection

• 1,315 fully annotated examples remained from 1,851 collected questions after decomposition, answer annotation, and validation.Source: §3.1 · p. 5

3.2 Intermediate Answers and Evidence

• Intermediate answers were collected with supporting Wikipedia pages, sections, sentences, or table columns, producing 90K attributed question-answer-evidence records.The pipeline used 10 crowd workers for intermediate answer annotation and required evidence at multiple granularities.Source: §3.2 · p. 5

3.3 Data Validation

• Validation combined worker qualification, author review, ongoing feedback, and task-specific checks for factoid, list, and unanswerable questions.Authors reviewed all questions and decompositions, manually validated final complex-question answers, and used worker-LLM agreement for factoid answers.Source: §3.3 · pp. 5–6
• List answers were accepted when overlap exceeded 0.77 and normalized length difference stayed below 0.25, with re-annotation otherwise.Source: §3.3 · p. 6
• After re-annotation, 4,083 intermediate questions were labeled unanswerable, representing 4.5% of the data.Source: §3.3 · p. 6

4.1 Data Statistics

• 1,315 questions contain 90K intermediate questions, including 8,549 list questions and 40,125 Boolean questions.The benchmark also draws evidence from 36,194 distinct Wikipedia pages, including sentences, tables, and lists.Source: §4.1 · pp. 6–7

4.2 How Complex are MONACO Questions?

• 43.3 evidence pages per question and 5.1 reasoning steps show that MONACO is substantially more complex than prior benchmarks.MONACO questions average 14.5 words, yet require more intermediate reasoning than benchmarks with longer questions.Source: §4.1 · p. 6, §4.2 · p. 7, Table 3 · p. 7
• Over 38% of MONACO questions require aggregation and 17% require arithmetic, while its reasoning-template diversity is significantly higher than other benchmarks.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 How Natural are MONACO Questions?

• MONACO was evaluated against four natural and four partially machine-generated benchmarks using writer identity and perceived prior knowledge of the answer.The user study sampled 900 questions and collected judgments from 18 graduate students who were unaware of each question’s benchmark.Source: §4.3 · pp. 7–8
• MONACO questions appeared closer to natural benchmarks than artificial ones, with 49% labeled expert-written and 20% labeled machine-written.The study’s plotted measures concerned likely writer identity and whether the writer already knew the answer.Source: §4.3 · p. 8, Figure 5 · p. 8, Figure 6 · p. 8

5.2 Language Models Performance

• 61.2% F1 was the best closed-book score, achieved by O3, while all models remained substantially below perfect performance.Reasoning models generally outperformed strong non-reasoning models.Source: §5.2 · p. 9, Table 4 · p. 9
• Few-shot prompts containing complete reasoning chains improved every evaluated non-reasoning LLM, unlike zero-shot or short-explanation prompts.The full chains explicitly prompted models to generate intermediate reasoning steps before answering.Source: §5.2 · p. 9

5.3 Performance Breakdown and Analysis

• Performance declined for all models as intermediate answers and evidence pages increased, although O3 and GPT-5 pulled ahead when questions required more than 20 pages.Their advantage was small on questions requiring few pages or intermediate answers but widened on broader questions.Source: §5.3 · p. 9, Figure 7 · p. 10
• GPT-4O maintained 75–80% precision on intermediate list questions, but its recall sharply decreased as the number of expected answers grew.List QA was evaluated across 8,549 intermediate list questions using string-based precision and recall.Source: §5.3 · p. 10, Figure 8 · p. 10

5.4 Retrieval Augmented LLMs

• Providing all gold evidence documents improved GPT-4O and LLAMA 3.1-405B by 10 points over closed-book evaluation, yet performance reached only 58.7%.In end-to-end RAG, BM25 retrieval caused a sharp performance drop relative to closed-book evaluation, indicating persistent retrieval robustness problems.Source: §5.4 · pp. 10–11, Figure 9 · p. 11

Related QA Benchmarks

• Most natural QA benchmarks answer questions from a single text span, while newer multi-hop and list benchmarks broaden reasoning but remain limited in scope.The cited benchmarks include work on multi-hop reasoning and answers distributed across documents.Source: §6 · p. 11

Natural Multi-Document QA

• Naturally elicited multi-document benchmarks typically require no more than a handful of webpages, whereas MoNaCo questions usually require dozens.This difference makes MoNaCo substantially more time-consuming than those benchmarks.Source: §6 · p. 11

BrowseComp

• BrowseComp emphasizes niche, trivia-like questions with easily verified evidence, while MoNaCo models real-world information-seeking goals and broad evidence synthesis.MoNaCo answers require combining evidence spread across dozens of Wikipedia pages.Source: §6 · p. 11

WideSearch

• WideSearch contains 100 English questions, compared with 1,350 questions in MoNaCo.Source: §6 · p. 11

Retrieval-Augmented QA

• Existing retrieval benchmarks rarely require more than five evidence documents, positioning MoNaCo as a broader and more rigorous retrieval-augmented generation benchmark.Source: §6 · p. 11

Conclusion

• 1,315 MoNaCo questions are manually annotated with gold reasoning chains containing 90K intermediate questions, answers, and supporting evidence documents.The benchmark targets tasks requiring extensive factual knowledge, information retrieval, and reasoning across many documents.Source: §7 · p. 11

• Temporal effects were not measured, although timestamps and answer-change labels are released to support time-specific evaluation.49.8% of questions are time-independent, 34.1% change every few years, and 16.1% yearly.Source: §8 · p. 12
• The experiments exclude LLM-powered Deep Research and iterative multi-step RAG systems, leaving MONACO available as a testbed for them.Source: §8 · p. 12
• GPT-4.1 was used as an answer-equivalence judge, but it may incorrectly accept wrong answers or reject correct ones.The authors identify stronger future judge models as a possible improvement.Source: §8 · p. 12

A.1 Benchmark Ceiling Performance

• MONACO avoids direct human-ceiling estimation because its questions require combining information from 43 distinct webpages on average.Source: §A.1 · p. 18
• Question decompositions and intermediate answers were manually or independently validated, while aggregation and numerical comparison were executed automatically.List-question validation required more than 77% answer overlap between two human annotators.Source: §A.1 · p. 18

A.2 List QA Performance

• Table 5 and Figure 8 report precision, recall, and F1 performance for prompted LLMs on MONACO’s intermediate list questions.Source: Table 5 · p. 18, §A.2 · p. 18

A.3 Retrieval Performance

• Table 6 and Figure 9 provide LLM results when gold evidence documents are supplied as input.Source: Table 6 · p. 18, §A.3 · p. 18

B.1 Eliciting Complex Natural Questions: Additional Details

• Workers received personas, keywords, and unrelated time-consuming reference questions across 28 domains, without fixed templates or answer evidence.The approach produced questions judged realistic and more time-consuming than those in existing benchmarks.Source: §B.1 · p. 19

B.2 Decomposition Execution

• The decomposition executor extends the original QDMR representation to support 31 query operators.Source: §B.2 · p. 19

B.3 Natural Questions User Study

• The naturalness-study materials compare MONACO questions with questions from other information-seeking QA benchmarks.Source: §B.3 · p. 19, Figure 10 · p. 19

B.4 Prompts

• The prompt appendix includes system, retrieval-augmented, list-QA, factoid-QA, answer-agreement, and few-shot chain-of-thought prompts.Source: §B.4 · p. 19, Figure 11 · p. 20, Figure 13 · p. 21
• Figures 17 and 18 present excerpts of few-shot chain-of-thought prompts, respectively without and with intermediate answers; Table 7 lists all query operators supported by the decomposition executor.Source: Table 7 · p. 20, Figure 17 · p. 23, Figure 18 · p. 23