Source-linked AI summary
Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agents
Corby Rosset, Ho-Lam Chung, Guanghui Qin, Ethan C. Chau, Zhuo Feng, Ahmed Awadallah, Jennifer Neville, Nikhil Rao
TL;DR
Traditional QA benchmarks largely ask factoid questions with identifiable missing information, leaving a gap in evaluating complex questions with unclear information needs. Researchy Questions is a dataset of filtered search queries for studying these questions, and experiments report benefits from decomposition techniques on some tasks. The study also identifies limits in evaluating the quality of generated sub-questions.
Problem
Traditional QA benchmarks largely use factoid questions whose missing information and sub-questions are clear, leaving complex questions with unclear information needs insufficiently represented.
Source: §1 · pp. 1–3Method
The authors curate Researchy Questions from search queries, filtering for non-factoid questions and deduplicating queries to produce a dataset for evaluating QA systems and LLM agents.
Source: §1 · p. 3, §2.2 · p. 5, §2.4 · p. 6Results
Decompositional techniques improve accuracy on short-form multi-hop datasets over direct answering, while factored decomposition leads to the highest scores on long-form questions, particularly Wikihow and Researchy Questions.
Source: §5 · p. 7Takeaways & Limitations
The authors position Researchy Questions as a resource for studying how LLM agents search for and respond to unclear information needs.
Source: §3 · p. 7The study does not evaluate whether GPT-4’s released question decompositions lead a real retrieval system to the same gold documents as users’ clicks.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Existing question answering (QA) datasets are no longer challenging to most powerful Large Language Models (LLMs). Traditional QA benchmarks like TriviaQA, NaturalQuestions, ELI5 and HotpotQA mainly study ``known unknowns'' with clear indications of both what information is missing, and how to find it to answer the question. Hence, good performance on these benchmarks provides a false sense of security. A yet unmet need of the NLP community is a bank of non-factoid, multi-perspective questions involving a great deal of unclear information needs, i.e. ``unknown uknowns''. We claim we can find such questions in search engine logs, which is surprising because most question-intent queries are indeed factoid. We present Researchy Questions, a dataset of search engine queries tediously filtered to be non-factoid, ``decompositional'' and multi-perspective. We show that users spend a lot of ``effort'' on these questions in terms of signals like clicks and session length, and that they are also challenging for GPT-4. We also show that ``slow thinking'' answering techniques, like decomposition into sub-questions shows benefit over answering directly. We release $\sim$ 100k Researchy Questions, along with the Clueweb22 URLs that were clicked.
1 Introduction
Existing QA benchmarks largely test factoid or clearly decomposable questions, while complex open-domain questions can leave information needs unclear. Researchy Questions is introduced as a dataset of real search queries that are non-factoid, long-form, research-intensive, and multi-perspective.
1 Introduction
- Traditional factoid benchmarks can be memorized or solved with simple pattern matching, while multi-hop datasets still have factoid answers and clear sub-questions.The cited multi-hop datasets also have a distribution mismatch with human questions because some are synthetically generated from Wikipedia link paths.Source: §1 · pp. 1–2
- Existing decomposition studies use traditional QA benchmarks such as HotpotQA or simple games, leaving no suitable open-domain web benchmark for these techniques.Decomposition can reframe unclear problems into sub-questions with clearer information needs, but the passage says the appropriate benchmark does not yet exist.Source: §1 · p. 3
- Researchy Questions are real users’ non-factoid queries expecting long-form answers that require research, synthesis across evidence, and consideration of multiple perspectives.They are framed as complex search tasks with unclear information needs and no single correct answer.Source: §1 · p. 3
- The dataset is intended to study how LLM agents handle unclear information needs in complex questions and to support evaluation of question-answering systems or agents.It also functions as a search/retrieval dataset because finding and incorporating appropriate evidence matters to trustworthiness and groundedness.Source: §1 · pp. 3–4
- The authors release about 96K real search-engine queries with two-level question decompositions, user-aggregated clicks on ClueWeb22 URLs, and ordered sub-queries.Clicked URLs are offered as a signal of information users found useful, in part to help assess whether sub-questions lead to relevant information.Source: §1 · p. 4
2 Researchy Questions Construction
Researchy Questions are real search-log queries filtered to identify non-factoid questions requiring substantial research and decomposition. The construction pipeline filters and deduplicates queries, then applies GPT-4 quality checks before release.
2.1 Stage 1: Mining Search Logs
- Queries came from a commercial search engine’s July 2021–August 2022 logs, with at least one click; frequency and intent rules further filtered them.Queries had to occur at least 50 times, and the rules included English, question intent, multiple clicked URLs, and exclusions for navigational, local, retail, coding, and medical intent.Source: §2.1 · pp. 4–5
- Researchy Questions show more clicks, longer-dwell sat clicks, and unique documents, while harder questions appear in longer sessions.Table 4’s caption describes the left panel as query subsets and the right panel as session statistics.Source: Table 4 · p. 5
2.2 Stage 2: Factoid Classifier
- A BERT-large classifier trained on GPT-3-labeled examples retained 1.0M queries scoring at least 0.75 for non-factoid status.The training sample contained 200k questions from the 15.7M QnA Queries; the threshold was chosen by manual inspection.Source: §2.2 · p. 5
2.3 Stage 3: Decompositional Classifier
- A second BERT-large classifier, trained on ChatGPT labels for about 40k queries, identified 146k candidates above a manually selected decompositional threshold of 0.6.The classifier scored whether questions required asking sub-questions; the authors note the procedure was imperfect.Source: §2.3 · p. 5
2.4 Stage 4: Deduplication
- Semantic-intent clustering grouped queries whose pairwise distances were within ε, retaining the most frequent query as each group’s representative.About 63% of queries were singletons, average group size was 3.8, and deduplication yielded 102k questions; some topic clusters remained.Source: §2.4 · pp. 5–6
2.5 Stage 5: Final GPT-4 Filtering
- GPT-4 quality checks removed questions flagged as ambiguous or incomplete, assumptive, or unsafe, leaving 96k queries for release.About 3% were removed as ambiguous or incomplete, another 2% as assumptive, and another 2% for safety reasons.Source: §2.5 · p. 6
3 Characterizing Researchy Questions
Researchy Questions combine substantial research needs with multiple perspectives, and their click traces provide a large set of documents for studying how agents handle unclear information needs. GPT-4 judged them to require more sub-questions, knowledge, and reasoning than comparison datasets.
Dataset scale and click coverage
- 96k questions have 350k unique clicked documents; each question averages 4.9 +/- 3.5 clicked documents, while each document is associated with 1.4 +/- 2.3 questions.48% of the unique documents are in the English subset of Clueweb22 Set B; the rest are in Sets A or L.Source: §3 · p. 6
Question complexity
- GPT-4 judged Researchy Questions to require the most sub-questions, knowledge, and reasoning among the compared longer-form QA datasets.The knowledge, reasoning, and multi-facetedness comparison is with Natural Questions.Source: §3 · p. 6
Question characteristics
- The distribution of first words is reported for Researchy Questions and the intermediate datasets used in the filtering funnel.MS Marco queries provide a comparison: 1.64% start with “why.”Source: §3 · p. 6
Pivotal Facts
- Clicked URLs contain surprising information that can substantially change an answer, but the authors lack a way to quantify how prevalent these Pivotal Facts are.The authors propose the dataset for studying how LLM agents search for and respond to such unknown unknowns.Source: §3 · pp. 6–7
4 Agreement with User Search Behavior
Search behavior signals indicate that users spend more effort on non-factoid questions. Researchy Questions are associated with deeper click consumption and longer sessions than factoid-oriented queries.
Click behavior
- Researchy Questions lead to more clicks, sat-clicks, and unique URLs than other query subsets, indicating more in-depth consumption of more diverse information.The authors report this pattern for queries that are both non-factoid and decompositional.Source: §4 · p. 7
Session behavior
- Users were twice as engaged answering non-factoid questions as in factoid-y sessions, and sessions were six times longer than the overall average.The comparison uses session-level behavioral signals rather than individual clicks.Source: §4 · p. 7
5 Evaluating Answer Techniques to Researchy Questions
The paper compares direct answering with two decomposition techniques for GPT-4. Factored decomposition leads to the highest scores on long-form questions, particularly Researchy Questions, while chain-of-thought decomposition may suit logic-intensive questions better.
Evaluation setup
- The paper proposes side-by-side evaluation because Researchy Questions do not have one correct answer, using a closed-book answer as a possible reference.Source: §5 · p. 7
- The evaluation compares direct answering, chain-of-thought decomposition, and factored decomposition, which answers sub-questions independently before a final recomposition call.GPT-4 supplies the answers and also serves as the judge; long-form datasets are assessed by overall Score, while short-form datasets emphasize Accuracy.Source: §5 · p. 7
Results
- Factored decomposition leads to the highest scores on long-form questions, particularly Wikihow and Researchy Questions, while chain-of-thought may suit logic-intensive questions better.The authors expect Researchy Questions scores with factored decomposition to be much higher if retrieved information is incorporated.Source: §5 · pp. 7–8
6 Related Work
Related work connects Researchy Questions to studies of search behavior, retrieval-augmented generation, and agentic question answering. These lines of work examine user search signals, iterative retrieval and generation, and agents that interact with tools or the web.
6.1 Human Behavior in Search Sessions
- Search-session studies use clicks and session signals to distinguish exploring from struggling; the cited work associates exploring sessions with more clicks and the paper reports more clicks for non-factoid questions.Complex search tasks have information-seeking behaviors described as nearly congruent with Researchy Questions.Source: §6.1 · p. 8
6.2 Iterative Retrieval-Augmented Generation
- Iterative retrieval-augmented generation methods combine LLMs with retrieval, varying how retrieval and generation are coordinated across steps.Examples include Self-Ask, IRCoT, Iter-RetGen, and Beam Retrieval.Source: §6.2 · p. 8
6.3 Agentic Question Answering
- Agentic QA frameworks and web agents support dynamic interaction between LLMs and tools, including retrieval systems and web pages.Examples include Demonstrate-Search-Predict, AutoGen, WebGPT, and WebAgent; consumer-facing systems are also described as decomposing queries and retrieving pages.Source: §6.3 · p. 8
7 Conclusion
The conclusion presents Researchy Questions as a large dataset for complex multi-document, multi-perspective QA and reports preliminary evidence favoring decompositional answering over direct answering. It notes that the lack of gold answers makes model headroom difficult to quantify and leaves evaluation questions open.
7 Conclusion
- The paper presents Researchy Questions as a large dataset for complex question answering over multiple documents and perspectives, targeting LLM-assisted web search agents.The authors report that the queries are mined from search logs and involve significantly more effort than other types of search queries.Source: §7 · p. 8
- The authors report preliminary evidence that decompositional answering techniques yield stronger results than answering directly.They identify measuring sub-question quality and studying how LLM agents find and interact with Pivotal Facts as remaining work.Source: §7 · p. 8
- The questions have no gold answers, making it difficult to quantify existing models’ “headroom.”The authors say there nevertheless seems to be substantial room for improvement, while noting that the evidence is qualitative.Source: §7 · p. 8
Limitations
The limitations include a lack of a strong measure for candidate sub-question quality and no real-retrieval evaluation of GPT-4’s released decomposition. The study also leaves backward decomposition and quantifying Pivotal Facts unaddressed, and the dataset is not multilingual.
Limitations
- The study does not propose a good measure of candidate sub-question quality, using clicked-document information only as a loose proxy.The authors explicitly acknowledge that good sub-questions leading to clicked-document information and measuring sub-question quality are not the same thing.Source: Limitations · p. 9
- GPT-4’s released closed-book decomposition was not evaluated for whether it retrieves the same gold documents in a real system built from ClueWeb22.The authors describe clicked URLs as a powerful signal, not a direct evaluation of retrieval performance.Source: Limitations · p. 9
- The study does not explore producing decompositions backward from clicked documents, though the authors say this approach could potentially be better.They contrast this possibility with the forward direction that would need to be implemented in practice.Source: Limitations · p. 9
- Pivotal Facts remain observational and unquantified, and the dataset is not multilingual because curation involved uncertainty, trial and error, and frequent manual inspection.The authors suggest the same framework could be used to construct a multilingual version.Source: Limitations · p. 9
Ethics Statement
The authors obtained approval to release the dataset after IRB proceedings addressing privacy, safety, and legal guidelines. They describe safety review as assessing risk of harm rather than judging users’ motives, with GPT-4 serving as one tool in a broader process.
Ethics Statement
- Dataset release approval followed thorough IRB proceedings intended to ensure compliance with privacy, safety, and legal guidelines.Source: Ethics Statement · p. 9
- The authors say their role is to assess whether answering a query could pose a reasonable risk of harm, not to speculate about users’ motives.They state that query motives are beyond the scope of their work.Source: Ethics Statement · p. 9
- GPT-4 is described as one of several tools in the safety process, not as a moral authority.Source: Ethics Statement · p. 9
A GPT-4-based Filtering Details
The filtering process used GPT-4 to label question attributes and focused on excluding queries too incomplete or underspecified to answer meaningfully, as well as presumptuous queries.
A GPT-4-based Filtering Details
- GPT-4 labeled attributes of each Researchy Question using a prompt shown in Figure 7.Source: A GPT-4-based Filtering Details · p. 14
- Filtering focused on removing queries too incomplete or underspecified to answer meaningfully, including queries that refer to earlier queries or subjects in a session.The passage also identifies overly presumptuous questions as a removal target.Source: A GPT-4-based Filtering Details · p. 14
A.1 Additional Safety Filtering
The authors added GPT-4 harmfulness scoring to safety filtering and removed roughly 2% of queries judged unsatisfactory, citing the risk that answering them could cause harm.
A.1 Additional Safety Filtering
- Roughly 2% of queries were removed from the released dataset for unsatisfactory harmfulness scores.GPT-4 rated whether queries could be harmful, offensive, biased, sexually explicit, or otherwise inappropriate for a 12-year-old.Source: §A.1 · p. 14
- The authors judged that answering some removed queries posed too high a risk, even when they might reflect genuine information-seeking.The authors note that some removed queries may have been motivated by genuine information-seeking intent.Source: §A.1 · p. 14