Source-linked AI summary
SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
Thinh Pham, Nguyen Nguyen, Pratibha Zunjare, Weiyuan Chen, Yu-Min Tseng, Tu Vu
TL;DR
SEALQA addresses the limited evaluation of search-augmented models on fact-seeking questions with conflicting, noisy, or irrelevant results. It introduces three curated benchmark flavors and evaluates current models, finding persistent weakness across noisy search and long-context settings. The benchmark is released as a resource for future research.
Problem
Existing evaluations often use short factual queries answered directly by top-ranked results, inadequately testing reasoning under noisy, conflicting, or ambiguous search evidence.
Method
The paper constructs SEALQA with SEAL-0, SEAL-HARD, and LONGSEAL, using expert-vetted questions and noisy multi-document retrieval settings.
Results
Frontier models underperform across SEALQA flavors, remain vulnerable to noisy search results, and often gain unreliably from increased test-time compute.
Takeaways & Limitations
SEALQA exposes unresolved weaknesses in search-augmented reasoning, especially filtering misleading evidence and identifying relevant documents among distractors.
Takeaways & Limitations
SEALQA is intentionally small because developing its questions is costly and complex.
Abstract
from arXiv · showhide
We introduce SealQA, a new challenge benchmark for evaluating SEarch-Augmented Language models on fact-seeking questions where web search yields conflicting, noisy, or unhelpful results. SealQA comes in three flavors: (1) Seal-0 (main) and (2) Seal-Hard, which assess factual accuracy and reasoning capabilities, with Seal-0 focusing on the most challenging questions where chat models (e.g., GPT-4.1) typically achieve near-zero accuracy; and (3) LongSeal, which extends SealQA to test long-context, multi-document reasoning in "needle-in-a-haystack" settings. Our evaluation reveals critical limitations in current models: Even frontier LLMs perform poorly across all SealQA flavors. On Seal-0, frontier agentic models equipped with tools like o3 and o4-mini achieve only 17.1% and 6.3% accuracy, respectively, at their best reasoning efforts. We find that advanced reasoning models such as DeepSeek-R1-671B and o3-mini are highly vulnerable to noisy search results. Notably, increasing test-time compute does not yield reliable gains across o3-mini, o4-mini, and o3, with performance often plateauing or even declining early. Additionally, while recent models are less affected by the "lost-in-the-middle" issue, they still fail to reliably identify relevant documents in LongSeal when faced with numerous distractors. To facilitate future work, we release SealQA at huggingface.co/datasets/vtllms/sealqa.
1 INTRODUCTION
SEALQA is introduced as a rigorously curated benchmark for reasoning over noisy, conflicting, and ambiguous search results. Its three flavors target frontier-model failure, broader difficult queries, and long-context evidence selection.
- 1 INTRODUCTION: SEALQA targets complex factual reasoning that existing benchmarks inadequately capture.Questions require entity and event disambiguation, temporal tracking, chart or table interpretation, counting, cross-lingual reasoning, and false-premise detection.
- 1 INTRODUCTION: SEAL-0 contains 111 questions iteratively refined until multiple models failed across several attempts.The core set targets frontier non-reasoning models such as GPT-4.1 with browsing and uses the resulting 0% accuracy threshold in its name.
- 1 INTRODUCTION: SEAL-HARD contains 254 broader difficult questions that include SEAL-0 and additional items missing its strict failure threshold.
- 1 INTRODUCTION: LONGSEAL contains 254 needle-in-a-haystack questions requiring long-context, multi-document reasoning amid irrelevant, noisy, or misleading content.Each question has one document containing or implying the answer, buried among distractors.
- 1 INTRODUCTION: The benchmark is intentionally small because question development is costly and complex, while smaller scale supports lower evaluation costs and more frequent updates.A team of six NLP researchers worked for eight months through multiple development cycles.
- 1 INTRODUCTION: Evaluations reveal significant limitations across SEALQA flavors, including vulnerability to noisy results, unreliable test-time scaling, and difficulty prioritizing relevant evidence.The benchmark is publicly released as a dynamic, versioned resource for future work.
2 DATA COLLECTION
SEALQA was constructed through expert-guided question development, adversarial difficulty filtering, and noisy-retrieval document pairing. Its protocol supports verifiable, freshness-aware questions and evaluates factual consistency with a reliable automated rater.
- 2 DATA COLLECTION: SEALQA questions are designed with single, unambiguous answers, webpage support, freshness labels, and realistic ambiguous or misleading search behavior.Questions are categorized as never-, slow-, or fast-changing according to answer stability.
- 2 DATA COLLECTION: Each question undergoes multi-round review by at least two graduate-level annotators, expert approval, URL and answer verification, and clarity cleaning.Questions changing too frequently are excluded, with effective and expected review dates recorded.
- 2 DATA COLLECTION: The benchmark spans diverse domains and reasoning categories, with advanced reasoning appearing in 72.4% of questions.Questions average 31 tokens and reach a maximum of 69 tokens.
- 2 DATA COLLECTION: SEAL-0 retains only questions that GPT-4-class and Llama-4 models failed across 10–15 attempts, then combines rejected difficult questions into SEAL-HARD.
- 2 DATA COLLECTION: LONGSEAL pairs each SEAL-HARD question with one helpful document and up to 50 relevant-looking hard negatives, randomly inserting the gold document.The resulting collection contains over 7.6K documents for noisy long-context reasoning.
- 2 DATA COLLECTION: Evaluation uses a GPT-4O MINI auto-rater that labels answers correct, incorrect, or not attempted, with 98% agreement against unified human ratings.
3 EXPERIMENTS
Experiments show that current language models struggle with noisy, conflicting, and distracting search contexts. Search can help some models, but test-time scaling and retrieval often produce unreliable or harmful results.
- SEAL-0 and SEAL-HARD: Models perform poorly on SEAL-0 and SEAL-HARD without search, with accuracies ranging from 0.0% to 5.4% and 0.0% to 22.4%, respectively.Model size does not consistently predict performance, and reasoning-focused models do not consistently outperform general-purpose chat models.
- SEAL-0 and SEAL-HARD: Models underperform especially on cross-lingual reasoning, false-premise detection, recent information, and rapidly changing questions.Performance degrades more when search results are unhelpful than when they contain conflicting answers.
- Search effects: Search improves accuracy for many models, but built-in agentic search outperforms retrieval-based prompting in several comparisons.O3 reaches 14.4% on SEAL-0 and 32.7% on SEAL-HARD; FRESHPROMPT slightly reduces GPT-4.1 accuracy from 15.0% to 14.6%.
- Search effects: Naive search can reduce accuracy, as GPT-4.1-MINI falls from 13.8% to 11.8% with built-in search.The benchmark’s noisy and conflicting results can make retrieval and integration harmful.
- Noise sensitivity: Advanced reasoning models are highly sensitive to noisy search, with DEEPSEEK-R1-671B falling from 22.4% to 11.0% under FRESHPROMPT.FRESHPROMPT helps on fast-changing and 2025-specific questions but sharply harms performance on never-changing and pre-2024 questions; SELF-ASK and FRESHPROMPT both harm DEEPSEEK-R1-671B and O3-MINI.
- Test-time scaling: Increasing test-time compute does not reliably improve accuracy: O3 reaches 11.7%, 17.1%, and 14.4% at low, medium, and high effort.O3-MINI plateaus, while O4-MINI peaks at low effort with 6.3% and declines at higher efforts.
- Human comparison: O3-HIGH reaches 28.0% accuracy, below human averages of 38.8% with open search and 50.4% with oracle access.The comparison highlights difficulties in both retrieving relevant information and reasoning through conflicting evidence.
- LONGSEAL: On LONGSEAL, accuracy declines as hard negatives increase, including GPT-4O-MINI’s drop from 24.0% to 6.3% and then 3.9%.The degradation is more pronounced in smaller or less capable models.
4 RELATED WORK
SEALQA builds on work measuring factuality, reasoning, and retrieval under dynamic or difficult information conditions. It emphasizes challenges involving misinformation, irrelevant context, conflicting sources, and time-sensitive knowledge.
- Knowledge conflict: Prior work identifies vulnerabilities to misinformation, irrelevant context, and conflicting sources, while retrieval quality strongly influences model outputs.Models may prioritize surface-level relevance over credibility indicators such as scientific references.
- Benchmark landscape: SEALQA complements benchmarks such as MMLU, MATH, GPQA, and HUMANITY’S LAST EXAM by evaluating factuality, reasoning, and retrieval challenges.The related benchmarks focus on overlapping but distinct aspects of language-model evaluation.
5 CONCLUSION
SEALQA evaluates search-augmented language models on difficult factual questions involving noisy or conflicting search results across three benchmark flavors. Results show that frontier models underperform, remain vulnerable to noise, and do not reliably benefit from more test-time compute.
- Benchmark scope: SEALQA contains SEAL-0, SEAL-HARD, and LONGSEAL for challenging search-augmented reasoning and long-context needle-in-a-haystack evaluation.LONGSEAL tests identifying relevant information among distractors.
- Main findings: Frontier language models, including agentic models with search tools, underperform and remain vulnerable to noisy search results.The benchmark exposes limitations across its different evaluation settings.
- Main findings: Increasing test-time compute often fails to produce reliable performance gains.Performance may plateau or decline as reasoning effort increases.
- Main findings: LONGSEAL shows reduced susceptibility to lost-in-the-middle effects but continued difficulty identifying relevant information amid distractors.The main challenge is reliable relevance identification in long contexts.
CODE OF ETHICS AND ETHICS STATEMENT
The paper states that SEALQA uses publicly available sources for academic research and excludes harmful content and private information.
- Data use: SEALQA sources are publicly available, used for academic research, and handled in compliance with original copyright terms.The dataset is freely available to researchers for academic purposes.
- Safety and integrity: The authors verify that the data exclude racial discrimination, violence, and private information.The paper states that the data and experiments follow scientific standards supporting authenticity and accuracy.
REPRODUCIBILITY
The reproducibility materials point readers to the dataset, annotation, and experimental-settings sections, while SEAL-HARD remains challenging for frontier LLMs even after release.
- The datasets and annotation process are detailed in Section 2, with experimental settings presented in Section 3.
- SEAL-HARD continues to challenge current frontier LLMs despite potential data contamination or direct agentic-model access.
B SAMPLE SEALQA QUESTIONS
The sample questions illustrate broad reasoning coverage and an auto-rater that distinguishes correct, incorrect, and not-attempted answers using factual completeness and consistency criteria.
- SEALQA questions test a broad range of reasoning skills often overlooked in existing benchmarks.
- The grading template assigns CORRECT, INCORRECT, or NOT_ATTEMPTED to each predicted answer.
- Answers are incorrect when factual statements contradict the gold target, including hedged incorrect statements.
- Answers are not attempted when they omit required gold information without contradicting it.
- Answers can be correct when they contain the question-relevant gold information, even if they omit extra target details or differ in wording and order.
- The rubric rejects internally inconsistent answers and permits hedging only when the complete gold target is included without incorrect information.
D SEAL-HARD RESULTS BY QUESTION CATEGORY
SEAL-HARD results show uneven performance across question categories, with particularly weak results for cross-lingual reasoning, false-premise detection, and rapidly changing information; uniformly unhelpful search is especially damaging.
- Models perform poorly across SEAL-HARD question categories, especially on cross-lingual reasoning, false-premise detection, and rapidly changing information.
- Performance degrades more with uniformly unhelpful search results than with conflicting answers.
- SEAL-HARD evaluates search-based systems using FRESHPROMPT, while dagger-marked results use ChatGPT’s built-in search.
E SEAL-HARD RESULTS BY ANSWER TYPE
Answer-type breakdowns show that open-weight models more often decline to answer SEAL-HARD questions than the named proprietary models.
- Open-weight models such as LLAMA-4-SCOUT and DEEPSEEK-R1 choose NOT_ATTEMPTED more often than GPT-4.1, O4-MINI, and O3.
- Figure 8 breaks SEAL-HARD results down into correct, incorrect, and not-attempted answers.
G QUALITATIVE ANALYSIS
The qualitative analysis compares how six models reason and use external knowledge across different GPT-4.1 search configurations and reasoning models. It finds that search access and prompting change behavior, but inaccurate answers and degraded reasoning remain common.
- GPT-4.1 without search sometimes cites relevant URLs but often answers inaccurately because its knowledge is outdated.
- FRESHPROMPT improves GPT-4.1’s detection of false-premise questions and generally produces more concise responses.
- The analysis reveals clear differences across models in their reasoning and use of external knowledge.
H SAMPLE MODEL OUTPUTS
The sample outputs illustrate models reasoning through fact-seeking questions involving rankings, counts, borders, and rainforest descriptions. The accompanying tables identify cases where models begin correctly but later shift to incorrect conclusions, while search-based results use FRESHPROMPT unless built-in search is indicated.
- Table 13 marks reasoning that begins correctly in green and later shifts to an incorrect conclusion in red.
- The Spotify example shows a model repeatedly verifying yearly top-artist claims while working toward the female-artist answer.
- The ICPC example requires counting U.S. victories after 1997, while the border example asks for the longest continuous international border.
- The rainforest example combines geographic scale and biodiversity claims, including a stated 27% share of remaining tropical forest area and 10% of known species.
- Search-based results use FRESHPROMPT unless the model’s built-in search is specifically indicated.
- Table 14 presents SEAL-0 as a significant challenge for both closed and open-source models.