Source-linked AI summary
Arctic Questions, Missing Answers: A Dataset and Benchmark for LLM Abstention in Arctic Science
Benjamin Wilcox, Dawei Gao, Pradeeban Kathiravelu, Douglas Causey, Kewei Sha, Yunhe Feng
TL;DR
Frequent abstention alone does not show whether a model responds to the absence of a valid choice, a question that matters for evidence-grounded Arctic science. The paper builds ArcticQA and a paired benchmark to test this distinction, finding broad baseline variation but only a modest average abstention increase after answer removal.
Problem
Arctic scientific answers depend on specific conditions, and frequent abstention alone does not establish whether LLMs respond when no option is valid.
Source: §1 · p. 1Method
The paper builds ArcticQA from primary Arctic research with automated answer and distractor checks, then compares matched answer-present and answer-absent questions.
Source: §1 · p. 2Results
Answer-present abstention ranged from 0.0% to 63.0%, while answer removal increased abstention by 5.05 percentage points on average.
Source: §6 · p. 6Takeaways & Limitations
Evaluating Arctic scientific abstention should consider false abstention, false commitment, and the accuracy of non-abstaining responses jointly.
Source: §6 · p. 6The paired multiple-choice evaluation does not establish generalization to spontaneous abstention in open-ended scientific work or beyond the evaluated setting.
Source: §7 · p. 7
Abstract
from arXiv · showhide
Large language models (LLMs) should abstain from scientific multiple-choice questions when no option is valid, but frequent abstention alone does not demonstrate sensitivity to answer availability. We introduce ArcticQA, a dataset of 194 questions derived from primary Arctic research, with automated checks of answer support and distractor contradiction against source evidence. We further develop ArcticAbstain, a paired benchmark comparing answer-present and answer-absent conditions, with the correct answer replaced by a distractor in the latter and an explicit abstention option in both. We evaluate eight models from the Gemini, Claude, and ChatGPT families at high reasoning effort, with three trials per condition, yielding 9,312 recorded responses. Answer-present abstention rates range from 0.0% to 63.0%, whereas replacing the correct answer increases abstention by 5.05 percentage points on average. These findings highlight substantial baseline differences and the need to evaluate abstention frequency and responsiveness jointly. The dataset and benchmark are available at https://github.com/BenWilcox8/arctic-qa.
1 Introduction
Arctic scientific questions depend on local conditions, yet no dedicated benchmark had tested whether LLMs abstain appropriately when choices conflict with primary Arctic evidence. The paper introduces evidence-grounded ArcticQA and paired ArcticAbstain, finding wide variation in baseline abstention but a limited average increase after answer removal.
1 Introduction
- Arctic studies vary by location, season, population, and measurement conditions, and prior work documents spatial biases in terrestrial sampling and citation.A plausible general answer may therefore be incorrect within a particular question’s scope.Source: §1 · p. 1
- Prior abstention evaluations had not systematically tested scientific questions grounded in primary Arctic research, leaving performance under Arctic-specific evidence unresolved.Benchmark curation must make conditions explicit and answers traceable to findings while evaluating distractors within the same scope.Source: §1 · p. 1
- ArcticQA grounds questions in primary Arctic findings and separately checks answer support and distractor contradiction against source evidence.The checks preserve question scope because an option’s mere absence from a paper does not establish that it is false.Source: §1 · pp. 1–2
- ArcticAbstain pairs answer-present and answer-absent versions to measure false abstention, false commitment, and responsiveness beyond raw abstention rates.In the answer-absent version, the correct answer is replaced by a distractor while the question and other options remain.Source: §1 · p. 2
- Answer-present abstention ranged from 0.0% to 63.0%, while replacing the correct answer increased abstention by 5.05 percentage points on average.The evaluation covered eight models on 194 questions at high reasoning effort.Source: §1 · p. 2
2 Related Work
Prior work studies abstention, scientific QA, and distractor validation, but the cited approaches do not directly evaluate responses to self-contained, source-bounded Arctic multiple-choice items.
2 Related Work
- Abstain-QA evaluates willingness to answer across answerable and unanswerable prompts, while AbstentionBench broadens analysis across unanswerable question types and reports reasoning-oriented training can increase forced problem solving.Source: §2 · p. 2
- Scientific abstention systems may perturb context or combine claim decomposition with retrieval and verification, whereas ArcticQA tests a self-contained item without those pipelines.The cited claim-verification work includes SciFact-Open.Source: §2 · pp. 2–3
- ArcticQA draws on paper-grounded QA and evidence-bounded question design, while requiring standalone questions and source checks for distractor validity (Wan et al., 2024).Prior work treats distractor generation and evaluation as distinct tasks; ArcticQA uses per-option checks and an explicitly non-gold machine-acceptance label.Source: §2 · p. 3
3 ArcticQA Dataset
ArcticQA builds questions around scoped findings from primary Arctic research and uses evidence checks to validate both answers and distractors.
3 ArcticQA Dataset
- ArcticQA links each question to a specific finding and supporting evidence, with validation of the answer and each distractor within the question’s scope.The construction pipeline comprises paper selection, finding extraction and question generation, QA validation, and distractor generation and validation.Source: §3 · p. 3
3.1 Paper selection and eligibility
- Metadata filtering reduced the search set to 4,420 candidate papers for full-text eligibility assessment; geographic eligibility used a 66.56◦N threshold for terrestrial studies or a fixed list of eligible Arctic marine regions.Source: §3.1 · p. 3
3.2 Finding extraction and question generation
- An LLM identifies and validates a paper’s primary finding, then a separate writer formulates a standalone question that preserves its scope and conditions.The answer-first approach followed by answer reconstruction adapts scientific-paper QA generation (Wan et al., 2024) and draws on roundtrip consistency.Source: §3.2 · p. 3
3.3 QA validation
- A blinded reconstructor answers without seeing the proposed answer, and a separate verifier checks entailment, scope, claim type, and whether the question context discloses the answer.Only QA pairs that pass these checks proceed to distractor generation.Source: §3.3 · p. 3
3.4 Distractor generation and validation
- Four retained distractors must be contradicted by specific source evidence within the question’s scope; mere absence from the source is insufficient.Candidates are rejected if a reasonable interpretation could support them as correct, including under the stated location and time conditions.Source: §3.4 · p. 4
4 ArcticAbstain Benchmark
ArcticAbstain measures abstention on matched Arctic science questions with and without a valid answer among the options. Its paired conditions distinguish abstaining despite a valid option from committing when none is valid.
4 ArcticAbstain Benchmark
- ArcticAbstain compares matched answer-present and answer-absent questions to measure how abstention changes when a correct option is unavailable.The absent condition replaces the correct answer with a validated distractor.Source: §4 · p. 4
4.1 Paired abstention protocol
- Both conditions retain the same question, three shared distractors, and an explicit abstention option; the correct answer is replaced in the absent condition.The options’ order is randomized to reduce systematic effects of option position.Source: §4.1 · p. 4
4.1.1 Answer-present condition
- In the answer-present condition, choosing the abstention option is false abstention, while choosing the gold answer is correct.The item contains the gold answer, three distractors, and the abstention option.Source: §4.1.1 · p. 4
4.1.2 Answer-absent condition
- In the answer-absent condition, abstention is the only correct response, and selecting any distractor is false commitment.No substantive option is valid within the question’s stated scope; the underlying question remains answerable from its source.Source: §4.1.2 · p. 4
5 Experiments
Across six experimental subsections, the study evaluates eight models using answer-present and answer-absent questions, multiple abstention and accuracy metrics, and repeated trials. Results show wide variation in baseline abstention, modest average increases after correct-answer removal, and a need to interpret abstention alongside committed-answer accuracy.
5.1 Experimental setup
- The evaluation uses eight models from Gemini, Claude, and ChatGPT, with high reasoning effort and three trials per condition for 194 questions, totaling 9,312 recorded responses.Invalid responses are excluded from metric calculations, so recorded-response totals differ from valid-response metric denominators.Source: §5.1 · p. 4
5.2 Evaluation metrics
- Metrics distinguish false abstentions from appropriate abstentions and assess overall success, abstention precision, and abstention recall across both answer-availability conditions.ACC credits correct answers when a valid option is present and abstentions when none is valid; recall’s denominator also includes incorrect substantive answers from both conditions.Source: §5.2 · p. 5
- Overall abstention rate measures withholding frequency but combines false and appropriate abstentions, while R-Acc measures answer accuracy conditional on selecting a substantive option.R-Acc should be interpreted alongside abstention rate to account for how frequently a model answers.Source: §5.2 · p. 5
5.3 Aggregation and statistical analysis
- Aggregate metrics are medians across three trials, while paired condition rates pool trials; question-level bootstrap intervals and sign-flip tests use Holm correction across eight models.The bootstrap uses 10,000 resamples, and adjusted p < 0.05 is the significance threshold.Source: §5.3 · p. 5
5.4.1 Variation in baseline abstention
- Answer-present abstention ranges from 0.0% to 63.0%, and model-family membership alone does not characterize behavior in this evaluation.ChatGPT Astra has the highest rate at 63.0%.Source: §5.4.1 · pp. 5–6
5.4.2 Response to correct-answer removal
- 5.05 percentage points is the mean abstention increase after correct-answer replacement; shifts range from 1.5 to 11.1 points, with Holm-corrected significance for two models.Claude Fable 5.1 and ChatGPT Astra increase by 10.8 and 11.1 percentage points, respectively; Figure 1 displays each model’s paired rates.Source: §5.4.2 · p. 6, Figure 1 · p. 6
5.4.3 Accuracy and abstention must be interpreted jointly
- ChatGPT Astra has the highest median ACC (0.541), abstention F1-score (0.618), and R-Acc (0.540), yet its answer-present abstention rate is 63.0%.Its overall abstention rate is 0.686; all models’ conditional answer accuracy and tendency to abstain capture distinct aspects of performance and are both needed to assess reliability.Source: §5.4.3 · p. 6
6 Conclusion
The paper introduces ArcticQA and ArcticAbstain for evaluating LLM abstention in Arctic science. Across eight models, answer-present abstention varied widely, and removing the correct answer increased abstention by 5.05 percentage points on average, underscoring the need to assess abstention frequency and responsiveness together.
6 Conclusion
- ArcticQA is grounded in primary Arctic research, and ArcticAbstain pairs answer-present and answer-absent conditions to evaluate LLM abstention.Source: §6 · p. 6
- Removing the correct answer increased abstention by 5.05 percentage points on average, while answer-present rates ranged from 0.0% to 63.0%.The findings distinguish baseline tendency to abstain from responsiveness to answer availability.Source: §6 · p. 6
- Reliable evaluation should consider false abstention, false commitment, and the accuracy of non-abstaining responses jointly.The dataset and benchmark provide a foundation for assessing and improving reliability in Arctic scientific question answering.Source: §6 · p. 6
7 Limitations
ArcticAbstain is a geographically bounded case study whose coverage depends on discovery queries, full-text availability, and model-guided processing order. Its dataset validation and evaluation setup also limit what the results establish.
7 Limitations
- Coverage reflects discovery queries, full-text availability, and model-guided processing order, so the dataset is not representative of all Arctic research or other scientific domains.Source: §7 · p. 6
- Dataset construction uses model-generated content and automated validation without domain-expert review, leaving the residual error rate unmeasured.machine_accepted_unverified denotes automated acceptance, not expert-verified ground truth.Source: §7 · p. 6
- The evaluation covers 194 questions and eight models at high reasoning effort, with metrics combining answer-present and answer-absent conditions.The paired task assesses rejection of invalid option sets rather than spontaneous abstention in open-ended scientific work.Source: §7 · p. 7, Table 1 · p. 7, Table 2 · p. 7
- The results do not establish training exposure, explain observed response policies, or demonstrate generalization beyond the evaluated setting.Source: §7 · p. 7