UQ: Assessing Language Models on Unsolved Questions

Fan Nie; Ken Ziyu Liu; Zihao Wang; Rui Sun; Wei Liu; Weijia Shi; Huaxiu Yao; Linjun Zhang; Andrew Y. Ng; James Zou; Sanmi Koyejo; Yejin Choi; Percy Liang; Niklas Muennighoff

Source-linked AI summary

UQ: Assessing Language Models on Unsolved Questions

Fan Nie, Ken Ziyu Liu, Zihao Wang, Rui Sun, Wei Liu, Weijia Shi, Huaxiu Yao, Linjun Zhang, Andrew Y. Ng, James Zou, Sanmi Koyejo, Yejin Choi, Percy Liang, Niklas Muennighoff

arXiv:2508.17580v1Aug 25, 2025cs.CLcs.AIcs.LG

TL;DR

Existing benchmarks are increasingly saturated or trade off difficulty against real-world relevance, so UQ evaluates models on naturally arising unsolved questions instead. It combines a curated dataset, oracle-free validators, and community verification; compound validators improve validation accuracy but remain limited in precision.

Problem
Existing benchmarks are increasingly saturated, while exam-style and user-interaction benchmarks struggle to jointly provide difficulty and real-world relevance.
Method
UQ combines a curated dataset of unsolved questions, compound LLM-based validators, and community verification to evaluate candidate solutions.
Results
Compound validation strategies improve Claude 3.7 Sonnet accuracy from 21.6% to 73.2% and precision from 13.26% to 20%.
Takeaways & Limitations
UQ provides an evaluation paradigm centered on difficult, realistic questions whose solutions can yield meaningful insights despite lacking ground-truth answers.
Takeaways & Limitations
Validator evaluation relies partly on surrogate data that may not perfectly match the UQ-Dataset distribution.

Abstract

from arXiv · show

Benchmarks shape progress in AI research. A useful benchmark should be both difficult and realistic: questions should challenge frontier models while also reflecting real-world usage. Yet, current paradigms face a difficulty-realism tension: exam-style benchmarks are often made artificially difficult with limited real-world value, while benchmarks based on real user interaction often skew toward easy, high-frequency problems. In this work, we explore a radically different paradigm: assessing models on unsolved questions. Rather than a static benchmark scored once, we curate unsolved questions and evaluate models asynchronously over time with validator-assisted screening and community verification. We introduce UQ, a testbed of 500 challenging, diverse questions sourced from Stack Exchange, spanning topics from CS theory and math to sci-fi and history, probing capabilities including reasoning, factuality, and browsing. UQ is difficult and realistic by construction: unsolved questions are often hard and naturally arise when humans seek answers, thus solving them yields direct real-world value. Our contributions are threefold: (1) UQ-Dataset and its collection pipeline combining rule-based filters, LLM judges, and human review to ensure question quality (e.g., well-defined and difficult); (2) UQ-Validators, compound validation strategies that leverage the generator-validator gap to provide evaluation signals and pre-screen candidate solutions for human review; and (3) UQ-Platform, an open platform where experts collectively verify questions and solutions. The top model passes UQ-validation on only 15% of questions, and preliminary human verification has already identified correct answers among those that passed. UQ charts a path for evaluating frontier models on real-world, open-ended challenges, where success pushes the frontier of human knowledge. We release UQ at https://uq.stanford.edu.

1 Introduction

UQ proposes evaluating language models on 500 curated unsolved questions, addressing the difficulty–realism tension in existing benchmarks while introducing dataset curation, automated validation, and community verification. Its components provide challenging real-world inputs, screening signals for human review, and a platform for continuous knowledge-driven evaluation.

26.6% versus 9.1%: OpenAI Deep Research surpassed o1 on Humanity’s Last Exam within weeks, illustrating how quickly difficult benchmarks can saturate.Existing benchmarks such as MMLU [12], GPQA [43], and ARC-AGI-1 [4] have also become saturated by frontier models.
Unsolved questions jointly offer difficulty and realism, unlike exam benchmarks that can be contrived and real-world-query benchmarks that can be easy, saturated, or manipulable.They arise naturally from human information-seeking and retain intrinsic value independent of model performance, but require validating question quality and candidate solutions without ground-truth answers.
UQ-Dataset contains 500 diverse, challenging Stack Exchange questions curated through engagement filters, LLM judgments, and PhD-level human review.The pipeline screens for well-definedness, difficulty, approachability, and objectiveness across STEM and non-STEM domains, including mathematics, physics, CS theory, bioacoustics, sci-fi, and mythology.
UQ-Validators combine low-level correctness and consistency checks, repeated or iterated judgments, and voting or sequential aggregation to screen candidate answers before human verification.The framework leverages the observed generator–validator gap and transfer across datasets, using frontier models as stronger validators than generators.
UQ-Platform hosts questions, model answers, validation results, and provenance while enabling users and developers to submit, review, and rate contributions.This live platform completes the evaluation cycle through continuous community-driven verification and reproducibility.
Because unsolved questions may lack perfect automated evaluation or be solved elsewhere, UQ treats its dataset, validators, and platform as complementary components for stress testing, expert review, and advancing knowledge.The system can expand beyond its 7,000+ candidate questions through public and community-driven contributions as models improve.

2 UQ-Dataset: A Collection of Unsolved Questions with Desirable Properties

UQ-Dataset is a curated collection of 500 challenging, unsolved Stack Exchange questions selected through rule-based, LLM-based, and human filtering. Its multi-stage process increases difficulty and quality while supporting ongoing updates as questions are solved or verified.

2.1 Dataset Creation: The creation pipeline combines engagement-based heuristic rules, repeated dual-model judgments, and human review to select questions that are old, high-interest, difficult, well-defined, approachable, and objective.Rule-based filters require questions to be at least two years old, meet site-dependent view and vote thresholds, rank highly among unanswered questions, and have no answers; LLM criteria use averaged numerical scores and unanimous binary votes.
2.3 Dataset Curation and Updates: UQ includes a 25-question UQDiamond subset, a 30-question held-out development set, and semi-live curation that replaces solved questions and credits models whose answers are human-verified.The benchmark can incorporate newly verified unsolved questions over time and may paraphrase verified model answers back to their original sources when appropriate.
2.2 Dataset Analysis: 500 questions remain after filtering roughly 3 million candidates: rule-based filtering leaves 33,916, LLM filtering 7,685, and human review produces the final dataset.The questions come from 80 manually selected Stack Exchange communities; human review removes duplicates, trivial, off-topic, and policy-violating items.
2.2 Dataset Analysis: LLM filtering lowers averaged expert solvability from 77.8% to 32.2% and attempted-answer correctness from 51.2% to 14.1%, while binary quality criteria reach 100%.The filter uses a general-purpose model to attempt answers and a reasoning model to assess difficulty, approachability, well-definedness, and objectivity.
2.2 Dataset Analysis: The dataset is dominated by Science, followed by Technology and Life & Arts, with domains probing different capabilities such as open-ended proofs and browsing.Questions are distributed across high-level Stack Exchange domains and sites, and the platform provides human- and LLM-friendly formats with markdown and metadata.

3 UQ-Validators: Assessing Candidate LLM Solutions to Unsolved Questions

UQ-Validators use oracle-free strategies to screen candidate answers when ground-truth is unavailable, prioritizing high precision and human-review assistance over proving correctness. On HLE surrogate data, compound validators improve accuracy and precision, transfer to UQ, reduce bias, but remain unreliable for precise rankings and high-confidence approval.

3.1 Motivation: Generator-Validator Gap Widens with Model Capability and Shows Transfer: At 20% answer accuracy, o3 achieves 65% validation accuracy, and validator performance transfers to held-out UQ without additional tuning.As model capability increases, validation accuracy improves faster than answer accuracy; transferred validators show similar accuracy patterns and generator–validator gaps on UQ and HLE [41].
3.2 Validator Design Goal and Strategies: UQ-Validators are designed to minimize false positives, combining low-level correctness, fact/logic, and cycle-consistency checks with reflection, sampling, voting, or staged pipeline verification.The default performant pipeline uses iterative reflection over cycle consistency, fact/logic checking, and correctness across three verification turns.
Finding #1: Compound Validator Strategies Outperform Simple Prompting Baselines: 73.2% validation accuracy and 20% precision, versus 21.6% and 13.26% for vanilla prompting, show that compound strategies improve validation performance for Claude 3.7 Sonnet.Across validators, unanimous voting is stricter than majority voting; structured prompts and, less consistently, additional calls, models, and sequential steps can improve performance.
Finding #2: Attaining High Precision is Difficult: The best UQ-Validator reaches only 40% precision, with a sharp precision–recall tradeoff that makes high-confidence approval difficult on extremely hard questions.Few correct answers limit true positives, while adding iterations does not reliably increase precision; the 5-iter o3 validator lowers both precision and recall relative to the 3-iter version.
Finding #2: Attaining High Precision is Difficult: Human reviewers largely agree with the best validator’s verdicts and reasoning traces across 25 questions, supporting its use to assist downstream review despite imperfect automated validation.Because validators lack ground-truth answers and can be wrong, the evaluation uses HLE as a difficult, diverse surrogate with known answers; domain-specific tools such as Lean may be stronger but less general.
Finding #3: Simple Validators Show Over-Optimism and Self-Bias: Simple LLM validators overestimate self and sibling models, while increasing capability reduces but does not eliminate this self-evaluation bias.Gemini favors itself, Claude is optimistic across answer models, and OpenAI o-series models overrate sibling o-series systems.
Finding #4: Compound Validator Strategies Mitigate Over-Optimism and Self-Bias: The 3-iter o3 pipeline largely removes over-optimism and preferential treatment of same-family models, improving fairness across evaluated models.This compound validator’s bias reduction accompanies the broader gains from scaling validation strategies.
Finding #5: Model Rankings Are Unstable Across Validator Performance: Validator-based model rankings shift unpredictably across validator strengths, so UQ-Validators cannot safely produce automated leaderboards; better answer generators also need not be better validators.Every tested answer model reaches first place under at least one validator, whereas the strongest o3 pipeline converges to ground truth; this motivates community verification through UQ-Platform.

4 UQ-Platform: An Open Platform for Community-Based Evaluation

UQ-Platform is an open, community-based hub that completes evaluation through expert verification of model answers and continuous maintenance of unsolved questions. Its public workflows and compounding feedback aim to improve evaluation quality over time.

UQ-Platform enables domain experts to verify model responses, assess question quality, and maintain unsolved questions, making evaluation continuous rather than static.It is publicly and freely accessible and hosts the UQ-Dataset and UQ-Validator results.
The platform combines question browsing, reproducible answer submissions, public human reviews, additional AI reviews, resolution statistics, and model rankings based on verified resolved questions.Candidate answers appear alongside confidence and correctness ratings, comments, validator results, and prompts supporting reproducibility; initial rankings may be limited because few questions are solved and not all answers are verified.
Designed as an AI-native mirror of Stack Exchange, UQ-Platform centralizes frontier-model attempts, expert assessments, reproducibility information, and performance tracking on questions reflecting actual information needs.
UQ-Platform’s evaluation quality compounds because validator screening reduces human verification effort, while model improvements and human feedback improve validators and increase resolvable questions.The resulting cycle is intended to make the platform increasingly useful to reviewers and answer-seekers.
The platform hopes to sustain expert participation through convenience, public attribution, and educational value, including verifier recognition and opportunities to critique model errors and hallucinated citations.

5 Partial Model Evaluation

Frontier models achieve low UQ-Validator pass rates, with reasoning models ranking above non-reasoning models. Among partially human-verified candidates, most solutions are incorrect, though 10 questions passed secondary validation and O3-PRO produced at least four meaningful accepted answers.

UQ-Validator pass rates: All models had low UQ-Validator pass rates, with o3 and Gemini 2.5 Pro outperforming Claude 3.7 Sonnet and GPT-4o.The model ranking roughly mirrors rankings from recent benchmarks.
Human verification: 10 questions passed secondary human validation, while most solutions were wrong among the 91 verified questions out of 144 UQ-validation passes.The validated questions included 6 math, 1 physics, 1 Stack Overflow, 1 statistics, and 1 retrocomputing question.
Human verification: O3-PRO produced meaningful answers to at least four questions accepted by human reviewers, ending the project’s initial streak of zero verified solutions.On the UQ diamond subset, four answers passed UQ-Validator, but none of the three human-verified answers were correct.
Human verification: Candidate answers are available on UQ-Platform for community verification, which will inform future updates to human verification results.Samples of human-verified incorrect and correct answers are visualized in Appendices E.3 and E.4.

6 Related Work

Prior benchmarks either use expert-crafted questions that diverge from real information needs or realistic interactions that quickly saturate and skew simple. UQ instead applies LLM validators to triage responses to unsolved questions without ground-truth answers.

Exam-based Benchmarks: Exam-based benchmarks broadened from narrow skills to expert-crafted, artificially difficult questions, exposing edge-case failures but diverging from how human information needs arise.These suites include SQuAD [42], GLUE [51], MMLU [12], MATH [13], HumanEval [2], and newer tests such as FrontierMath and GPQA [43].
Realistic Benchmarks: Realistic benchmarks draw on user interactions, preferences, software tasks, or tool use, but retrieval systems solve many queries, prompts skew simple, and terminal-bench reaches 50% pass rates within months.These benchmarks also remain vulnerable to adversarial manipulation [15] [45].
LLM-as-a-Judge: LLM-as-a-judge methods address evaluation when exact-match metrics fall short, but UQ uses LLM validators (§3) to triage unsolved-question responses without ground truth against clear criteria.Prior work reports roughly 80 % human agreement for GPT-4 judges while documenting position and verbosity biases [66].

7 Discussions & Limitations

The discussion identifies dataset-representation and annotation limits, validator-design and evaluation constraints, and platform biases that affect UQ’s interpretation and scalability. UQ is presented as a complementary objective rather than a broadly representative measure of unsolved questions.

7.1 UQ-Dataset: UQ-Dataset questions may reflect neglect, historical search difficulty, Stack Exchange source bias, and STEM skew rather than intrinsic unsolvedness or broad real-world representativeness.The authors mitigate these issues with validator screening, manual inspection, and engagement filtering, but note that research-level open problems such as are underrepresented.
7.1 UQ-Dataset: The authors frame human-generated questions as a distinct, complementary evaluation objective despite measurement-validity concerns raised by prior work [48] [44].They argue that UQ questions arise organically and that solving them provides direct real-world value.
7.2 UQ-Validators: UQ-Validators rely on surrogate data, have an underexplored oracle-free design space, incur cost and latency at higher capacity, and may miss hallucinated citations.External data such as Humanity’s Last Exam may mismatch UQ’s distribution; stronger models and ensembles improve accuracy, while browsing-specialized systems such as deep research agents could improve reference verification.
7.3 UQ-Platform: UQ-Platform participation may be biased toward LLM hobbyists and researchers, offers sparse ranking signal at launch, and requires continuous moderation against adversarial engagement [15] [45].The platform nevertheless provides an AI-native venue for accessing and verifying generative-AI solutions, unlike Stack Exchange’s heavily censored setting.

8 Concluding Remarks … A.3 Additional Details on Human Filtering

UQ proposes an oracle-free evaluation paradigm built from naturally difficult, realistic unsolved questions, supported by dataset construction, validation, and community evaluation. Its appendix details broad Stack Exchange sourcing, heuristic filtering, and human selection for quality and domain diversity.

8 Concluding Remarks: UQ combines UQ-Dataset inputs, UQ-Validators for outputs, and UQ-Platform community evaluation to support future scaling in hard-to-verify domains.The authors envision newer datasets with harder or differently sourced questions and future exploration of generator-validator interaction.
A.1 List of Source Stack Exchange Sites: The dataset began with unanswered questions from 80 Stack Exchange sites, but only 35 sites, or 43.75%, remained after filtering.Stack Overflow and its multilingual sites were counted as one site.
A.2 Additional Details on Rule-based Filtering: Rule-based filtering required questions to be at least two years old, sufficiently viewed and upvoted, highly ranked among unanswered questions, and answer-free.Site-dependent thresholds included 200–2000 views and 5–75 net upvotes; a views-to-votes ratio of at most 5000 further excluded weakly engaged questions.
A.2 Additional Details on Rule-based Filtering: Additional rules removed questions with “why” in the title, images, or unrelated tags such as “homework,” “advice,” “policy,” and “recommendation.”These heuristics trim millions of unanswered questions before LLM judging and expert review rather than exhaustively defining eligibility.
A.3 Additional Details on Human Filtering: Human filtering used top-ranked unanswered questions for several high-volume communities because strong moderation and engagement made them likely to satisfy the desired unsolved-question properties.For smaller or specialized communities, reviewers jointly considered content and engagement signals such as vote counts.
A.3 Additional Details on Human Filtering: The high-volume selection allocated community-specific quotas, including 200 MathOverflow questions, 108 Mathematics questions, 40 Theoretical Computer Science questions, and 35 Science Fiction & Fantasy questions.Other listed communities used thresholds such as votes ≥10 or smaller top-ranked selections.
A.3 Additional Details on Human Filtering: Smaller communities received manual or mixed selection, including 5 History questions, 2 Linguistics questions, and four-question selections for Retrocomputing and Quantum Computing.These selections combined voting with review where specified.
A.3 Additional Details on Human Filtering: Remaining sites were manually selected across diverse domains, ensuring high-quality questions that automated filtering might miss in lower-volume or specialized communities.Examples include Biology, Artificial Intelligence, Economics, Puzzling, Medical Sciences, Quantitative Finance, and Chemistry.

A.4 Additional Dataset Statistics … B.1 Additional Discussions on Domain-Specific UQ-Validators

The supplementary material details dataset composition and update practices, then explains how domain-specific validators can improve precision while remaining outside UQ’s broadly applicable design. Together, these additions clarify dataset reproducibility and the tradeoffs between specialized and general validation.

A.4 Additional Dataset Statistics: The filtering statistics include overall stage counts, domain-specific counts across five Stack Exchange categories, and site-level breakdowns for both dataset subsets.Tables S2 and S3 report filtering counts, while Tables S4 and S5 report site-level composition for the 25-question diamond subset and full UQ-Dataset.
A.4 Additional Dataset Statistics: Table S2 reports question-pool sizes at each filtering stage, complementing the filtering statistics presented in Section 2.2.The table is also referenced alongside Figures 2 and 4.
A.4 Additional Dataset Statistics: Table S3 provides per-stage counts for Science, Technology, Life & Arts, Culture & Recreation, and Business categories.These five high-level domains are categorized by Stack Exchange.
A.5 Dataset Updates and Versioning: Explicit dataset versioning will tie results to specific snapshots and track question additions and removals over time.A new version may be issued when at least 20% of the UQ-Dataset is manually verified as solved by qualified domain experts; no update is currently planned.
A.5 Dataset Updates and Versioning: Version changes will be synchronized across the UQ-Platform, Hugging Face, GitHub, and the paper.This keeps public release channels aligned when the dataset is updated.
B.1 Additional Discussions on Domain-Specific UQ-Validators: Domain-specific invariants can produce stronger oracle-free validators, including formal proof checks, adversarial code tests, and conservation-law or dimensional-consistency checks.These techniques apply respectively to competition mathematics, programming challenges, and chemistry or physics.
B.1 Additional Discussions on Domain-Specific UQ-Validators: Hard-coded domain rules approach deterministic correctness filtering but sacrifice breadth, so UQ favors general checks and leaves richer specialized validators for future work.Designing specialized validators requires translating a domain’s formal specification into machine-checkable assertions.

B.2 Additional Results on Generator-Validator Gap · B.3 Additional Discussions on Human/UQ-Validator Agreement

Additional results show that validation ability generally outpaces answer-generation ability as models improve, although the trend is noisy and model-family dependent. Human/UQ-Validator agreement is currently difficult to measure because most models produce only false validation answers, with moderate agreement observed for Gemini 2.5 Pro.

B.2 Additional Results on Generator-Validator Gap: Validation accuracy grows faster than answer accuracy on hard questions, widening the generator–validator gap as model capability increases.Figure S1 extends the analysis with two additional models, though the trend is noisier than in the original result.
B.2 Additional Results on Generator-Validator Gap: The generator–validator gap remains model-family dependent: a stronger answer generator is not necessarily a stronger validator, with o3 outperforming Gemini 2.5 Pro as a validator.
B.3 Additional Discussions on Human/UQ-Validator Agreement: Most answer models produce false validation answers because the validation questions are difficult, so both UQ-Validators and human reviewers classify them as false.
B.3 Additional Discussions on Human/UQ-Validator Agreement: Cohen’s κ is undefined for these uniformly false cases, shown as “-” in Table S6, which augments Table 2 with an inter-rater reliability measure.
B.3 Additional Discussions on Human/UQ-Validator Agreement: Gemini 2.5 Pro achieves Cohen’s κ = 0.468 after producing one correct answer, indicating moderate human/UQ-Validator agreement.
B.3 Additional Discussions on Human/UQ-Validator Agreement: The paper expects human/UQ-Validator agreement measurements to become more meaningful as models improve and produce more correct answers.

B.4 Additional Results on UQ-Validators Performance … Finding #8: Weaker Models Fail Earlier in UQ-Validator Pipeline

Across UQ-Validator strategies, stricter consensus and sequential verification increase precision and accuracy while reducing recall, with multi-model ensembling achieving the strongest overall trade-off. Validation also benefits from additional inference and model diversity, while stronger answer models are more likely to pass all stages and weaker models fail earlier.

B.4 Additional Results on UQ-Validators Performance: Validator strength scales with model quality: baseline Correctness accuracy rises from ≈30% on Claude Sonnet 3.7 to ≈71% on O3.Stricter unanimous voting raises precision by ≈2–6 percentage points but can reduce recall by 20–40 percentage points.
B.4 Additional Results on UQ-Validators Performance: The o3 3-Iter pipeline achieves the best single-model trade-off, with 81.7% accuracy and 31.0% precision; tighter consensus mechanisms and multi-turn pipelines make validation stricter and convert recall into precision.
B.4 Additional Results on UQ-Validators Performance: 85.4% accuracy and 40.0% precision are achieved by a two-model, 3-Iter unanimous pipeline, although recall falls to 24.6%.This is the highest reported accuracy and precision, while majority voting over 3–5 models preserves approximately 80–91% recall at lower precision.
B.5 Additional Findings: Multi-model unanimous voting with o3 and Gemini 2.5 Pro attains the best tested validation accuracy, indicating that model diversity reduces judgment variance beyond additional single-model turns.Among single-call strategies, a structured Correctness prompt substantially outperforms the generic vanilla baseline at the same small budget.
Finding #7: Validation Strategies are (Somewhat) Amenable to Test-Time Scaling: Validation accuracy generally increases with more API calls, while sequential pipelines, unanimity voting, and deeper pipelines outperform single-prompt baselines at higher cost.Gains diminish as the call budget grows, creating a cost–accuracy trade-off; the o3 3-Iter pipeline is used as the main validator unless otherwise stated.
Finding #8: Weaker Models Fail Earlier in UQ-Validator Pipeline: Claude Opus 4 and DeepSeek-R1 frequently fail factual checks at Stage 2, suggesting fluent but factually unreliable answers.Failures are distributed across stages, indicating that each stage adds meaningful filtering without concentrating excessive strictness at the end.
Finding #8: Weaker Models Fail Earlier in UQ-Validator Pipeline: Stronger answer models fail less often in early validation stages and produce more fully validated answers, with o3-pro achieving the highest all-stage pass rate.o3-pro and Gemini 2.5 Pro rarely fail Stage 1, whereas GPT-4o and Claude Sonnet 3.7 fail early more frequently.

C Additional Experimental Details … E.2 Sample Judgment Reasoning Traces by UQ-Validator

The appendices specify model and evaluation settings, document human and Stack Exchange considerations, illustrate diverse UQ questions, and show human-reviewed UQ-Validator reasoning traces.

C.1 Model Versions: UQ uses deterministic temperature 0.0 sampling by default and reports the specific model versions used throughout the project in Table S8.Table S8 lists the model versions used throughout the project unless otherwise stated.
C.2 Additional Hyperparameters: Additional settings include gpt-4o at temperature 0.3 for candidate answers, o4-mini with three calls for quality judgments, high reasoning effort for o3-mini and o4-mini, and a 16,000-token Claude 3.7 Sonnet thinking budget.UQ-Validator pass rates use o3 with the three-iteration pipeline shown in Figure 6.
C.3 Anecdotal Human Performance: Human solvers have resolved 2 / 500 UQ-Dataset questions, providing a potential human-performance leaderboard entry while retaining those questions for model assessment.The two resolved questions comprise one solved by the authors and one by a Stack Exchange user.
D.1 Content Permissions and Licensing: UQ reuses Stack Exchange content under CC BY-SA, attributing source sites, linking original posts, marking edits or truncation, and noting the share-alike license in papers, the platform, and releases.The applicable license version depends on the post date, and Stack Exchange terms require appropriate attribution.
D.2 Uploading Candidate Answers to Stack Exchange: Posting verified AI answers requires site-specific policy compliance: some Stack Exchange sites permit referenced AI content, whereas Mathematics, Science Fiction & Fantasy, and Stack Overflow prohibit it.MathOverflow warns that AI-generated mathematical content may be deleted and can lead to moderator sanctions for repeated violations.
E.1 Sample Questions from UQ-Dataset: The sample UQ questions span mathematics, MathOverflow, science fiction, theoretical computer science, physics, and history, covering problems from power bases and measurable plane partitions to story identification, ETH, brane statistics, and Swedish-Finnish roads.Examples include whether every ring of integers extends to one with a power basis, whether a disk convolution can vanish, and whether an ETH-based 2^o(n) lower bound exists for n-bit inputs.
Sample Question from Physics: The examples include open-ended technical and historical questions, such as generalized brane quantum statistics and the first purpose-built overland route from Sweden into Swedish Finland.The physics example asks about generalized TQFTs and non-trivial brane examples, while the history example defines what counts as a road.
E.2 Sample Judgment Reasoning Traces by UQ-Validator: The appendix presents sample UQ-Validator judgment reasoning traces that were verified by human reviewers, with a ✓Human-reviewed box indicating approval.The traces are illustrative rather than a separate quantitative evaluation.

E.2.1 Sample #1: Mathematics · E.2.2 Sample #2: History

The mathematics sample presents a candidate proof of continuity for the Riesz component, but validator review finds its extension to all bounded continuous functions invalid. The history sample asks when and where the first purpose-built overland route from Sweden into Swedish Finland was constructed, under an explicit definition of “road.”

E.2.1 Sample #1: Mathematics: The mathematics question asks whether continuity of x↦φ_x(f) implies continuity of the Riesz component, after the author could prove only upper semicontinuity for compactly supported functions.The question concerns σ-compact Polish spaces and normalized positive linear functionals.
E.2.1 Sample #1: Mathematics: The candidate’s intended conclusion is continuity of x↦(φ_x)_0(f) for every f∈C(X), equivalently weak continuity of the associated Radon measures μ_x.The proposed proof assumes σ-compact Polish X, φ_x(1)=1, and continuity of x↦φ_x(f).
E.2.1 Sample #1: Mathematics: The candidate correctly establishes continuity for compactly supported functions, but this does not justify the advertised result for all bounded continuous functions.The validator specifically finds Steps 1–3 essentially correct while identifying Step 4 as the decisive gap.
E.2.1 Sample #1: Mathematics: The validator judged the mathematics answer factually incorrect because its claimed uniform approximation of every bounded continuous function by compactly supported functions is false.The counterexample f≡1 on ℝ shows the approximation cannot converge uniformly, so the final continuity claim for all of C(X) is unproved.
E.2.2 Sample #2: History: The history sample asks when and where the first overland route was constructed from Sweden into Swedish Finland, beyond the known Swedish post road through Åland [1].The question is motivated by limited evidence from the War of 1808–9 [2] and references to Finnish road networks [3] and a possible Finnmark path [4].
E.2.2 Sample #2: History: The historical evidence cited does not specify where the northern roads were, and the possible Finnmark path [4] is absent from the corresponding Finnish Wikipedia article.The source discussion reports only a vague reference to new roads into Finland and conflicting documentation about the path.
E.2.2 Sample #2: History: For the history question, “road” means a purpose-built or purpose-developed route used regionally, excluding gradually formed hunting tracks but including maintained routes between townships.This definition is intended to distinguish constructed infrastructure from organically developed paths.

Sample #2 Candidate Answer by Gemini 2.5 Pro … Sample #3 Human Verdict on Candidate Answer: Incorrect

This section presents candidate answers evaluated by UQ-Validator and human review, including incorrect answers resulting from imprecision, unsupported claims, factual fabrication, incomplete substantiation, and an incorrect citation. It also includes examples of plausible solutions or categorical constructions that do not fully establish their claims.

Sample #2 Candidate Answer by Gemini 2.5 Pro: The Gemini answer correctly identifies the Tornio route but is judged incorrect because it gives no firm date, concrete evidence, or citations for the first Sweden–Finland road.The route was formalized through the Swedish road and postal system by the early seventeenth century, including Tornio gästgiveri posts by 1622, but the answer only gives broad seventeenth- to eighteenth-century timing.
E.3 Sample Answers Passing UQ-Validator but Human-Verified As Incorrect: The UQ appendix frames these examples as question–answer pairs that passed UQ-Validator but were subsequently ruled incorrect during human verification.The section is referenced by Section 5 and reports 144 such pairs in total.
E.3.1 Sample #1: Science Fiction & Fantasy: The sample book-identification answer names Wilson Tucker’s The Year of the Quiet Sun and matches the recalled plot, but human review flags fabricated or incorrect publication details.The answer’s proposed solution agrees with the accepted identification, while its stated length and publication history are specifically challenged.
Sample #2 Question: No algorithm is known to recover eigenvalue signs from absolute values in truly sub-cubic time; best known methods cost O(n^3) or O(n^ω log(1/ε)) for accuracy ε.Absolute values reveal A^2, whose eigenspaces cannot distinguish +a_i from −a_i; resolving the ambiguity requires applying A, and no lower-bound proof establishes an Ω(n^3) cost.
Sample #2 Human Verdict on Candidate Answer: Incorrect: The human verdict rejects the eigenvalue-sign answer because it only states that no faster method is known, without providing either a faster algorithm or a lower-bound proof.The candidate argues that known approaches require essentially full spectral work, but this does not establish impossibility.
Sample #3 Question: The forcing answer proposes the effective topos Eff[ P ] as the canonical setting, replacing Sets so internally dense subobjects correspond to recursively enumerable dense sets.The resulting sheaf topos contains a generic meeting precisely the computably given dense sets required in recursion-theoretic forcing.
1. Reminder: ordinary forcing in topos language: Ordinary forcing is expressed as a sheaf topos on a poset equipped with the coverage generated by dense sieves, with the generic meeting all internally dense subobjects.Over Sets, the externally visible dense subobjects are ordinary set-theoretic dense subsets, yielding familiar generics such as Cohen-generic filters.
2. What changes for recursion-theoretic forcing?: Changing the base topos to Eff makes internal quantification over subobjects of the natural numbers correspond to recursively enumerable sets, producing effective generics.The effective topos validates the computational interpretation through its correspondence with partial recursive functions and its internal principles.
2. What changes for recursion-theoretic forcing?: Eff[P] interprets its generic object as an ω-sequence meeting all r.e. dense sets.
3. Why this deserves to be called “canonical”: Eff[P] is canonical because it follows the usual forcing recipe, is initial among toposes mapping geometrically to Eff with a generic filter, and specializes to the ordinary forcing topos over Sets.
5. Take-away: For any computable forcing poset P, forming Eff[P] gives the canonical topos whose generic object is an effective generic filter meeting every computably (r.e.) dense set.
Sample #3 Human Verdict on Candidate Answer: Incorrect: The cited reference by J. van Oosten and A. Pitts is incorrect because that exact reference does not exist in Proceedings of the American Mathematical Society 120 (1994), issue 3.

E.3.4 Sample #4: Theoretical Computer Science · Related partial results

The question of whether plain DPLL with random variable selection solves every LLL-satisfiable formula in polynomial expected time remains open. Existing results explain the gap between constructive resampling and DPLL search, but provide neither a general upper bound nor a matching counterexample.

E.3.4 Sample #4: Theoretical Computer Science: The procedure starts with a random assignment and, while an unsatisfied clause exists, resamples all variables in that clause.
E.3.4 Sample #4: Theoretical Computer Science: The unresolved problem asks whether DPLL, which uses random variable splits and standard unit-pure-literal pruning [3], also has polynomial expected runtime on every such formula.Unlike resampling, DPLL explores both assignments through a search tree before backtracking.
E.3.4 Sample #4: Theoretical Computer Science: A polynomial upper bound would require showing that every tree-resolution refutation inconsistent with the unknown satisfying assignment is short, which is currently unavailable.Moser–Tardos avoids assignments inconsistent with the current partial assignment, whereas DPLL must exhaust one branch before trying the other.
E.3.4 Sample #4: Theoretical Computer Science: A lower bound would require a satisfiable formula satisfying p = 2^-t and d < 2^t/e - 1 that forces random-branching DPLL to build an exponentially large search tree.Known lower-bound constructions violate the LLL condition, so the challenge is embedding one inside a satisfiable wrapper without exceeding the dependency threshold.
Related partial results: Random k-CNF results below the satisfiability threshold do not settle worst-case LLL instances, while known satisfiable-formula DPLL lower bounds use constants too large for the LLL condition.These partial results constrain neither side of the original worst-case question.
Related partial results: Allowing clause learning with suitable restarts can simulate Moser–Tardos and yield a polynomial bound, but this extends beyond the plain DPLL framework under study.The positive result therefore applies only after adding inference capabilities excluded by the original question.
Related partial results: Plain random-variable DPLL is open on LLL-satisfiable formulas: neither a polynomial expected-time proof nor a super-polynomial lower-bound instance is known.The question is explicitly unresolved, with no proof of universal polynomial expected time and no qualifying hard instance.

Sample #4 Human Verdict on Candidate Answer: Incorrect … Sample #2 Human Verdict on Candidate Answer: Correct (Borderline)

Human verification distinguishes one candidate as incorrect because it does not resolve the conjecture and contains unsupported or difficult-to-verify claims, while another is judged correct but borderline. The verified examples show that candidate answers can solve difficult physics and mathematics questions using general-relativistic ray tracing, analytic approximations, and elementary flux methods.

Sample #4 Human Verdict on Candidate Answer: Incorrect: The Sample #4 candidate is incorrect because it neither determines whether the conjecture is true or false nor provides substantive content beyond a reasonable but non-trivial discussion.Its related-results claims are plausible but underspecified, uncited, and partly difficult to verify.
E.4 Sample Answers Passing UQ-Validator and Human-Verified As Correct: The validated-answer sample contains 144 question-answer pairs, including 90 human-verified questions, of which 10 are judged correctly solved.
E.4.1 Sample #1: Physics: For the black-hole physics question, the candidate combines gravitational shift, orbital Doppler beaming, black-hole shadowing, and black-body balance, with full general-relativistic ray tracing as the precise method.The procedure specifies choosing a Kerr black hole and orbit, constructing a local tetrad, tracing photon geodesics, computing redshift, integrating absorbed intensity, and imposing radiative equilibrium.
2) A simple analytic approximation: The analytic black-hole approximation ignores lensing, frame-dragging, and the shadow, yet captures order-unity beaming corrections for mildly relativistic orbits and enormous gravitational factors in deep potentials.
How one gets ∼890 ◦C for “Miller’s planet”: For Miller’s planet, gravitational time dilation alone predicts about 42 K, while full angular integration with extreme orbital Doppler beaming yields 1163 K, approximately 890 °C.The same template predicts temperatures of order 1000 K for the extreme orbits producing a 1 hr →60 000 hr dilation factor.
[From Question Original Poster on Physics]: The physics answer was accepted by the question’s author despite omitting a fully worked Doppler-beaming factor, because it provided enough detail for independent derivation.
E.4.2 Sample #2: Mathematics: For the Chern-number integral, the mathematics candidate rewrites the integrand as a Coulomb-field flux and uses the divergence theorem and local Jacobian signs to derive the piecewise constant result without Chern classes.The resulting value is 1 for −2 < m < 0, −1 for 0 < m < 2, and 0 otherwise.
Sample #2 Human Verdict on Candidate Answer: Correct (Borderline): The Sample #2 candidate is judged correct but borderline because it reaches the correct solution through a standard method, while possibly not satisfying the author’s preference for an even more direct calculation.

E.4.3 Sample #3: Mathematics / Math Overflow … Prompt of Iterated Reflection

This section presents a mathematics question that remains open beyond A_1 ⊊ A_2 alongside related statistics and science-fiction answers, then specifies structured prompts for filtering and validating question quality, correctness, consistency, factuality, reflection, and baseline judgments.

E.4.3 Sample #3: Mathematics / Math Overflow: The chain’s infinitude is open: unconditionally, only A_1 ⊊ A_2 is known, while later inclusions may remain proper or stabilize.Schanuel’s conjecture would imply a decisive positive answer, but no proof is known for general countable fields.
4. Are the 𝐴𝑛necessarily fields?: It is unknown whether every A_n is a field, and whether e belongs to any A_n; e is known only to lie outside A_1.Determining whether e lies in A_2 already requires resolving an outstanding problem in exponential Diophantine geometry.
Key Insight: For diagonal scaling D, the QR factor Q of an N×k Gaussian matrix follows the matrix angular central Gaussian distribution on V_k(ℝ^N), with parameter Σ = D^2.Its density is proportional to |Σ|^-k/2|Q^TΣ^-1Q|^-N/2.
Derivation: For k = 1, the result reduces to the angular central Gaussian density, and equal diagonal scales yield the uniform distribution on the sphere.When Σ = I_N, the general MACG distribution is uniform on the Stiefel manifold under Haar measure.
3. Case 𝐷2𝑀and Other Scalings: For D^pM, the same characterization uses Σ = D^2p, extending the Haar-measure case to arbitrary diagonal scalings.The QR factor remains MACG-distributed with the corresponding covariance parameter.
[From Question Original Poster on Statistics]: The statistics answer was judged correct or sound on initial human inspection, although some arguments may contain minor errors or proceed too quickly.The original poster conveyed appreciation while intending a more careful review.
Sample #1 Candiate Answer by Human: A human answer identifies the story as “Time is Out of Joint” (1971), from Jeff Hawke’s Cosmos, matching the Europa setting, time travel, and Aries computer.The candidate answer describes the interplanetary research mission and the Aries system developed by Drew Lockett.
E.6 Prompts for LLM-based Filtering: LLM-based filtering uses a dual-model setup in which GPT-4o generates a candidate answer and o4-mini judges question quality, difficulty, solvability, approachability, and objectivity.The filter evaluates whether questions are well-defined and difficult for both models and human experts.
Prompt of LLM-based Filtering: The filtering prompt compares each question with an LLM-generated answer and assesses answer correctness, expert solve probability, answerability, clarity, and objective verifiability.Questions must be valid, potentially solvable, clearly stated, unambiguous, and judged strictly and objectively.
E.7 Prompts for UQ-Validators: UQ-Validator prompts implement low-level correctness, fact/logic, and cycle-consistency checks alongside mid-level iterative sampling, while repeated sampling and voting require no prompts.High-level majority or unanimous voting aggregates separate inference calls rather than using additional prompt text.
Prompt of Cycle Consistency: The cycle-consistency prompt verifies that an answer addresses the original question’s core topic, key requirements, and requested solution rather than drifting to tangential content.It compares the original question with one inferred from the answer and returns a structured [[Y]] or [[N]] relevance decision.
Prompt of Fact/logic check: The fact/logic prompt checks answers for incorrect facts, calculations, formulas, claims, reasoning, misleading explanations, incompleteness, ambiguity, and misconceptions.Minor typographical or grammatical errors are tolerated, but clear factual or logical flaws produce rejection.
Prompt of Correctness: The correctness prompt accepts an answer only when it is correct, useful, complete, and sufficiently detailed, requiring at least 95% confidence except that puzzle completeness may be relaxed.Partial, high-level, open-problem, incomplete, or erroneous answers are rejected.
Prompt of Vanilla Baseline: The vanilla baseline prompt simply asks whether the answer is correct for the question and requires an Accepted: [[Y]] or Accepted: [[N]] decision.Unlike the stricter correctness prompt, the supplied baseline text does not specify additional evaluation dimensions.
Prompt of Iterated Reflection: The iterated-reflection prompt asks the judge to reconsider its assessment and issue a final [[Y]] or [[N]] decision after checking confidence.The reflection step preserves the same decision format while explicitly requesting a second evaluation.