Source-linked AI summary
Qworld: Question-Specific Evaluation Criteria for LLMs
Shanghua Gao, Yuchang Su, Pengwei Sui, Curtis Ginder, Marinka Zitnik
TL;DR
Open-ended LLM evaluation needs criteria that reflect each question’s context rather than fixed task-level rubrics. Qworld generates such criteria through recursive expansion, achieving strong expert coverage and novelty while revealing capability differences that coarse evaluations miss.
Problem
Open-ended answers can be valid in multiple ways, while static task-level criteria miss requirements that vary with question context and intent.
Method
Qworld uses a recursive expansion tree to decompose each question into scenarios, perspectives, and fine-grained binary criteria.
Results
Qworld achieves Coverage of 0.89 and Uniqueness of 0.79 on HealthBench, while revealing capability differences and changing model rankings across HealthBench and Humanity’s Last Exam.
Takeaways & Limitations
Question-specific criteria extend expert coverage and surface evaluation dimensions that fixed rubrics and coarse metrics miss.
Takeaways & Limitations
Scoring is performed on concrete per-question binary criteria, while labels such as human-expert dimensions are post-hoc descriptive summaries rather than normative scoring definitions.
Abstract
from arXiv · showhide
Evaluating large language models (LLMs) on open-ended questions is difficult because response quality depends on the question's context. Binary scores and static rubrics fail to capture these context-dependent requirements. Existing methods define criteria at the dataset level or generate them in a single pass, which limits their ability to explore the evaluation space implied by each question. We introduce One-Question-One-World (Qworld), a method that generates question-specific evaluation criteria using a recursive expansion tree. Given a question, Qworld decomposes it into scenarios, perspectives, and fine-grained binary criteria through hierarchical and horizontal expansion. The resulting criteria specify what a high-quality answer must address for that question. On HealthBench, Qworld covers 89% of expert-authored criteria and generates 79% novel criteria validated by human experts. Experts rate Qworld criteria higher in insight and granularity than those produced by prior methods. When applied to 11 frontier LLMs on HealthBench and Humanity's Last Exam, Qworld reveals capability differences in dimensions such as long-term impact, equity, error handling, and interdisciplinary reasoning that coarse rubrics do not capture. By generating evaluation criteria for each question, Qworld enables assessment of LLM responses that is tailored to the question rather than based on fixed task-level criteria.
1 Introduction
Qworld addresses the difficulty of evaluating open-ended answers whose quality depends on each question’s context by generating tailored evaluation criteria. Its criteria expand coverage beyond expert references and expose distinctions that generic rubrics collapse.
- Open-ended questions permit multiple valid responses and require judging qualities tied to context and intent, which accuracy and BLEU do not capture.
- Task-level criteria miss question-specific requirements, while manual curation can omit subtle dimensions and expert-authored question-level criteria are costly to scale.
- Qworld constructs each question’s “world” through recursive expansion into scenarios, perspectives, and fine-grained criteria specifying what a high-quality answer must address.
- 89% Coverage and 79% Uniqueness show that Qworld covers most expert-authored criteria while generating many criteria absent from those references.Human evaluators also rate Qworld criteria higher in Insight and Granularity than prior methods.
- Qworld-generated criteria reveal distinctions such as patient communication versus safety-critical risk management and pedagogical clarity versus mathematical rigor.The criteria integrate with human review, LLM-as-a-Judge, and agentic evaluation frameworks.
2 Related Work
Related work moves from coarse evaluation toward question-level criteria, but existing approaches trade scalability against contextual coverage. Qworld’s recursive expansion tree is presented as an alternative to single-pass or fixed-dimension generation.
- Open-ended evaluation commonly uses LLM-as-a-Judge and specialized judge frameworks because n-gram metrics fail to capture multiple dimensions of quality.
- Expert benchmarks provide high-quality question-level criteria but require annotation for every question, limiting scalability.
- Single-pass checklist methods lower authoring costs but generally generate criteria from one viewpoint or a fixed set of dimensions.
3 Qworld Approach
Qworld models each question with a criteria set whose binary, weighted checks support adaptive evaluation. Its Recursive Expansion Tree alternates horizontal coverage expansion with hierarchical decomposition from scenarios to perspectives to criteria.
- Problem Formulation: For each question Qi, Qworld produces a tailored criteria set Ci, and each criterion specifies a verifiable condition with a scoring function for candidate answer A.
- Output and scoring: An evaluator assigns αc when answer A satisfies criterion c for Qi and 0 otherwise, after which criterion scores are aggregated and normalized.
- Problem Formulation: Scenario grounding infers intent, implicit constraints, audience, stakes, and background knowledge so contexts such as self-management and red-flag recognition remain distinct.
- Problem Formulation: Perspective elicitation identifies question-specific axes such as correctness, completeness, reasoning quality, safety, style, and practicality before criteria are instantiated.Each concrete criterion is binary, carries importance weight αc, and scores either αc or 0.
- Recursive Expansion Tree for Criteria Generation: RET builds a three-level tree whose nodes represent scenarios, perspectives, and criteria, returning consolidated level-3 leaves as Ci.Reviewer agents merge overlapping perspectives and criteria that check the same requirement across branches.
- Recursive Expansion Tree for Criteria Generation: RET alternates repeated horizontal expansion, which adds missing sibling aspects, with hierarchical decomposition, which creates finer-grained child nodes.The process initializes scenarios from Q and applies the two expansion phases at levels 1 and 2.
4 Experiments
Qworld is evaluated for criteria quality on HealthBench and for its ability to distinguish frontier LLM capabilities on HealthBench and HLE. Its question-specific criteria achieve strong coverage and novelty, reveal dimensions that coarse taxonomies collapse, and alter model comparisons.
- Criteria quality: Qworld achieves Coverage of 0.89, exceeding single-turn prompting methods (0.46–0.53), contrastive generation (0.54), and retrieval-based generation (0.83).Adding retrieval increases Coverage to 0.90, maintaining a 0.07 advantage over EvalAgent (0.83).
- Criteria quality: Qworld obtains Uniqueness of 0.79, meaning 79% of generated criteria are absent from expert-authored references.For the hand numbness question, unique criteria include warning about high-risk activities where numbness may impair grip.
- Criteria quality: Qworld achieves an Insight score of 0.83 (0.84 with retrieval), improving over the prior best by 0.40, and reaches Granularity of 0.85.Insight measures non-obvious, context-dependent requirements, while Granularity reflects specificity and actionability.
- Criteria quality: The Value of unique criteria remains above 0.90 as the number of criteria increases from 15 to 60, indicating that later expansion adds useful rather than redundant criteria.The added criteria were missed by the expert list rather than intentionally excluded as irrelevant.
- Benchmarking capabilities: Qworld-defined dimensions separate failure modes such as Clarity, Empathy & Support, and User Empowerment that expert taxonomies subsume under Communication Quality.It also introduces dimensions absent from expert criteria, including Sustainability and Equity.
- Benchmarking capabilities: On HLE, Qworld replaces healthcare-specific dimensions with abstract reasoning dimensions and changes model rankings, with GPT-5 ranking #1 under sQworld versus Gemini 3 Flash under sHLE.Across HealthBench, sQworld is approximately 20% lower than sHEALTHBENCH, reducing score saturation through fine-grained checks.
5 Conclusion
Qworld generates question-specific criteria through recursive expansion, matching expert coverage while surfacing additional requirements that reveal capability gaps coarse metrics miss.
- Qworld decomposes each question into scenarios, perspectives, and fine-grained evaluation criteria through a recursive expansion tree.
- Coverage of 0.89 and Uniqueness of 0.79 show that Qworld matches expert coverage while generating novel criteria on HealthBench.
- Insight of 0.83, 0.40 above the prior best, indicates that human evaluators rated Qworld’s criteria more highly on this dimension.
- Across HealthBench and Humanity’s Last Exam, Qworld reveals gaps such as Sustainability and Equity that coarse metrics miss and changes model rankings.
A.1 Experimental Setup
The experiments evaluate Qworld’s criteria quality and benchmarking utility across HealthBench and Humanity’s Last Exam, using multiple criteria-generation baselines and evaluation measures.
- Experimental setup: The setup uses HealthBench for intrinsic criteria evaluation and Humanity’s Last Exam to test generality on frontier reasoning.
- Datasets: HealthBench contains 5,000 open-ended medical queries with physician-designed criteria, while HLE lacks expert criteria and evaluates generality.
- Evaluation: Criteria quality is measured by Coverage and Uniqueness on 1,000 HealthBench questions, while model benchmarking covers all 5,000 questions.
- Models and judges: The benchmark includes 11 state-of-the-art closed- and open-source models, with GPT-4.1 serving as generator, criteria-quality judge, and response evaluator.
- Baselines: Qworld is compared with TICK, RocketEval, OpenRubrics, and EvalAgent, representing single-turn, reference-guided, preference-guided, and retrieval-augmented generation.
- Findings: Qworld produces substantially lower average scores than prior automated criteria, which tend to under-penalize omissions, while its checkable requirements preserve model differences.
- Metrics: Specificity measures domain-specific vocabulary, while Implicitness measures whether criteria surface properties absent from the instruction.
B.3 Ablation Study on Expansion Steps
Increasing recursive expansion depth improves Qworld’s intrinsic criteria metrics, with gains continuing until saturation near the final expansion stages.
- Coverage, Uniqueness, and Specificity improve substantially as expansion depth K increases, beginning to saturate only at the final stage or stages.
- As expansion proceeds, Qworld pulls further ahead of the EvalAgent baseline, indicating increasingly meaningful contributions from recursive expansion.
B.4 Ablation on the Three-Level Hierarchy
The three-level scenario–perspective–criterion hierarchy improves criteria quality, and matched-budget comparisons indicate that Qworld uses generation tokens more efficiently than repeated-sampling baselines.
- Three-level hierarchy: Removing the scenario level reduces Coverage by 0.025 and Uniqueness by 0.016 on the 100-question HealthBench subset.
- Three-level hierarchy: Removing the perspective level causes a further 0.083 Coverage drop and 0.153 Uniqueness drop, supporting the hierarchy’s value.
- Per-token efficiency: At matched or higher per-question token budgets, Qworld remains more efficient than sequential repeated-sampling CoT and Tree-of-Thought baselines.
- Per-token efficiency: The recursive scenario-perspective-criterion expansion uses the budget more effectively to discover non-overlapping evaluation dimensions rather than simply generating more tokens.
B.6 Cost and Latency Analysis
Qworld separates one-time criteria generation from response judging, allowing question-specific criteria to be reused across models while offering single-call and multi-call cost-quality options.
- The generation cost is paid once per question, and the same criteria can evaluate all model responses without using candidate models during generation.Increasing the number of evaluated models therefore increases response-scoring cost, not criteria-generation cost.
- Qworld-skill preserves the scenario-perspective-criterion decomposition in one call, while removing the multi-step machinery of the full pipeline.The full pipeline uses 56.4 calls, whereas Qworld-skill performs the decomposition in a single forward pass.
- The multi-step pipeline adds 0.05 Coverage and 0.20 Uniqueness over Qworld-skill, reaching 0.92/0.79 versus 0.87/0.59.This comparison isolates the contribution of decomposition from the benefits of multi-step execution.
- In the single-call setting, Qworld-skill improves Coverage from 0.61 to 0.87 and Uniqueness from 0.37 to 0.59 at $0.016 per question.The comparison uses TICK, RocketEval, and OpenRubrics as one-call baselines.
- Judging cost scales linearly with the number of criteria and can be adjusted by applying the top-K criteria ranked by importance weight.The resulting score becomes stricter as more criteria are included.
B.7 Statistical Analysis of Generated Criteria
Qworld analyzes how many criteria it generates and how adding criteria changes evaluation outcomes. More criteria preserve relative model rankings while lowering absolute scores, indicating stricter evaluation constraints rather than noise.
- Figure 11 shows per-question criteria counts and the distribution of criteria points across pipeline stages on HealthBench.The two panels characterize generation volume and how criteria points are distributed.
- As criteria count increases, model rankings remain largely stable while absolute scores consistently decrease.The stable rankings suggest added criteria refine evaluation without arbitrarily reshuffling models, while lower scores indicate stricter standards.
- The score decrease with more criteria indicates that added criteria impose meaningful constraints rather than acting as noise.Figure 12 links the stricter evaluation effect to the consistent score drops across models.
C.1.2 Quantitative Metrics
Qworld evaluates generated criteria through expert-coverage, novelty, specificity, and human-rating metrics. These measures assess whether criteria represent expert requirements, add overlooked requirements, contain domain-specific information, and align with human judgments of quality and importance.
- Coverage: Coverage measures the proportion of expert-curated criteria semantically represented in the generated criterion set.An LLM-based judge returns whether a criterion is covered or entailed by a comparison set.
- Uniqueness: Uniqueness measures the proportion of generated criteria introducing requirements absent from the expert set.Higher uniqueness indicates discovery of valid criteria overlooked by experts.
- Criteria taxonomy: Qworld induces 500+ fine-grained tags that span diverse response-quality aspects and support clustering into higher-level evaluation dimensions.The tags provide the raw material for organizing criteria into broader categories.
- Specificity: Specificity uses normalized inverse word frequency to measure the information density of a criterion’s most specialized terms.Observed specificity values depend on corpus size, so relative comparisons on the same corpus are more informative than absolute values.
- Weight validation: Criteria assigned point values of 10 received Value = 3 ratings in 95% of cases, compared with 76% for criteria assigned 5–6 points.This alignment was measured over 324 Qworld criteria from 32 questions.
C.4 Criterion-Level Discrimination: Concrete Examples
Criterion-level profiles reveal differences between models that aggregate scores can hide. Qworld organizes fine-grained, question-specific requirements into interpretable perspectives for comparing distinct behaviors.
- Qworld’s fine-grained criteria distinguish models across interpretable perspectives instead of reducing each question to one dataset-level score.Criteria can be organized into dimensions such as Safety, User Empowerment, and Actionability.
- Two models can receive essentially tied aggregate scores while passing different sets of question-specific requirements.The HealthBench examples use identical Qworld criteria to expose materially different per-criterion behavior.
- Ear-popping example: GPT-5 is more medically substantive in triage, whereas Claude-Opus-4.5 is more epistemically cautious about limits, test interpretation, and specialist referral.These contrasting profiles concern mechanism and urgency calibration versus disclaimers, refusal, and redirection.
- Diabetes-emergency example: On the diabetes-emergency question, GPT-5 scores 0.444 and Claude-Opus-4.5 scores 0.415, yet their profiles differ in emergency action versus conservative safety communication.GPT-5 emphasizes urgency and concrete steps, while Claude avoids prescribing and considers resource constraints.
- Taxonomy construction: The taxonomy clusters similar tags before expert aggregation, while retaining distinct scored criteria for clinically different questions.For example, chest pain and medication-interaction requirements can share safety labels while remaining question-specific.
D.2 Case Study
Qworld builds question-specific evaluation criteria by expanding a question into scenarios, perspectives, and fine-grained binary criteria. A heat-rash case study illustrates how this process captures medical safety, usability, personalization, and communication requirements.
- Case Study: The heat-rash case identifies 13 scenarios, 26 perspectives, and 36 criteria through successive expansion.The case concerns recurring neck heat rashes in a hot, humid climate and asks for prevention tips and portable relief items.
- Case Study: The generated criteria cover prevention, treatment, warning signs, and escalation to healthcare providers.Examples include cooling and hygiene measures, treatment options, infection warnings, dehydration symptoms, and worsening or persistent rashes.
- Case Study: The criteria also evaluate answer brevity and whether the response avoids unnecessary questions when no additional context is needed.These criteria target communication quality and context awareness alongside medical content.
- Case Study: Qworld expands the case beyond generic advice to include portability, sensitive-skin safety, emergency recognition, plain language, and supportive tone.Additional criteria address public or outdoor use, vulnerable skin, urgent symptoms, health literacy, empathy, and user autonomy.
- Generation Prompt: The generation prompt requires binary, scenario-specific, balanced, self-contained criteria that cover diverse sub-aspects and harmful behaviors.A separate polarity classifier labels criteria as positive or negative without modifying their text.
E Discussion and Future Work
Qworld is presented as a criteria-generation layer that makes per-question evaluation more precise while preserving practitioner-defined categories and standard scoring interfaces. The discussion extends this role to training, debugging, deployment, and future decomposition research.
- Discussion: Practitioners compare model responses against concrete, question-specific criteria rather than relying on a single aggregate score.Criteria can be filtered by a practitioner-defined quality such as Safety.
- Discussion: Qworld supports discovery of new evaluation categories, which experts can equip with normative definitions.Examples include User Empowerment and Actionability.
- Discussion: Question-specific criteria ground categories in fine-grained requirements while preserving an interpretable operational scoring interface.The category meaning remains authored by an expert or chosen by the practitioner, while Qworld supplies concrete criteria and importance weights.
- Future Work: Criterion-level signals can support denser reward supervision, diagnosable evaluation, targeted fine-tuning, and inference-time self-checks or safety gates.Deployment uses include revising missing requirements or routing failures to a fallback model or human review.
- Future Work: Future work may learn decomposition axes and develop a practitioner-facing evaluation layer for selecting criteria by a quality of interest.The paper notes that the recursive expansion tree is not inherently limited to three abstraction levels.