ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists

Jie Ruan; Inderjeet Nair; Shuyang Cao; Amy Liu; Sheza Munir; Micah Pollens-Dempsey; Tiffany Chiang; Lucy Kates; Nicholas David; Sihan Chen; Ruxin Yang; Yuqian Yang; Jasmine Gump; Tessa Bialek; Vivek Sankaran; Margo Schlanger; Lu Wang

Source-linked AI summary

ExpertLongBench: Benchmarking Language Models on Expert-Level Long-Form Generation Tasks with Structured Checklists

Jie Ruan, Inderjeet Nair, Shuyang Cao, Amy Liu, Sheza Munir, Micah Pollens-Dempsey, Tiffany Chiang, Lucy Kates, Nicholas David, Sihan Chen, Ruxin Yang, Yuqian Yang, Jasmine Gump, Tessa Bialek, Vivek Sankaran, Margo Schlanger, Lu Wang

arXiv:2506.01241v3Jun 2, 2025cs.CL

TL;DR

Expert-level benchmarks often miss realistic long-form workflows and task-specific evaluation requirements. ExpertLongBench addresses this gap with 11 rubric-driven tasks across 9 domains and CLEAR, which extracts and compares checklist items from outputs and references; the best evaluated model reaches only 33.4 average F1, while open-weight models can support scalable evaluation.

Problem
Existing expert-level benchmarks underrepresent realistic long-form workflows, task-specific requirements, and grounded evaluation references.
Method
ExpertLongBench combines 11 multidisciplinary expert tasks with expert-designed rubrics, references, and CLEAR’s checklist-based comparison of model outputs against references.
Results
The best model achieves 33.4 average F1 across tasks, while models match over 67% of required aspects but remain far from correct.
Takeaways & Limitations
Open-weight models can support cost-effective, scalable, and reproducible checklist-based benchmarking, with Qwen2.5-72B correlating 0.88 with GPT-4o scores.
Takeaways & Limitations
Fact decomposition methods lack task-specific control over fact granularity, which can produce inconsistent evaluations.

Abstract

from arXiv · show

This paper introduces ExpertLongBench, an expert-level benchmark containing 11 tasks from 9 domains that reflect realistic expert workflows and applications. Beyond question answering, the application-driven tasks in ExpertLongBench demand long-form outputs that can exceed 5,000 tokens and strict adherence to domain-specific requirements. Notably, each task in ExpertLongBench includes a rubric, designed or validated by domain experts, to specify task requirements and guide output evaluation. Furthermore, we propose CLEAR, an evaluation framework that supports accurate evaluation of long-form model outputs in our benchmark. To achieve fine-grained, expert-aligned evaluation, CLEAR derives checklists from both model outputs and references by extracting information corresponding to items in the task-specific rubric. Checklist items of model outputs are then compared with corresponding items of reference outputs to assess their correctness, enabling grounded evaluation. We benchmark 13 popular large language models (LLMs) and analyze components in CLEAR, showing that (1) existing LLMs, with the top performer Gemini-2.5-Pro achieving only a 33.4 F1 score, require significant improvement for expert-level tasks; (2) models can generate content corresponding to the required aspects, but far from correct; and (3) accurate checklist extraction and comparison in CLEAR can be achieved by open-weight models for more scalable, reproducible, and low-cost usage.

1 Introduction

ExpertLongBench addresses the mismatch between realistic expert workflows and short-form benchmark evaluation by combining long-form, domain-specific tasks with expert-written references and CLEAR’s rubric-grounded assessment. Across 13 evaluated LLMs, the strongest model reaches only 33.4 average F1, while models often cover required aspects without stating them correctly.

Motivation: Existing benchmarks emphasize multiple-choice, short-form, or question-answering settings rather than complete expert workflows and task-specific requirements.Long-form expert benchmarks also suffer from limited references, which weakens grounding and prevents recall estimation.
ExpertLongBench: 11 tasks across 9 domains use realistic expert workflows, long-form outputs, expert-written references, and inputs or outputs exceeding 200K or 5K tokens.Examples include drafting legal briefs and clinical notes; a proficient legal practitioner may need over 10 hours for complex case summarization.
CLEAR: CLEAR uses expert-designed rubrics to extract checklist items from model outputs and references, then compares corresponding information for grounded, fine-grained evaluation.The rubric specifies application requirements such as identifying a legal case’s cause of action, legal basis, and requested remedy.
Benchmark findings: 33.4 average F1 is the best score across 13 evaluated frontier LLMs, showing that ExpertLongBench’s end-to-end expert tasks remain difficult.The benchmark includes both open-weight and proprietary model families.
Benchmark findings: Models match over 67% of required checklist aspects despite low overall quality, indicating that apparently expert-aligned content can still be incorrect and misleading.The paper also reports that Qwen2.5-72B and GPT-4o produce checklist-based evaluation scores with Pearson correlation 0.88.

2 Related Work

Prior benchmarks largely evaluate expert knowledge through exam-style or limited question-answering tasks, while existing long-form and checklist methods do not fully capture domain-specific workflow requirements. ExpertLongBench combines diverse workflow tasks with rubric-based, reference-grounded checklist evaluation.

Evaluating expert-level knowledge: Earlier benchmarks mainly use multiple-choice, short-answer, or brief open-ended questions, limiting their representation of realistic domain-specific work.ExpertQA answers average about 100 words and target information seeking rather than end-to-end workflows.
Evaluating expert-level knowledge: ExpertLongBench extends prior work with diverse long-form tasks aligned to complete expert workflows and practical lengthy inputs.It addresses the limited scope of ExpertQA, DOLOMITES, and ResearchQA, which focus on shorter answers or narrower writing and research problems.
Evaluating long-form generations: Fact decomposition can produce inconsistent evaluations because it lacks task-specific control over fact granularity.This limitation is distinct from the benchmark’s broader problem of insufficient workflow coverage.
Evaluating long-form generations: Existing checklist and LLM-judge approaches are often insufficiently domain-specific or ungrounded in references, hindering accurate assessment of coverage and relevance.CLEAR derives checklists from expert-designed rubrics and compares extracted model and reference facts at the item level.

3 Benchmark Construction with Multi-disciplinary Expert Tasks

ExpertLongBench is constructed as a multidisciplinary benchmark of real-world expert tasks with long inputs, human references, expert-guided rubrics, and checklist-mapped references. Its design emphasizes objective evaluation criteria, representative difficulty, and reference quality.

Benchmark overview: 11 tasks cover real-world expert workflows, including six newly collected tasks, with selection requiring definable rubrics, domain expertise, and practical workflow grounding.Each sample includes task input, a human-written reference, and a checklist-mapped reference.
Benchmark overview: 1,050 samples have average input and human-reference lengths of 36,204 and 851 tokens, respectively.The benchmark includes public and private subsets to support experimentation while reducing contamination and overfitting risks.
Task sources and sample selection: Data combines 6 newly curated tasks with 5 adapted existing tasks, selected with domain researchers and practitioners and paired with checklist-based rubrics and mapped references.The sample pool is refined around diversity and task-specific difficulty factors, with 100 representative samples selected for tasks initially exceeding 100 examples.
Expert-guided Rubric Design: Rubrics combine expert-guided criteria with refinements of established protocols to encode professional standards and practical requirements at checklist-level granularity.For T1LegalMDS, rubric creation required over 10 hours of expert effort; the rubrics are intended to exceed what existing LLMs can automatically replicate.
Checklist-mapped Reference Creation: Checklist-mapped references extract content for each rubric item, with human and LLM evaluation finding over 90% faithfulness and coverage on two tasks.GPT-4o performs the extraction for most tasks, while exceptions use data-collection-time construction or already structured references.

4 Checklist-based Performance Assessment using CLEAR

CLEAR maps model outputs and references to expert-defined checklist items, scores their semantic alignment, and aggregates item-level results into task performance. Validation supports Qwen2.5-72B for cost-effective mapping and GPT-4o judging with high agreement from an alternative judge.

4 Checklist-based Performance Assessment using CLEAR: CLEAR evaluates outputs by comparing checklist-mapped model responses with checklist-mapped references, using expert rubrics to support grounded assessment.The framework explicitly validates design choices for both checklist mapping and checklist comparison.
4.1 Evaluation Process: A checklist item receives score 1 only when all reference information is semantically contained in the model response; precision, recall, and accuracy are then aggregated from item-level results.Task-level performance is obtained by averaging sample-level metrics.
4.2 Evaluation Component Validation: Qwen2.5-72B is selected as the open-weight checklist mapper after achieving average F1 of 90.1 across tasks T1, T6, T7, and T8.The choice prioritizes availability, extraction performance, cost effectiveness, and reproducibility over using GPT-4o for every model output.
4.2 Evaluation Component Validation: Cohen’s Kappa between GPT-4o and Gemini-2.0-Flash annotations is 0.81, 0.87, 0.89, and 0.85 for T1, T6, T7, and T8, respectively.These values are reported as near-perfect agreement and support using GPT-4o exclusively for final checklist evaluations.

5 Experiments

Experiments show that current LLMs struggle with expert-level long-form tasks, and apparent checklist coverage often fails to translate into correct outputs.

5 Experiments: Larger models generally outperform smaller family counterparts on average, but no single model wins consistently across individual tasks.Mistral-Large-Instruct ranked second on T4 while Mistral-Nemo-Instruct ranked eleventh; on T10, the smaller model ranked first.
5 Experiments: Proprietary models are not always superior, while Qwen models suffer on T10 because their 32K context length is shorter than the task’s average 64K input.Qwen2.5-72B trailed the leading T10 model by 17.5.
5 Experiments: Checklist coverage has an overall negative correlation with F1, indicating that models can appear complete while producing incorrect content.The analysis aggregates results across models and tasks and highlights a risk of misleading users through superficially expert-aligned outputs.

6 Towards Reproducible and Low-Cost Evaluation

The paper studies lower-cost, reproducible alternatives to GPT-4o for checklist evaluation by comparing model judges and their combinations across tasks.

6 Towards Reproducible and Low-Cost Evaluation: Qwen2.5-72B achieved the highest average correlation among single judges with GPT-4o, supporting lower-overhead checklist assessment.The paper identifies closed-model costs, privacy risks, and version changes as motivations for alternatives.
6 Towards Reproducible and Low-Cost Evaluation: Majority pooling improved average correlation from 0.65, 0.80, and 0.78 for three individual judges to 0.82.The three individual judges were Llama3.1-8B-Instruct, Mistral-Nemo-Instruct, and Qwen2.5-7B; majority pooling usually outperformed mean pooling except for large open-source combinations.
6 Towards Reproducible and Low-Cost Evaluation: Task complexity strongly correlated with judgment agreement with GPT-4o, enabling model-size choices based on task difficulty.Smaller models aligned well on less complex tasks, whereas Qwen2.5-72B was recommended for challenging T1 and T2; majority pooling suited binary checklist scores.

7 Skill Decomposition Analysis

Skill decomposition reveals that expert-level knowledge and increasingly difficult reasoning remain major barriers, while diverse difficulty levels expose model-specific strengths and weaknesses.

7 Skill Decomposition Analysis: Performance consistently declined at Graduate-level knowledge and with higher reasoning difficulty, highlighting barriers beyond knowledge memorization.The analysis distinguishes memorization, understanding, applying, and creating across multiple knowledge levels.
7 Skill Decomposition Analysis: Some models performed worse on low reasoning difficulty than medium difficulty, likely because long-tail expert knowledge is rarely memorized despite supporting reasoning-based approximation.This pattern is attributed in the passage to limited exposure during training and retained reasoning or generalization ability.
7 Skill Decomposition Analysis: Diverse difficulty levels reveal divergent model strengths: Claude-3.7-Sonnet ranked first for memorization but relatively low for knowledge creation.Such evaluations expose differences that general benchmarks may overlook.

8 Conclusion

The paper introduces a multi-domain benchmark and checklist-based evaluation framework for realistic expert workflows, then shows that current LLMs remain inaccurate and that open-weight components can support scalable evaluation.

8 Conclusion: EXPERTLONGBENCH contains 1,050 samples across 11 tasks and 9 domains, while CLEAR compares rubric-derived checklist items from model outputs and references.The benchmark targets realistic expert applications requiring long-form outputs and domain-specific standards.
8 Conclusion: Experiments show that current LLMs often match required aspects superficially but lack accuracy, whereas open-weight models can replace proprietary checklist-mapping and evaluation components.The paper presents this substitution as facilitating low-cost, reproducible, and scalable benchmarking.

Outline

The appendix is organized into sections covering benchmark construction, task details, evaluation, experiments, analyses, implementation, comparisons, limitations, impacts, and author contributions.

Appendix A–B cover benchmark overview and detailed task definitions, including sample elements, preprocessing, rubrics, and checklist-mapped references.
Appendix C–F document checklist-accuracy prompts, supplementary experiments, CLEAR analysis, and skill decomposition details.
Appendix G–H provide model specifications, inference and cost details, and comparisons with existing evaluation benchmarks.
Appendix I–J address performance with ground-truth rubrics, limitations, broader impacts, licensing, new assets, and author contributions.

A Benchmark Description

This section describes ExpertLongBench’s expert-level, long-form tasks across legal, materials-science, and pedagogical domains, including their data construction, domain-informed rubrics, and checklist-mapped references. It also details the processing and evaluation procedures for assessing specialized generation and essay grading against explicit and implicit requirements.

A Benchmark Description: ExpertLongBench requires long-form, objective tasks needing domain expertise, reliable human references, real-world impact, and realistic user-centered problem solving.Adapted datasets must also be publicly accessible and preferably recent to reduce contamination risk.
B.1.1 Task Definition: Legal case summarization condenses multiple case documents into accessible summaries of essential facts, legal principles, procedural developments, and outcomes.The task supports access to justice, legal research, decision-making, and professional efficiency [Shen et al., 2022].
B.1.3 Data Acquisition and Preprocessing: Our legal dataset contains 1,393 Clearinghouse cases, from which diversity- and difficulty-based filtering produced 454 difficult samples and 100 high-quality, long-reference samples.Selection considered topic diversity, document length, appeals, complaints, and dockets; GPT-4o judged checklist coverage of human summaries.
B.1.5 Evaluation Rubric: The legal summarization rubric was developed with legal researchers over roughly 11 hours and contains 26 checklist items, including conditional requirements.Items cover parties, causes and bases of action, remedies, counsel, filings, opinions, judges, decrees, monitor reports, rulings, and factual basis.
B.1.6 Checklist-mapped Reference: Checklist-mapped references use GPT-4o for individually extracted human-reference items and Qwen2.5-72B for grouped model-output items, with role prompts and N/A outputs to limit hallucination.The 26 items are grouped into five model-output prompts containing four to six items each; 30 additional difficult samples received human verification.
B.2.1 Task Definition: Statement of Fact generation transforms multi-proceeding courtroom transcripts into a standalone, comprehensive, objectively grounded legal account with attributed quotations and procedural details [LSD Law, n.d.].The task targets child-welfare and parental-rights cases, where manual drafting can require 20–40 hours or more [TypeLaw, 2021].
B.2.3 Data Acquisition and Preprocessing: The Statement of Fact dataset comprises 113 expert-provided transcript samples processed into clean structured text across Word and PDF formats, using direct extraction and OCR for scans.OCRmyPDF handled deskewing, noise cleaning, and compression when PDFs lacked embedded text.
B.2.3 Data Acquisition and Preprocessing: The legal SOF dataset retains page numbers, concatenates cleaned transcripts per case, and selects 100 information-dense samples averaging 14 documents and 5K+ tokens each.The selected cases pair transcripts exceeding 5K tokens with human reference SOFs also exceeding 5K tokens.
B.2.5 Evaluation Rubric: The legal rubric contains 41 domain-expert-designed checklist items spanning case background, child and parent circumstances, court events, services, permanency, termination, and final rulings.Three experts developed the checklist through three discussion-based iterations, with conditional items such as alternative adjudication outcomes.
B.2.6 Checklist-mapped Reference: The legal checklist-mapped reference groups the 41 items into nine topical and length-based sections so models can process large inputs while focusing on fewer sections at a time.The section gives the extraction prompt in Table 9 but withholds concrete examples because the data are proprietary.
B.3.1 Task Definition; B.3.5 Evaluation Rubric: Materials-science synthesis explanation generation asks models to justify precursor and processing choices using structural, reactivity, thermodynamic, kinetic, atmospheric, and duration considerations.The six-item rubric covers precursor structure, reactivity, physical and chemical properties, temperature and heating, atmosphere, and duration.
B.3.2 Task Significance; B.3.4 Illustrative Example: The synthesis task addresses a time-consuming expert workflow: writing explanations from scratch takes a materials-science PhD student 1–2 hours, while existing recipe datasets generally lack such explanations.The sample format pairs a complex-oxide thin-film recipe with explanations of why major synthesis steps and conditions were selected.
B.3.3 Data Acquisition and Preprocessing: The materials-science dataset semi-automatically extracts 50 recipe-explanation pairs from solid-state synthesis papers using Llama-3.3-70B-Instruct, GPT-4o, and undergraduate chemistry annotators.Five GPT-4o outputs per paper improve recall, source sentences support faithfulness, and annotators discard papers lacking explanations.
B.4.1 Task Definition: The pedagogical-feedback task evaluates multi-turn biology tutoring dialogues in which models assess student accuracy, detect repeated difficulty, provide adaptive feedback, and transition between sub-problems.Single errors require indirect guidance without answers, whereas repeated mistakes permit explicit error identification and corrective guidance.
B.4.1 Task Definition; B.4.2 Task Significance; B.4.3 Data Acquisition and Preprocessing: The pedagogical task targets accurate, context-sensitive feedback that follows an explicit plan, corrects misconceptions, and promotes active learning rather than immediate answer disclosure.Its data come from GPT-4o-simulated, human-validated biology dialogues grounded in socio-constructivist tutoring, with evaluation criteria covering accuracy, relevance, completeness, and motivation.
B.4.3 Data Acquisition and Preprocessing: T4EduPAE extracts student-response, subproblem, feedback-error, correction, and repeated-mistake information from tutor dialogues to evaluate contextually appropriate feedback.GPT-4o performs automated extraction followed by human verification; repeated mistakes are inferred when consecutive responses address the same subproblem and are both incorrect or partially correct.
B.4.3 Data Acquisition and Preprocessing: The sampled T4EduPAE set contains 109 distinct-problem conversations with approximately balanced lengths from 5 to 13 turns and coverage of five final-outcome categories.The categories distinguish correct, incorrect, and partially correct endings, with or without reattempts where applicable.
B.4.4 Illustrative Example: An illustrative T4EduPAE dialogue requires the tutor to assess a partially correct mammal-survival answer and guide deeper reasoning without simply revealing the answer.The human reference balances affirmation of partial correctness with guidance toward improved reasoning.
B.4.5 Evaluation Rubric: The T4EduPAE rubric evaluates response classification, repeated-mistake detection, error identification and rectification, direct-versus-indirect feedback types, and subquestion transitions.Correct responses require non-N/A error fields, while direct identification and rectification are expected after multiple inaccurate or partially accurate attempts.
B.4.6 Checklist-mapped Reference: Checklist-mapped references derive error and correction elements from tutor data, assign direct feedback types after repeated mistakes, and use dataset transition values only when correct answers leave the problem unsolved.The mapped reference provides rubric-comparable targets for evaluating generated outputs.
B.5.1 Task Definition: The essay-evaluation task tests whether LLMs can match trained-instructor judgment when grading assignments against both public requirements and unstated private expectations.The task reflects instructors’ holistic, context-sensitive use of private criteria beyond general student-facing rubrics [Bloxham et al., 2011, Jeong, 2015, Broad, 2003].

B.5.3 Data Acquisition

This section details data acquisition and checklist construction across expert-oriented tasks spanning economics, clinical, molecular, protein, ESG, and security-agent applications. It describes datasets, prompts, rubrics, checklist-mapped references, and evaluation procedures designed to support challenging, domain-specific outputs and task-specific constraints.

B.5.3 Data Acquisition: The economics task uses 100 difficult student essays with instructor feedback mapped to 20 private rubric items covering missing concepts and economic errors.Samples were selected for longest feedback because it typically reflects more errors requiring careful analysis.
B.5.5 Evaluation Rubric: The economics rubric evaluates whether feedback detects missing concepts, incorrect economic claims, and failures to address inter-market effects, with binary Yes/No checklist alignment.Affirmative responses indicate that the corresponding rubric problem is present, and model responses are compared item by item with the reference.
B.6.1 Task Definition: Clinical note generation converts patient-doctor conversations into structured SOAP notes covering Subjective, Objective, Assessment, and Plan information.The task aims to produce accurate, comprehensive documentation from unstructured clinical dialogue.
B.6.3 Data Acquisition: The clinical dataset contains 207 ACI-Bench encounters, from which 100 samples were selected for difficulty and diversity based partly on human-reference length.Transcripts were produced using virtual-assistant, virtual-scribe, and ambient-clinical-intelligence methods.
B.6.5 Evaluation Rubric: The clinical rubric contains 29 checklist items spanning SOAP content, including symptoms and history, social context, review of systems, vital signs, examination, diagnosis, and treatment planning.The rubric was independently developed through research and verified against reputable sources [Contributors, 2023, Podder et al., 2023].
B.6.6 Checklist-mapped Reference: Clinical checklist references group the 29 items into five SOAP- and response-length-based groups to make extraction more manageable, with human verification on 30 additional difficult and diverse samples.The grouped structure focuses extraction on distinct clinical sections and supports assessment of reference faithfulness.
B.7.1 Task Definition: Molecule description generation translates SMILES strings into accurate, structured natural-language descriptions of molecular structures and chemical features.The task addresses the difficulty of interpreting complex symbolic molecular representations and supports drug discovery, materials science, and chemical education.
B.7.3 Data Acquisition and Preprocessing: The molecular-description task uses 33,010 ChEBI-20 SMILES–description pairs and evaluates 100 samples whose human references exceed 500 characters.Descriptions shorter than 20 words were excluded, and the subset targets more complex molecule descriptions using reference length as a difficulty proxy.
B.7.4 Illustrative Example: The molecular task prompts models to produce precise structure and pharmacological descriptions, while a chemistry-expert rubric defines six checklist items and supports checklist-mapped references.The rubric covers structure, biological function and applications, chemical classifications, conjugate bases, conjugate acids, and origins; extraction uses one grouped prompt because only six items are required.
B.8.1 Task Definition: The protein-captioning task generates descriptions from amino-acid sequences using 100 SciKnowEval samples longer than 900 characters and expects biologically specific, scientifically grounded outputs.References are expert-written and curated by UniProt, and role-based prompting instructs models to act as protein researchers using domain-specific terminology.
B.8.5 Evaluation Rubric: Two biology graduate students refined a five-item protein-description rubric covering domains or motifs, functional role, cellular localization, Gene Ontology, and interactions, with one extraction prompt producing checklist-mapped references.The rubric was iteratively agreed upon across three meetings, and the checklist-mapping procedure follows the same human-and-model extraction approach used in T1.
B.9.1 Task Definition: The clinical diagnosis task infers a primary discharge diagnosis and supporting reasoning from diagnosis-masked SOAP-note sections, using 100 difficult, diverse cases sampled from DiReCT’s 25 disease categories.Nine licensed physicians annotated the source data and three senior medical experts verified the annotations; the final sample covers 18 diagnoses, and models select from predefined diagnoses to enable reference matching.
B.9.5 Evaluation Rubric: T9 uses global and conditional instance-level checklist items, evaluating diagnosis for every case but evidence and reasoning only when corresponding content appears in the human reference.Its checklist-mapped reference is generated directly from the model output, so no separate reference-generation prompt is needed.
B.10.1 Task Definition: The ESG task summarizes paired MSCI and company-published reports using 100 company samples drawn from 20 sectors, with sector-specific rubrics based on MSCI’s 33 key issues, 10 themes, and underlying indicators.Reports are converted from PDF to text, relevant ESG pages or sections are retained, and reference summaries come from MSCI’s web interface; checklist mapping presents all sector-specific rubric items together.
B.11.1 Task Description: The task requires binary safety labels and detailed MTO explanations identifying motivations, triggers, outcomes, and any trigger or attack tools.The label is 0 for safe traces and 1 for dangerous traces.
B.11.3 Data Acquisition and Preprocessing: The dataset contains 100 samples: 19 from AgentHarm and 81 from R-Judge, selected for scenario diversity and challenging, extended interactions.R-Judge samples were filtered for risk descriptions exceeding 600 characters, while AgentHarm contributed behaviors including refusal and partial or failed compliance.
B.11.3 Data Acquisition and Preprocessing: R-Judge annotations follow the MTO schema and were produced by six domain-informed experts, while missing trigger and attack-tool labels were generated with GPT-4o and manually verified.AgentHarm trajectories supplied safety labels and invoked tools, which GPT-4o used to generate MTO responses that were also manually verified.
B.11.4 Illustrative Example: The illustrative example shows a malicious instruction injected through GitHubGetRepositoryDetails but not acted upon, with the reference recording the failed attack, binary label, and involved tool.The example demonstrates how execution traces, tool usage, environment responses, and risk descriptions are aligned.
B.11.5 Evaluation Rubric: The rubric extends R-Judge with Trigger Tools and Attack Tools checklist items to distinguish tools that advance an attack from those that deliver or embed malicious prompts.These distinctions support tracing attack origins, identifying exploitable tool-design weaknesses, and capturing intermediate steps toward harmful outcomes.
B.11.6 Checklist-mapped Reference: Checklist-mapped references are created for Motivation, Trigger, and Outcome, while binary labels are directly verified and tool fields are extracted with regex from structured outputs.The extraction process follows the checklist-mapped-reference procedure described in Section B.1.6.
C.1 Checklist Accuracy Assessment with Unstructured Reference: Prompt: Checklist evaluation measures faithfulness and coverage, using bidirectional semantic-containment checks to determine whether each response item correctly matches its reference item.Reversing model-response and reference roles checks containment in both directions.
D.1 Additional Main Results: Additional analyses report checklist accuracy, precision, recall, and coverage, plus task-wise F1 and accuracy plots with standard errors estimated from 10,000 bootstrap samples.The bootstrap procedure uses non-parametric resampling with replacement to estimate variability.

D.3 Impact of Input and Output Length on Model Performance

Length-based analyses show that input and output length do not uniformly determine performance, while checklist extraction and judge-selection studies support CLEAR’s evaluation design. The appendix validates GPT-4o-mapped checklists and selects Qwen2.5-72B as the final open-weight mapper.

D.3 Impact of Input and Output Length on Model Performance: Length analyses group samples into three contextually balanced input or output categories and retain only tasks with normalized IQR greater than 0.2.Input analysis includes T1, T2, T3, T4, T6, T7, T8, T9, T10, and T11; output analysis includes T1, T2, T3, T4, T8, and T9.
D.3 Impact of Input and Output Length on Model Performance: Performance declines with increasing input length only for certain tasks, while shorter inputs can reduce performance in T11 by limiting available information.The results suggest that datapoint complexity depends on factors beyond input size alone.
D.3 Impact of Input and Output Length on Model Performance: Longer human-reference outputs reduce performance in T1 and T8, but output length is not a consistent difficulty driver across tasks.Domain knowledge, reasoning depth, and task-specific requirements can matter more than sequence length.
E.1 Validation of Checklist Mapping by GPT-4o: Additional Details: GPT-4o checklist mapping achieves 99.99% faithfulness for T1 and 95.12% for T6 under human validation.Faithfulness measures whether extracted content appears in the source without unsupported additions and matches the requested information.
E.1 Validation of Checklist Mapping by GPT-4o: Additional Details: Automated validation reports T1/T6 coverage of 90.3%/94.3% and faithfulness of 94.8%/97.9%, supporting reliable checklist extraction.Claude-3.7-Sonnet and Gemini-2.0-Flash serve as independent evaluators rather than GPT-family judges.
E.2 Model Selection for Checklist Mapper: Additional Details: Qwen2.5-72B achieves competitive or best checklist-mapping performance and substantially exceeds its smaller variants, motivating its use in final assessments.The evaluation compares Qwen2.5-7B, Qwen2.5-14B, and Qwen2.5-32B alongside Llama-3.3-70B-Instruct and Mistral-Large-Instruct.
E.3 Correlation of the scores assigned by different models and its combinations with GPT-4o: Full table: The appendix evaluates correlations between model or model-combination scores and GPT-4o judgments using task-wise results reported in Table 51.The table distinguishes mean and majority pooling for combinations.

E.4 Linking Task Complexity with the Judgement Correlation

The paper relates task complexity to judge agreement and decomposes benchmark difficulty into skills and knowledge or reasoning levels. It defines standardized annotations for these dimensions and analyzes them with GPT-4o-based assessment.

E.4 Linking Task Complexity with the Judgement Correlation: Tasks with higher average model performance tend to show stronger judgment correlation with GPT-4o, while low-performing tasks T1 and T2 have the weakest average correlation.The regression’s moderately high R2 explains substantial variance, and its high intercept indicates that checklist-mapped response evaluation can remain easier than generating checklist-specific content.
F. Skill Decomposition Analysis Details: The decomposition defines eight skills, including contextual understanding, decision making, workflow expertise, information aggregation, information condensing, information grounding, problem identification, and problem solving.These skills characterize capabilities needed to complete checklist items.
F.1 Skill Definitions: Skill proficiency uses four levels—N/A, Basic, Intermediate, and Advanced—while knowledge difficulty distinguishes College- and Graduate-level items.Reasoning difficulty separately spans Low, Medium, High, and Very High levels corresponding to memorization, understanding, applying, and creating knowledge.
F.1 Skill Definitions: The analysis also operationalizes information aggregation, condensing, grounding, problem identification, and problem solving using task-relevant definitions adapted from prior work.The full skill definitions and examples are provided through the paper’s accompanying materials.
F.2 Determining Levels: GPT-4o assigns each checklist item skill and difficulty levels using the rubric item, an example, and standardized level descriptions.Skill levels indicate how strongly an item exercises a skill, whereas difficulty levels are inferred from descriptive guidance.

F.3 Results and Analysis

Skill- and difficulty-level analyses reveal uneven capabilities across expert tasks. Models perform relatively well on decision making and problem solving but struggle more with information-centered skills and graduate-level knowledge.

F.3 Results and Analysis: Models perform comparatively well on Decision Making and Problem Solving but more weakly on Information Aggregation, Information Grounding, and Problem Identification.These comparisons are reported using checklist F1 scores across eight general skills.
F.3 Results and Analysis: Performance consistently drops from College to Graduate knowledge levels, indicating difficulty with deeper or more specialized domain expertise.The analysis reports this pattern in Table 54 and Figure 11, including reasoning-difficulty categories.

G Experiment Details

The experiments document model specifications, inference settings, and proprietary-model costs. Open-weight models use reproducible greedy-decoding inference, while inputs exceeding context limits are truncated.

G.1 Model Information: The model inventory records parameter counts, context lengths, and pre-training knowledge cutoff dates from official documentation.These details are summarized in Table 55.
G.2 Inference Implementation: Open-weight models are run with vLLM, greedy decoding at temperature 0, default maximum context lengths, and truncation for inputs exceeding those limits.The procedure applies to model outputs obtained for the experiments.
G.3 Cost Report for Proprietary Models: The total reported cost for proprietary-model evaluation is $1108.05 after using batch prediction APIs that provide a 50% discount.Costs are reported in Tables 56–58.

H Comparison with Existing Benchmarks

EXPERTLONGBENCH extends expert-task evaluation beyond short-form and QA benchmarks by supporting specialized, long-form generation with task-specific requirements. In a T2LegalSFG study, exposing models to the ground-truth rubric substantially improved performance but left results low.

H Comparison with Existing Benchmarks: 15,801 maximum output tokens and 1,998,517 maximum input tokens distinguish EXPERTLONGBENCH from prior benchmarks by supporting extended expert-domain content generation.Its maximum output length is based on human-written references, estimating the tokens reasonably required for a task.
H Comparison with Existing Benchmarks: EXPERTLONGBENCH targets specialized, end-to-end expert workflows, whereas earlier benchmarks emphasize multiple-choice, short-answer, QA, or general open-ended tasks.The comparison includes MMLU, AGIEval, GPQA, ExpertQA, WildBench, and BIGGEN BENCH.
I.1 Performance with Ground-truth Rubric: GPT-4o’s accuracy rose from 4.3 to 29.7 and F1 from 6.2 to 32.5 when given the detailed rubric on T2LegalSFG.Table 61 reports performance for generic and detailed prompts scaled to 0–100.
I.1 Performance with Ground-truth Rubric: Providing models with the detailed evaluation rubric improved one reported score from 7.9 to 28.6, while both models remained below 33 F1 and 30 accuracy.The rubric significantly enhanced alignment with task-specific requirements and more accurate outputs, although performance remained relatively low.

I.2 Limitations of LLMs in Generating High-Quality Evaluation Rubric

The appendix finds that LLM-generated evaluation checklists remain unreliable for expert-level assessment and documents broader benchmark, language, methodological, and coverage limitations. It also records the benchmark’s broader-impact rationale, licensing arrangements, and author responsibilities.

I.2 Limitations of LLMs in Generating High-Quality Evaluation Rubric: 26, 29, and 6 expert checklist items contrasted with 20, 22, and 7 GPT-4o items across T1LegalMDS, T6HealthCNG, and T7ChemMDG.The appendix manually evaluates the model-generated items and summarizes their problems in Table 64.
I.2 Limitations of LLMs in Generating High-Quality Evaluation Rubric: GPT-4o-generated checklists omitted expert criteria, introduced subjective or unsuitable items, lacked granularity, and misunderstood domain-specific section coverage.Examples include missing conjugate acid/base criteria, redundant peer-review checks, coarse assessment accuracy, and misplaced chief complaints.
I.2 Limitations of LLMs in Generating High-Quality Evaluation Rubric: The appendix concludes that rigorous and complete evaluation rubrics require human oversight and domain expertise rather than relying solely on LLM generation.The stated concern is that model-generated criteria can be subjective, incomplete, unsuitable, overly broad, or contextually misplaced.
I.3 Limitation: The benchmark is limited to English, off-the-shelf model evaluation, no concrete model-improvement strategies, and 11 tasks covering only a small fraction of real-world expert applications.The authors also note that LLM-based evaluators may be erroneous or inconsistent in complex or ambiguous cases.
I.4 Broader Impacts: The work frames expert-domain benchmarking as a response to risks from inaccurate or hallucinated outputs in law, medicine, and education.EXPERTLONGBENCH and CLEAR are intended to support grounded, transparent, fine-grained assessments aligned with expert judgments.
I.5 Licenses: Dataset licensing is summarized in Table 65, with seven tasks released under CC BY-NC-SA 4.0 and four tasks remaining private.The released tasks are T1, T3, T4, T6, T7, T8, and T11; private tasks are T2, T5, T9, and T10.
J Author Contribution: The author-contribution record assigns benchmark design, rubric formulation, evaluation-framework development, experiments, dataset release, and task-specific data or rubric work across the research team.It identifies lead authors and contributors for individual tasks, with all five core authors contributing to writing.