Source-linked AI summary
Overview of the NTCIR-19 Automatic Evaluation of LLMs 2 (AEOLLM-2) Task
Junjie Chen, Yuxi Dong, Haitao Li, Yiqun Liu, Qingyao Ai
TL;DR
Long-form LLM evaluation remains underexplored, especially for deep research reports whose evaluation reliability has not been systematically validated. AEOLLM-2 introduces a subtask that compares automatic evaluation scores with human annotations. RSLIR performs best overall and on three report dimensions, while SCUNLP leads on readability.
Problem
Automatic evaluation reliability remains insufficiently validated for long-form deep research reports, which are underrepresented in popular LLM-as-a-Judge benchmarks.
Source: §1 · pp. 1–2Method
Deep Research Evaluation compares participants’ automatic scores for LLM-generated reports with human-annotated labels.
Source: §1 · p. 2Results
Takeaways & Limitations
The results highlight the need for fine-grained evaluation across multiple dimensions when assessing long-form responses.
Source: §5 · p. 5
Abstract
from arXiv · showhide
In this paper, we provide an overview of the NTCIR-19 Automatic Evaluation of LLMs 2 (AEOLLM-2) task. Building on the success of the NTCIR-18 core task AEOLLM, we proposed AEOLLM-2 for NTCIR-19 to further investigate automatic evaluation methods for Large Language Models (LLMs), particularly in long-form text generation scenarios. In AEOLLM-2, we introduced a new subtask, Deep Research Evaluation, which focuses on the automatic evaluation of long-form deep research reports generated by LLMs. Participants developed evaluation methods to automatically assess the quality of these reports, and the performance of each method was measured by comparing its scores against human-annotated ground-truth labels. This year, we received 91 runs from 10 teams in total. This paper describes the background of the task, the dataset construction, the evaluation measures, the participants' methods, and the final evaluation results.
1 Introduction
AEOLLM-2 addresses gaps in automatic LLM evaluation by adding Deep Research Evaluation for long-form reports, a setting that remains underexplored. The task provides a practical dataset and evaluation pipeline and drew 91 runs from 10 teams.
1 Introduction
- Automatic evaluation can reduce extensive human involvement, costs, and improve objectivity, motivating research into efficient, general LLM evaluation methods.The paper identifies this as an increasingly significant research direction.Source: §1 · p. 1
- Existing benchmarks often use multiple-choice questions and reference-based metrics, while prior LLM-as-a-Judge methods remain limited on long-form generation.Multiple-choice formats differ from open-ended practical questions, and reference-based metrics may not accurately reflect output quality.Source: §1 · pp. 1–2
- Long-form deep research reports often exceed 1,500 tokens, while popular LLM-as-a-Judge benchmarks typically focus on texts under 500 tokens.The reliability of automatic evaluation methods for these reports has not been systematically validated.Source: §1 · p. 2
- Deep Research Evaluation assesses automatic methods by comparing their scores for LLM-generated reports against human-annotated labels.The subtask focuses on the reliability and effectiveness of evaluation in long-form text generation.Source: §1 · p. 2
- AEOLLM-2 received 91 runs from 10 teams and provides a practical dataset plus an automated pipeline for testing evaluation methods.The paper lists both dataset provision and an evaluation pipeline among its contributions.Source: §1 · p. 2
2 Task Framework
AEOLLM-2 uses DeepResearchBench reports as its data source, generates answers with popular LLMs, and manually annotates answer sets as gold standards. Participant evaluation results are compared with those annotations using accuracy and rank-correlation metrics.
2 Task Framework
- The task uses DeepResearchBench as its data source, generates answers with popular LLMs, and manually annotates answer sets as gold standards.The gold standards are used to evaluate the performance of participants’ methods.Source: §2 · p. 2
- Participant evaluation results are checked against manual annotations using accuracy, Kendall’s tau, and Spearman correlation.The framework calculates consistency with the manually annotated results.Source: §2 · pp. 2–3
3.1 Datasets
The dataset combines varied questions and prompts with long reports from multiple systems, and its annotations pair human-in-the-loop judgments with fully human readability scores. Data was divided into training, dry-run, and formal-run sets.
3.1.1 Question and Prompt Construction.
- 40 questions—29 Chinese and 11 English—each received four prompts designed to elicit reports with varied quality and analytical characteristics.Prompts varied in requested detail, comprehensiveness, balance, and neutrality to test whether evaluators distinguish quality beyond surface fluency.Source: §3.1.1 · p. 3
3.1.2 Answer Generation.
- Four systems generated 640 reports, with a median length of about 9,610 tokens; reports were evaluated for comprehensiveness, insight, instruction following, and readability.Each question yielded 16 responses across four prompt variants and four systems.Source: §3.1.2 · p. 3, §3.1.3 · p. 3
3.1.3 Human Annotation.
- RAG evidence and preliminary LLM scores support human-reviewed judgments for three dimensions, while readability is scored entirely by two human annotators.Substantial annotator disagreements trigger third-person adjudication; the average-measure ICC was 0.76, and human-score averages form final references.Source: §3.1.3 · p. 3
3.1.4 Data Acquisition and Usage.
- The data was split into 25% training, 25% dry-run, and 50% hidden formal-run sets.Training annotations were released for method development; the dry run supported refinement, and the hidden formal run determined final results.Source: §3.1.4 · p. 3
3.2 Evaluation Metrics
The section evaluates participant scores against human annotations with pairwise accuracy, Spearman’s rho, and Kendall’s tau across four dimensions and a weighted overall score. It defines how each metric compares scores or rankings and averages per-question values.
3.2 Evaluation Metrics
- The weighted overall score combines four rubric-dimension scores using rubric weights, applied identically to participant scores and human annotations.Source: §3.2 · p. 3
- Pairwise accuracy compares every answer pair and counts a tie as correct only when both participant scores and human annotations indicate a tie.Source: §3.2 · pp. 3–4
3.2.2 Spearman’s Rank Correlation Coefficient
- Spearman’s rho compares the ranks of answers derived from participant scores and human annotations.Source: §3.2.2 · p. 4
3.2.3 Kendall’s Rank Correlation Coefficient
- Kendall’s tau-b measures pairwise ordering consistency while accounting for ties in participant scores or human annotations.Concordant and discordant pairs, and pairs tied in only one set of scores, enter its calculation.Source: §3.2.3 · p. 4
3.2 Evaluation Metrics
- Each metric is computed separately for every question and then averaged across questions; higher values indicate stronger agreement with human annotations.Source: §3.2.3 · p. 4
3.3 Answer Organization
Answer files follow a format similar to those used by most TREC submissions, with seven whitespace-separated columns on each line. The columns identify the task, question, answer, and four dimension scores.
3.3 Answer Organization
- Each line must contain exactly seven columns separated by at least one space; column widths do not matter.Source: §3.3 · p. 4
- The seven columns are taskId, questionId, answerId, comprehensiveness_score, insight_score, instruction_following_score, and readability_score.Source: §3.3 · p. 4
- The first three columns identify the task, question within the task, and answer from an LLM; the remaining columns contain the four dimension scores.Source: §3.3 · p. 4
4 Participants’ Methods
Participants used LLM judges, rubric-based scoring, pairwise comparisons, prompt optimization, and feature-based adjustments to score deep research reports. LINKLab uniquely fine-tuned a judge, while other teams combined scoring signals in different ways.
4 Participants’ Methods
- RSLIR uses a DeepSeek-v4-Flash judge to score one dimension per call and aggregates criterion scores with official rubric weights.Because criterion order caused larger score variation than repeated runs, it averages scores from the original and reversed orders.Source: §4 · p. 4
- TMUHLLA ranks reports primarily with a holistic Gemma-4 31B judge and consults a secondary signal only to break exact ties.That signal combines rubric-conditioned Qwen3-32B scores, text features, and pairwise judgments retained only when both A/B orders agree.Source: §4 · p. 4
- LINKLab uniquely fine-tunes a LoRA-adapted Qwen3-8B to predict four-dimensional preferences from position-balanced human-score-derived labels.It compares full reports pairwise and aggregates preferences with a Bradley–Terry model for each dimension.Source: §4 · p. 4
- IITBHU uses Gemini 3.1 Flash Lite to score every rubric sub-criterion for a full report in one zero-shot call, followed by weighted aggregation.It achieved the second-best Kendall correlation on insight and readability.Source: §4 · p. 4
- SCUNLP scores roughly twenty question-specific rubric criteria separately, then combines them with rubric weights for continuous scores without ties.It also blends structural-feature regression into readability and corrects length bias in instruction following; it achieved the best formal-run readability results.Source: §4 · p. 4
- THUFAITH anchors each dimension with averaged pointwise scores from two LLM judges and fits pooled nearby-report comparisons for insight with a regularized Davidson–Bradley–Terry model.Four judges compare reports in both presentation orders.Source: §4 · p. 4
5 Official Results
RSLIR led the overall results and three evaluation dimensions, while SCUNLP led readability. Instruction following and readability remained harder to evaluate, and the three metrics showed generally consistent but not identical rankings.
5 Official Results
- RSLIR led weighted overall (Spearman correlation 0.7071, Kendall correlation 0.5592, accuracy 0.7796); it also led comprehensiveness, insight, and instruction following, while SCUNLP led readability.Table 4 reports the formal run on the reserved set.Source: §5 · pp. 4–5, Table 4 · p. 5
- The three metrics generally ranked methods consistently, but dimension-specific differences support considering multiple metrics for a more reliable and comprehensive assessment.For comprehensiveness, ISLab ranked second on Spearman and Kendall correlations, while THUFAITH ranked second on accuracy.Source: §5 · p. 5
- For readability, the best Spearman and Kendall correlations reached only 0.5197 and 0.3827, respectively, while accurately judging instruction adherence remains difficult for automatic evaluators.Source: §5 · p. 5
6 Conclusion
AEOLLM-2 gathered varied approaches to evaluating long-form LLM responses, with RSLIR leading overall and SCUNLP leading readability. The results also point to challenges that differ by evaluation dimension and a planned expansion to more diverse data and scenarios.
6 Conclusion
- RSLIR led overall and on comprehensiveness, insight, and instruction following, while SCUNLP led readability; metrics mostly aligned, and instruction following and readability were harder.The conclusion describes the three metrics as generally consistent while noting that they capture complementary aspects of evaluation performance.Source: §6 · p. 5
- AEOLLM-2 received 91 runs from 10 teams, covering diverse approaches to LLM evaluation.Source: §6 · p. 5
- The authors plan to extend AEOLLM-2 with more diverse data and scenarios to support more comprehensive and reliable LLM evaluation.Source: §6 · p. 5