Source-linked AI summary
LLMs as Evaluators: A Novel Approach to Evaluate Bug Report Summarization
Abhishek Kumar, Sonia Haiduc, Partha Pratim Das, Partha Pratim Chakrabarti
TL;DR
Human judgment is trusted for evaluating software summaries but is time-consuming, fatiguing, and difficult to scale and reproduce, while automatic metrics often have limited alignment with human assessment. The paper compares humans with GPT-4o, LLaMA-3, and Gemini on bug-title and bug-summary evaluation tasks. LLMs generally perform well, GPT-4o leads the other models, and human accuracy is affected by fatigue.
Problem
Human evaluation of software summaries is trusted but time-consuming and fatiguing, making evaluation difficult to scale and reproduce.
Source: §I · p. 1Method
The study compares humans with GPT-4o, LLaMA-3, and Gemini on selecting correct bug titles and bug report summaries from options.
Source: §I · p. 1Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Summarizing software artifacts is an important task that has been thoroughly researched. For evaluating software summarization approaches, human judgment is still the most trusted evaluation. However, it is time-consuming and fatiguing for evaluators, making it challenging to scale and reproduce. Large Language Models (LLMs) have demonstrated remarkable capabilities in various software engineering tasks, motivating us to explore their potential as automatic evaluators for approaches that aim to summarize software artifacts. In this study, we investigate whether LLMs can evaluate bug report summarization effectively. We conducted an experiment in which we presented the same set of bug summarization problems to humans and three LLMs (GPT-4o, LLaMA-3, and Gemini) for evaluation on two tasks: selecting the correct bug report title and bug report summary from a set of options. Our results show that LLMs performed generally well in evaluating bug report summaries, with GPT-4o outperforming the other LLMs. Additionally, both humans and LLMs showed consistent decision-making, but humans experienced fatigue, impacting their accuracy over time. Our results indicate that LLMs demonstrate potential for being considered as automated evaluators for bug report summarization, which could allow scaling up evaluations while reducing human evaluators effort and fatigue.
I. INTRODUCTION
The paper examines LLMs as evaluators for bug report summaries because human and automatic evaluation have important scalability and reliability limitations. It compares LLM-based evaluation with human judgment and proposes an automated alternative for software summarization assessment.
Motivation and research question
- Automatic metrics such as BLEU, ROUGE, and METEOR are difficult to interpret for software engineering and often do not correlate with human judgment.Source: §I · p. 1
- LLM evaluation is proposed as an alternative to human assessment of bug report summaries, addressing human evaluation's time, cost, scalability, and fatigue challenges.The study asks whether LLMs can accurately evaluate summary quality under structured evaluation criteria.Source: §I · p. 1
Study contribution
- The study compares GPT-4o, LLaMA-3, and Gemini with human evaluators on bug-title and bug-report-summary evaluation tasks using shared instructions and criteria.Source: §I · p. 1
- The study reports that LLMs performed well overall, with GPT-4o generally outperforming the other LLMs, while human accuracy declined with fatigue.Source: §I · pp. 1–2
- The work explores LLMs as evaluators for bug report summarization and presents a first comparison with human evaluators.Source: §I · pp. 1–2
- A replication package containing the study design and results is provided to support further research and replication.Source: §I · p. 2
II. RELATED WORK
Prior work identifies weaknesses in both automatic metrics and human evaluation for software summarization and related software-engineering tasks. These limitations motivate more reliable, standardized, and scalable evaluation approaches.
Research gap
- The paper positions its work against the absence of prior studies using LLMs as evaluators in software summarization tasks.Source: §II · p. 2
Limitations of automatic evaluation
- Studies report that BLEU, ROUGE, word-overlap, and related automatic metrics often disagree with developer or human judgments of software quality.Source: §II · p. 2
Limitations of human evaluation
- Prior research also finds human evaluations costly, subjective, difficult to standardize, and vulnerable to evaluator expertise and sample-size limitations.Source: §II · p. 2
III. STUDY ON EVALUATING BUG SUMMARIES
Bug summarization condenses reports into concise information that helps developers understand and prioritize issues. The study evaluates whether GPT-4o, LLaMA-3, and Gemini can assess such summaries against human judgments.
Study context
- Bug summarization highlights essential report information to support issue understanding, prioritization, efficient fixing, reduced debugging time, and software quality.Source: §III · p. 2
- The study compares GPT-4o, LLaMA-3, and Gemini with human evaluators for bug report summary assessment.Source: §III · p. 2
A. Dataset Preparation
The dataset is drawn from GitHub's Signal repository and includes diverse real-world bug reports for two evaluation tasks. It contains 20 reports for bug-title evaluation and 10 for bug-report-summary evaluation.
Dataset Preparation
- The study selects a diverse, representative subset of bug reports from the Signal repository hosted on GitHub.Source: A. Dataset Preparation · p. 2
- The dataset contains 20 reports for bug-title evaluation and 10 reports for bug-report-summary evaluation based on descriptions and comments.Source: A. Dataset Preparation · p. 2
1) Task Description:
The study compared humans and three LLMs on selecting correct bug titles and summaries using shared instructions and evaluation criteria.
Task 1
- Task 1 asked participants to select the best bug title from four options given a bug description.The 20 questions included easy, hard, and None Of The Above cases, evaluated for factual correctness and completeness.Source: 1) Task Description · p. 2
Task 2
- Task 2 asked participants to select the best bug report summary from options based on a bug description and associated comments.The 10 questions included easy, hard, and None Of The Above cases, evaluated for factual correctness, no hallucinations, and adequate coverage.Source: 1) Task Description · p. 3
Common procedure
- Humans and LLMs received the same instructions, definitions, examples, questions, and evaluation criteria for both tasks.The LLMs assessed factual correctness, completeness, and adequate coverage before selecting an option or None Of The Above.Source: 1) Task Description · p. 3
2) Evaluation Metric:
Accuracy measured how often evaluators selected the correct answer for each question, providing a common measure across tasks and difficulty levels.
Accuracy
- Accuracy was calculated as the number of correct evaluations divided by the total number of evaluations.The measure compared each evaluator’s selections with the correct answers.Source: 2) Evaluation Metric · p. 3
- This accuracy measure enabled comparison across different tasks and difficulty levels.It accounted for the performance of each evaluator.Source: 2) Evaluation Metric · p. 3
3) Large Language Model Selection:
The study selected GPT-4, Llama3, and Gemini because of their reported effectiveness on programming-related and software-text tasks.
Model selection
- The evaluation used GPT-4, Llama3, and Gemini as the three selected large language models.The models were chosen for demonstrated performance on code generation, bug detection, and software-related text understanding.Source: 3) Large Language Model Selection · p. 3
IV. RESULTS
The results compared human and LLM evaluators across bug-title and bug-summary tasks, factual correctness, completeness, difficulty levels, and participant groups. GPT-4o generally led, while performance varied on more complex and None Of The Above cases.
Participants
- The study included 13 Task 1 evaluators and 10 Task 2 evaluators with varied programming experience and occupations.Task 1 participants included students, research scholars, and professional developers; Task 2 had a similarly mixed group.Source: §IV · p. 3
RQ1: Overall performance
- GPT-4o generally led evaluation performance across bug-title and bug-summary tasks, while all evaluators struggled on challenging None Of The Above summary cases.In Task 1, GPT-4o maintained near-perfect accuracy at medium difficulty; in Task 2, it led across difficulty levels but still declined on None Of The Above cases.Source: §IV · pp. 3–4, Figure 2 · p. 4
RQ2: Factual correctness
- GPT-4o consistently outperformed the other evaluators on factual correctness for bug titles across all difficulty levels.Humans declined as difficulty increased, Llama-3 was moderate, and Gemini struggled particularly on medium and None Of The Above tasks.Source: §IV · p. 4, Figure 3 · p. 4
- For factual correctness in bug summaries, humans remained relatively accurate, Llama-3 excelled on None Of The Above tasks, and Gemini varied by difficulty.GPT-4o was competitive but more variable, while Gemini performed well on easier tasks and struggled on medium and None Of The Above tasks.Source: §IV · p. 4, Figure 3 · p. 4
RQ3: Completeness
- GPT-4o achieved perfect scores on easier bug-title completeness tasks, whereas humans and Llama-3 declined as difficulty increased.All evaluators declined on None Of The Above title tasks, except GPT-4o, which captured completeness more accurately.Source: §IV · p. 4, Figure 4 · p. 4
- Bug-summary completeness was highly consistent on easy tasks, and GPT-4o and Llama-3 maintained strong performance as difficulty increased.Humans declined slightly on None Of The Above tasks, while Gemini performed strongly and particularly excelled on those tasks.Source: §IV · p. 4, Figure 4 · p. 4
identify hallucinations in bug summaries?
GPT-4o was the most reliable evaluator for detecting hallucinations in bug report summaries, while performance varied across models and declined for humans as task difficulty increased.
- GPT-4o achieved perfect scores on easy and medium hallucination-detection tasks and remained the most reliable evaluator overall.Its performance showed minimal hallucination risk across difficulty levels.Source: identify hallucinations in bug… · p. 5
- Humans performed well overall but became less accurate as task difficulty increased.Source: identify hallucinations in bug… · p. 5
- LLaMA-3 and Gemini handled easier tasks well but showed substantial accuracy declines on more complex or NOTA tasks.LLaMA-3 particularly declined on NOTA tasks, while Gemini struggled on both medium and NOTA tasks.Source: identify hallucinations in bug… · p. 5
V. CONCLUSIONS AND FUTURE WORK
The preliminary study found that LLMs can evaluate issue report summaries effectively across criteria, with GPT-4o showing robust performance as human accuracy declined with task complexity. Future work will broaden the participant base and task coverage.
- LLMs performed remarkably well as evaluators across various issue report summary evaluation criteria.Source: §V · p. 5
- Human evaluators excelled on simpler evaluations but their performance declined as task complexity increased.Source: §V · p. 5
- GPT-4o demonstrated robust capabilities across different difficulty levels, supporting its potential as a reliable bug report summary evaluator.Source: §V · p. 5
- Future work will scale participant numbers and diversity and study LLM evaluators across a broader range of tasks.Source: §V · p. 5