PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Large Language Models Serve as Evaluators for Code Summarization?

Yang Wu, Yao Wan, Zhaoyang Chu, Wenting Zhao, Ye Liu, Hongyu Zhang, Xuanhua Shi, Philip S. Yu

arXiv:2412.01333v1cs.SE

TL;DR

Human evaluation of code summaries is labor-intensive, and standard automatic metrics often align weakly with human judgments. The paper proposes CODERPE, which uses role-playing LLM prompts to evaluate summaries, and reports 81.59% correlation with human assessment, including without reference summaries.

  • Problem

    Human evaluation is difficult to scale, while common reference-based metrics show relatively low correlation with human judgments, motivating investigation of LLM evaluators.

    Source: §I · p. 1

  • Method

    CODERPE prompts an LLM to take code-related roles and assess summary quality across coherence, consistency, fluency, and relevance.

    Source: §I · p. 2, A. Multi-Role Player Design · p. 4

  • Results

    81.59% correlation with human assessment is reported for CODERPE, including when reference summaries are absent.

    Source: §I · p. 2

  • Takeaways & Limitations

    The reported results present role-playing LLM evaluation as a promising alternative to traditional code-summary metrics.

    Source: §VII · p. 11

    The study uses one LLM agent in multiple roles and leaves additional roles and multi-agent collaboration for future work.

    Source: C. Threats to Validity · p. 10

Abstract

from arXiv · show

Code summarization facilitates program comprehension and software maintenance by converting code snippets into natural-language descriptions. Over the years, numerous methods have been developed for this task, but a key challenge remains: effectively evaluating the quality of generated summaries. While human evaluation is effective for assessing code summary quality, it is labor-intensive and difficult to scale. Commonly used automatic metrics, such as BLEU, ROUGE-L, METEOR, and BERTScore, often fail to align closely with human judgments. In this paper, we explore the potential of Large Language Models (LLMs) for evaluating code summarization. We propose CODERPE (Role-Player for Code Summarization Evaluation), a novel method that leverages role-player prompting to assess the quality of generated summaries. Specifically, we prompt an LLM agent to play diverse roles, such as code reviewer, code author, code editor, and system analyst. Each role evaluates the quality of code summaries across key dimensions, including coherence, consistency, fluency, and relevance. We further explore the robustness of LLMs as evaluators by employing various prompting strategies, including chain-of-thought reasoning, in-context learning, and tailored rating form designs. The results demonstrate that LLMs serve as effective evaluators for code summarization methods. Notably, our LLM-based evaluator, CODERPE , achieves an 81.59% Spearman correlation with human evaluations, outperforming the existing BERTScore metric by 17.27%.

I. INTRODUCTION

Human review of code summaries is difficult to scale, while commonly used reference-based metrics correlate weakly with human judgments. This study introduces CODERPE, a role-playing LLM evaluator, and reports strong alignment with human assessment.

I. INTRODUCTION

  • Human evaluation is labor-intensive, while BLEU, ROUGE, METEOR, and BERTScore compare generated summaries with references and correlate relatively weakly with human evaluations.The paper asks whether LLMs can evaluate summaries without reference summaries.Source: §I · p. 1
  • In the motivating example, automatic metrics score a ChatGPT summary poorly even though manual inspection finds it high quality and linked to code details.Figure 1 juxtaposes the code, reference and generated summaries, and scores from metrics, human evaluation, and LLMs.Source: §I · p. 1, Figure 1 · p. 2
  • CODERPE prompts an LLM to act as a code reviewer, author, editor, or system analyst and evaluate coherence, consistency, fluency, and relevance.The study examines role-player design, rating forms, chain-of-thought prompting, and in-context example selection.Source: §I · p. 2
  • 81.59% correlation with human assessment is reported for CODERPE, including evaluation without reference summaries.The paper also reports that CODERPE evaluation favors ChatGPT summaries for semantic accuracy and diverse phrasing.Source: §I · p. 2
  • The study compares six code summarization models, including ChatGPT, using CODERPE.The models are CodeNN, Deepcom, Astattgru, Rencos, NCS, and ChatGPT.Source: §I · p. 2

II. BACKGROUND

The background groups code-summarization evaluation into n-gram, embedding, and human metrics. N-gram metrics compare generated and reference summaries through shared sequences or matched words.

A. Existing Code Summarization Evaluation Metrics

1) N-gram-based Metrics:

  • N-gram metrics measure similarity to a reference through shared n-grams; BLEU averages n-gram precision and penalizes overly short outputs.BLEU variants differ in smoothing and aggregation; the passage describes BLEU-CN, BLEU-DM, BLEU-DC, BLEU-NCS, BLEU-RC, and BLEU-FC.Source: 1) N-gram-based Metrics · pp. 2–3
  • ROUGE-L measures candidate-reference similarity using their longest common subsequence, emphasizing information order and balancing precision with recall.The example gives precision 0.6, recall 0.5, and a ROUGE-L score of 0.54 when β = 1.Source: 1) N-gram-based Metrics · p. 3
  • METEOR matches words between candidate and reference sentences and combines unigram precision and recall, with a recall-oriented emphasis.The example reports a METEOR score of 0.5 for precision 0.6 and recall 0.5.Source: 1) N-gram-based Metrics · p. 3

2) Embedding-based Metrics:

Embedding-based metrics compare generated and reference summaries through their representations rather than token overlap. BERTScore matches token embeddings and combines precision and recall.

2) Embedding-based Metrics:

  • BERTScore compares embeddings of generated and reference summaries, unlike n-gram metrics that assess semantic similarity through overlapping tokens.Source: 2) Embedding-based Metrics · p. 3
  • BERTScore matches candidate and reference tokens using pairwise embedding similarities, then computes a harmonic mean of precision and recall.Source: 2) Embedding-based Metrics · p. 3
  • A worked example reports BERTScore 0.82 from precision 0.90 and recall 0.76 using RoBERTa embeddings.Source: 2) Embedding-based Metrics · p. 4

3) Human Evaluation Metrics:

Human evaluation rates generated summaries against references using criteria such as fluency and coherence, but it is costly and subjective. The background also describes prompting, in-context learning, chain-of-thought, and role-playing approaches for LLM tasks.

3) Human Evaluation Metrics:

  • Human assessors compare references with model outputs using criteria such as fluency, coherence, and overall understanding, often on a five-point scale.Source: 3) Human Evaluation Metrics · p. 4
  • Professional evaluator expense makes human evaluation difficult to scale, while subjectivity calls for bias mitigation and consistency efforts.Source: 3) Human Evaluation Metrics · p. 4

B. Large Language Models

  • Prompting reformulates downstream tasks as textual prompts aligned with LLM training rather than full fine-tuning.Source: B. Large Language Models · p. 4
  • In-context learning uses k retrieved task examples as demonstrations in the prompt before the LLM predicts an answer.Source: B. Large Language Models · p. 4
  • Chain-of-thought prompting adds intermediate reasoning steps to demonstration examples to guide the model toward its final output.Source: B. Large Language Models · p. 4
  • The multi-role approach prompts an LLM to simulate expert roles, with a code-reviewer example comprising a role, task, evaluation steps, demonstrations, and test example.Source: B. Large Language Models · p. 4

III. LLMS AS CODE SUMMARIZATION EVALUATORS

CODERPE uses role-player prompting to evaluate generated code summaries in reference-based and reference-free scenarios. Its design assigns roles to assess summary dimensions, while prompting strategies provide task instructions and evaluation steps.

III. LLMS AS CODE SUMMARIZATION EVALUATORS

  • CODERPE uses role-player prompting to assess generated summaries in both reference-based and reference-free scenarios.Source: §III · p. 4

A. Multi-Role Player Design

  • The method assigns roles such as code reviewer, author, editor, or system analyst to assess one dimension, including coherence, consistency, fluency, or relevance.Source: A. Multi-Role Player Design · pp. 4–5

B. Prompting Strategies

  • Prompt instructions ask the LLM to read code, compare comments with code, and rate coherence from 0 to 4; evaluation examples may include a reference summary and human score.Source: B. Prompting Strategies · p. 5

3) Rating Forms:

The paper examines rating-form designs, chain-of-thought, in-context examples, evaluator settings, and re-evaluation of code summarization models. Its table compares GPT-4 CODERPE role players with conventional metrics.

3) Rating Forms:

  • Three rating forms vary the response format: score-only, analyze-rate before scoring, and rate-explain after scoring.Source: 3) Rating Forms · p. 5

1) CoT:

  • The chain-of-thought strategy adds detailed evaluation steps to guide the LLM's reasoning toward a final score.Source: 1) CoT · p. 5

2) ICL:

  • In-context learning supplies selected annotated examples, with the number of demonstrations adjustable as K.Source: 2) ICL · p. 5
  • RQ1 asks whether CODERPE aligns with human evaluation compared with traditional metrics.Source: 2) ICL · p. 5
  • RQ2 examines performance across evaluator types, demonstration counts, turns, and prompting strategies.Source: 2) ICL · p. 5
  • RQ3 re-evaluates six neural summarization models, while Table I compares GPT-4 CODERPE role players with conventional metrics.Source: 2) ICL · p. 6, Table I · p. 6

A. Datasets and Models to Re-Evaluate

The study uses TL-CodeSum and six code-summarization models to evaluate summaries against human judgments across four dimensions.

1) Datasets:

  • TL-CodeSum contains 87,136 Java code-summary pairs from 9,732 GitHub projects and is split 8:1:1; 300 annotated summaries provide ground-truth labels.The projects were collected from 2015–2016 and had at least 20 stars.Source: 1) Datasets · p. 6

2) Models to Re-Evaluate:

  • Six re-evaluated models are CodeNN, Deepcom, Astattgru, Rencos, NCS, and ChatGPT.They represent varied approaches, including AST traversal, recurrent and Transformer models, retrieval, and prompting.Source: 2) Models to Re-Evaluate · p. 6

1) Dimensions to Assess a Code Summary:

  • Human ratings assess coherence, consistency, fluency, and relevance on a 0–4 scale, with 15 experienced annotators rating 300 summaries.The annotators’ scores are averaged per summary, and Kendall’s Tau is used to check agreement.Source: 1) Dimensions to Assess a Code Summary · p. 6

2) Metrics to Assess the Alignment with Human:

The study compares automatic metrics with human scores using Kendall-Tau and Spearman correlations, which capture rank-based association.

C. RQ1: Performance of CODERPE Evaluator

CODERPE correlates more strongly with human judgments than conventional metrics across summary-quality dimensions. Its reviewer role reaches 82.23% Spearman correlation with references and 81.59% without them, while evaluator choice involves a performance-cost trade-off.

C. RQ1: Performance of CODERPE Evaluator

  • Spearman correlation rises from 64.32% for BERTScore to 82.23% with references and 81.59% without references for CODERPE acting as a code reviewer.CODERPE also significantly outperforms BLEU, ROUGE, METEOR, and BERTScore across coherence, consistency, fluency, and relevance.Source: C. RQ1: Performance of CODERPE Evaluator · p. 7
  • Reference-based and reference-free CODERPE evaluations differ only minimally, and the authors report that scores are not strongly dependent on reference summaries.Source: C. RQ1: Performance of CODERPE Evaluator · p. 7

1) Influence of Evaluator Types:

  • GPT-4 performs better than the GPT-3.5 models, while text-davinci-003 has well-rounded performance and GPT-4 carries higher API costs.The authors recommend choosing an evaluator according to performance needs and budget.Source: 1) Influence of Evaluator Types · p. 8

2) Influence of Number of Demonstration Examples:

The re-evaluation finds ChatGPT stronger than the other tested neural models under CODERPE, while evaluator performance also varies with demonstrations, turns, and prompt design.

2) Influence of Number of Demonstration Examples:

  • Four demonstrations generally outperform zero demonstrations, improving average Spearman correlation by 3.59% for text-davinci-003 in the reference-free setting.The study compares 0, 4, and 8 demonstrations and recommends four.Source: 2) Influence of Number of… · p. 8

3) Influence of Turns:

  • Three evaluation turns approximate human scores, with Kendall-Tau correlation of 55.54% and Spearman correlation of 77.51%; more turns generally correlate with improved performance.Source: 3) Influence of Turns · p. 8

4) Influence of Prompt Settings:

  • Chain-of-thought guidance improves fluency by 0.98% in Kendall-Tau and 1.95% in Spearman correlation, but is not universally beneficial.Source: 4) Influence of Prompt Settings · p. 8
  • The analyze-rate form consistently outperforms rate-explain, but form-based strategies do not necessarily beat score-only prompting.The authors conclude that CODERPE requires careful prompt design and base-LLM selection.Source: 4) Influence of Prompt Settings · p. 8

E. RQ3: Re-Evaluation of Current Models

  • ChatGPT summaries achieve approximately 90% across all four dimensions, with coherence rising from 56.25% for NCS to 89.50% for ChatGPT.The authors report this result as contrasting with prior work [27], which found lower ChatGPT performance using BLEU, METEOR, and ROUGE-L.Source: E. RQ3: Re-Evaluation of Current Models · p. 8, Table III · p. 8

V. DISCUSSION

The case study shows both strong agreement with human judgments in a good case and failure modes involving hallucinations and differing preferences for detail. The authors present LLM evaluation as complementary to human evaluation and draw implications for prompt, model, and role design.

A. When do LLM-based Evaluators Work and Fail?

  • In a good case, CODERPE gave ChatGPT’s summary the highest score, 4, matching manual judgments that it was better than summaries from other models.The authors attribute this to LLM summarization capabilities.Source: A. When do LLM-based Evaluators Work… · p. 9
  • An evaluator gave a consistency score of 4 to a summary containing a hallucination about printing a newline, because relevant functions were defined outside the supplied code.The authors suggest pretraining associations with the abbreviations may contribute to the oversight.Source: A. When do LLM-based Evaluators Work… · p. 9
  • ChatGPT received a score of 2 for a concise, effective summary, which the authors suggest may reflect the evaluator’s subjective preference for more detail.Source: A. When do LLM-based Evaluators Work… · p. 9
  • The authors recommend combining LLM and human evaluation, noting that subjective preferences in either can introduce variability into judgments.Source: A. When do LLM-based Evaluators Work… · p. 9

B. Implications

  • The authors report that LLM evaluators align with human judgments better than BLEU-R, ROUGE-L, METEOR, and BERTScore, and emphasize prompt strategies, backbone choice, and role design.They also report that CODERPE ranks ChatGPT above other neural models for code summarization.Source: B. Implications · p. 9

C. Threats to Validity

The paper concludes that CODERPE aligns more closely with human evaluations, while identifying constraints in role coverage, prompt design, and ground-truth data. The surrounding sections position this work within research on code summarization and LLM-based evaluation.

C. Threats to Validity

  • The study uses four roles and one LLM agent, relies on manually designed prompts, and treats 300 human-annotated summaries as ground truth, limiting role coverage, prompt scalability, and data diversity.The authors identify multi-agent collaboration, broader prompts, and more summaries spanning diverse code types as future work.Source: C. Threats to Validity · p. 10

A. Code Summarization

  • Code summarization research spans automated models trained on code-summary corpora, pretrained models such as CodeBERT and CodeT5, and recent LLM-based approaches.The paper focuses on evaluating generated summaries rather than generating them.Source: A. Code Summarization · p. 10

B. Code Summarization Evaluation

  • Automatic code-summary metrics compare outputs with references but may miss semantic and comprehension effects, while human evaluations can address some shortcomings but are labor-intensive.Source: B. Code Summarization Evaluation · p. 10

C. Natural Language Generation Evaluators

  • Prior LLM evaluation work includes role-player prompting for generated text [62], reference-free evaluation, and multi-agent approaches that simulate annotator consensus [17].Other cited work explores LLM evaluation in translation, text, and code-generation tasks.Source: C. Natural Language Generation… · p. 10

VII. CONCLUSION

  • CODERPE evaluates code summaries across coherence, consistency, fluency, and relevance using role-playing LLMs, and the authors report closer alignment with human evaluations than traditional metrics.The paper also investigates prompt strategies and evaluator robustness.Source: §VII · p. 11
Loading 2412.01333v1…