On the Evaluation of Neural Code Summarization

• The evaluation addresses a practical need: code summaries aid comprehension and maintenance, but documentation is labor-intensive and summaries are often missing, mismatched, or outdated.Source: §1 · p. 1
• The study systematically evaluates five representative approaches across six BLEU variants, four preprocessing operations, and three datasets.Source: §1 · p. 1
• BLEU variants produce rather different results for the same summaries, and human evaluation finds BLEU-DC most correlated with human perception.The paper warns that inconsistent implementations and direct comparisons across BLEU variants can make reported comparisons unfair or risky.Source: §1 · p. 2
• Different preprocessing combinations shift overall performance by -18% to +25%, and a simple ensemble technique can boost model performance.Source: §1 · p. 2
• Models perform inconsistently across datasets, whose corpus sizes, data-splitting methods, and duplication ratios significantly affect performance.Source: §1 · p. 2
• The authors provide actionable evaluation suggestions and a shared toolbox with metric implementations, preprocessing operations and combinations, datasets, model reimplementations, and results.Source: §1 · p. 2

2.1 Code Summarization

• Earlier template-based approaches require expert domain knowledge, while retrieval-based summaries can mismatch code semantics and behavior.Source: §2.1 · p. 2
• Neural machine-translation approaches generate code summaries, with models incorporating ASTs, API knowledge, type information, context, and other techniques; hybrids combine neural and retrieval methods.Source: §2.1 · p. 2

2.2 BLEU

• BLEU combines n-gram precision scores using a geometric mean and applies a brevity penalty to address inflated scores for short or repetitive sentences.Source: §2.2 · p. 3
• Prior code summarization work uses multiple BLEU variants with different calculation levels and smoothing methods, which can yield different scores for the same summary.Source: §2.2 · p. 3
• Reported BLEU scores can imply interpretations that conflict with reported model rankings; for example, the cited score interpretations label Astattgru harder to understand than Deepcom.Source: §2.2 · p. 3

3.1 Datasets

• The 12-dataset evaluation varies corpus size, data splitting, and duplication to examine how these characteristics affect model performance.The extended datasets include variants of TL-CodeSum, Funcom, and CodeSearchNet; training-set sizes define the small, medium, and large magnitudes.Source: §3.1 · pp. 3–4, §3.4 · p. 5

3.2 Evaluated Approaches

• Five representative approaches span encoder-decoder, AST-aware, Transformer, and retrieval-augmented neural summarization models.The evaluated systems are CodeNN, Deepcom, Astattgru, NCS, and Rencos.Source: §3.2 · p. 4

3.3 Experimental Settings

• Experiments use the approaches’ provided implementations where available, tune selected hyperparameters, and report means and standard deviations across three runs.Deepcom is reimplemented and checked against its published scores; early stopping is used, with a maximum of 200 epochs for TLC and TLCDedup and 40 for other datasets.Source: §3.3 · p. 4

3.4 Research Questions

• The research questions test BLEU variants, four widely used preprocessing operations and their combinations, and dataset size, splitting, and duplication effects.The human evaluation samples 300 generated summaries, uses five annotators, and compares BLEU variants with human ratings; corpus-level human scores are aggregated by arithmetic and geometric averages.Source: §3.4 · pp. 4–5

4.1 Analysis of Different Evaluation Metrics. (RQ1)

• BLEU variants produce different scores and can reverse model rankings on TLC; Astattgru beats CodeNN under BLEU-FC but trails it under other variants.Under BLEU-FC, all evaluated models except Rencos score below 20 on TLCDedup.Source: §4.1 · p. 5, Table 5 · p. 6

4.1.1 Sentence v.s. corpus BLEU.

• Sentence-level and corpus-level BLEU aggregate sentence contributions differently, so their scores are not comparable and the calculation level should be reported.Sentence-level BLEU averages sentence scores equally, whereas corpus-level BLEU aggregates n-gram counts across the corpus.Source: §4.1.1 · p. 6

4.1.2 Smoothing methods.

• Smoothing methods can yield substantially different n-gram precision values, especially for short sentences common in code summaries.Methods 1–4 replace zero counts with positive values, while methods 5–7 average matched n-gram counts in different ways.Source: §4.1.2 · p. 6

4.1.3 Bugs in software packages.

• NLTK-version bugs in smoothing methods 0 and 4 cause BLEU-DM and BLEU-DC scores for the same summaries to differ by up to 97%.The authors report that method 4 in NLTK 3.2.2–3.4.x can inflate precision, and recommend NLTK 3.6.x for method 4; comparisons should use the same correct implementation.Source: §4.1.3 · pp. 6–7

4.1.4 Human evaluation.

• BLEU-DC, sentence-level BLEU with method 4, correlates significantly better with human perception than the other compared variants.The authors attribute its stronger correlation to method 4 smoothing zero values without inflating precision as methods 2 and 3 do.Source: §4.1.4 · pp. 7–8, Table 7 · p. 7

4.2 The Impact of Different Pre-processing Operations (RQ2)

• Identifier splitting improves performance for all models, and its improvement is statistically significant; the other three operations show no statistically significant differences.The authors link splitting compound words to reduced OOV ratios and the ability to suggest neologisms unseen in training data.Source: §4.2 · p. 8
• P1101 ranks in the top five for every approach, P0000 ranks in the bottom five, and model rankings are generally consistent across pre-processing settings.The combination comparison covers 16 settings; the passage identifies P1101 as recommended and P0000 as not recommended.Source: §4.2 · p. 8
• -18% to +25%: pre-processing choices affect performance, no combination dominates across approaches, and an ensemble can improve results.The ensemble aggregates P1101, P0101, P0110, and P0111 using stacking; the authors suggest these combinations may contribute complementary information.Source: §4.2 · p. 8

4.3 How Do Different Characteristics of Datasets Affect the Performance?(RQ3)

• Model rankings do not consistently carry across datasets, supporting evaluation on multiple datasets rather than relying on one benchmark.The reported comparisons include Rencos leading on TLC but trailing Astattgru and NCS on FCM; CodeNN’s comparison with Astattgru also changes across datasets.Source: §4.3 · p. 8, Table 10 · p. 9

4.3.1 The impact of different corpus sizes.

• Larger training sets improve performance for the five approaches in both corpus-size groups, while their rankings are generally preserved; Deepcom improves less.The authors suspect Deepcom’s high OOV ratio, associated with its sparse vocabulary of concatenated AST-node types and values, limits its ability to leverage larger datasets.Source: §4.3.1 · p. 9, Table 11 · p. 10

4.3.2 The impact of different data splitting methods.

• All approaches perform better on method-split than project-split data, but their rankings remain basically unchanged across splitting methods.The authors attribute the performance difference to similar tokens and code patterns among methods from the same project; they consider one split reliable for model comparisons when time or data are limited.Source: §4.3.2 · p. 9, Table 12 · p. 10

4.3.3 The impact of different duplication ratios.

• BLEU scores rise as duplication increases, especially for Rencos, and model rankings can change; the authors recommend testing deduplicated data and different duplication ratios.They speculate Rencos benefits because duplicated samples can be retrieved by its retrieval module; they also recommend considering retrieval techniques when code duplication exists.Source: §4.3.3 · p. 9, Figure 3 · p. 10
• Even after controlling split method, duplication ratio, and dataset size, performance still varies greatly between datasets; the impact of training-data differences remains for future work.Source: §4.3.3 · p. 10

Programming languages

• Experiments use Java datasets only, so generalizing the findings to other programming languages requires further evaluation.The authors identify other languages as future work.Source: §5 · p. 10

The quality of summaries

• First-sentence Javadoc summaries may still be incomplete or mismatched, limiting the reference-summary quality.Extracting the first sentence is common under Javadoc guidelines, but does not ensure every summary is complete or matches its code.Source: §5 · p. 10

Models evaluated

• Findings may not hold for models outside the evaluated set, despite its coverage of several model characteristics.The set includes Transformer- and RNN-based, single- and multi-channel, and retrieval and non-retrieval models.Source: §5 · p. 10

Human evaluation

• Using arithmetic or geometric averages to aggregate sentence-level human scores may affect the study’s conclusion.The authors identify exploring other corpus-level human-evaluation methods as future work.Source: §5 · p. 10

6 RELATED WORK

• Prior surveys taxonomized code summarization methods and discussed their advantages, limitations, challenges, and evaluation techniques.Song et al. also discussed evaluation techniques used in existing methods.Source: §6 · p. 10
• Earlier studies examined duplication and dataset splitting, while other work reported that BLEU did not correlate with human judgments for single summaries.Allamanis et al. explored duplication; LeClair et al. compared project- and method-level splits; human studies addressed BLEU’s correlation with quality assessments.Source: §6 · p. 10
• Unlike prior surveys, this study experimentally examines five models across datasets and investigates BLEU inconsistencies, human correlation, preprocessing, and dataset factors.It consolidates findings into actionable evaluation guidelines.Source: §6 · p. 11

7 CONCLUSION

• Metrics, code preprocessing, and dataset choices all substantially affect evaluation results, complicating fair comparison without systematic experiments.The authors recommend explicitly reporting BLEU settings, using appropriate preprocessing, and considering dataset characteristics when evaluating and choosing models.Source: §7 · p. 11
• A shared toolbox includes BLEU variants, preprocessing operations, datasets, baseline implementations, and experimental results.The authors provide the code and data online to facilitate reproducibility.Source: §7 · p. 11
• The authors plan to extend the study beyond Java, explore more dataset attributes, develop a human-correlated automatic metric, and examine other software text-generation tasks.These are stated future-work directions.Source: §7 · p. 11