PaperlayerView exact revision on arXiv

Source-linked AI summary

Developer-Intent Driven Code Comment Generation

Fangwen Mu, Xiao Chen, Lin Shi, Song Wang, Qing Wang

arXiv:2302.07055v1cs.SE

TL;DR

Existing code-comment generators typically describe functionality without accounting for developers’ differing intents, although comments can express several kinds of information. DOME retrieves an exemplar and uses intent-guided selective attention to generate a comment for a specified intent. It outperforms state-of-the-art baselines on two Java datasets, and human evaluation finds its comments useful and relevant.

  • Problem

    Typical code-comment generation maps a snippet to a general description without modeling developer intent, despite comments often expressing multiple intents.

    Source: §I · p. 1

  • Method

    DOME retrieves a similar comment exemplar and uses intent-guided selective attention to select relevant source information for a specified intent.

    Source: A. Exemplar Retriever · p. 3, §IX · p. 11

  • Results

    DOME outperforms state-of-the-art baselines across BLEU, ROUGE-L, and METEOR on both Funcom and TLC, and human evaluation finds its comments useful and relevant.

    Source: §I · p. 2, 3) Results · p. 8

  • Takeaways & Limitations

    DOME generates intent-aware comments for the same source code under different developer intents, with reported potential for practical comment writing.

    Source: §I · p. 2

    Evaluation covers only two Java datasets, and the authors note Java may not represent other programming languages.

    Source: B. Threats to Validity · p. 10

Abstract

from arXiv · show

Existing automatic code comment generators mainly focus on producing a general description of functionality for a given code snippet without considering developer intentions. However, in real-world practice, comments are complicated, which often contain information reflecting various intentions of developers, e.g., functionality summarization, design rationale, implementation details, code properties, etc. To bridge the gap between automatic code comment generation and real-world comment practice, we define Developer-Intent Driven Code Comment Generation, which can generate intent-aware comments for the same source code with different intents. To tackle this challenging task, we propose DOME, an approach that utilizes Intent-guided Selective Attention to explicitly select intent-relevant information from the source code, and produces various comments reflecting different intents. Our approach is evaluated on two real-world Java datasets, and the experimental results show that our approach outperforms the state-of-the-art baselines. A human evaluation also confirms the significant potential of applying DOME in practical usage, enabling developers to comment code effectively according to their own needs.

I. INTRODUCTION

Existing comment generators map code to a general functional description, while real-world comments often express multiple developer intents. DOME targets this gap with intent-guided comment generation, supported by evaluations on two Java datasets and human evaluation.

I. INTRODUCTION

  • Comments can combine distinct intents: one reviewed Java sample averaged 2.81 intents per comment, and over 66.31% of comments had multiple sentences.In the motivating example, sentences cover functionality, rationale, implementation, usage, and code properties, while three existing generators describe only functionality.Source: §I · p. 1
  • DOME outperforms state-of-the-art baselines on both datasets across BLEU, ROUGE-L, and METEOR.On Funcom, improvements are 25.66%, 16.59%, and 18.38%, respectively; on TLC, they are 10.06%, 11.09%, and 14.93%.Source: §I · p. 2
  • DOME uses intent-guided selective attention to select source-code information relevant to a specified intent when generating comments.The approach is presented as the first to incorporate developer intents into comment generation.Source: §I · p. 2
  • The authors introduce COIN, an automated comment-intent labeling tool, and provide a publicly accessible dataset and source code.These resources are presented as supporting dataset construction, replication, and broader application.Source: §I · p. 2
  • Human evaluation finds that DOME generates useful and relevant comments across accuracy, adequacy, and naturalness.The evaluation also reports potential for practical use in helping developers comment code according to their needs.Source: §I · p. 2

II. BACKGROUND AND PROBLEM DEFINITION

The paper frames comment generation around six intent categories and specifies generation conditioned on code and a requested intent. DOME combines retrieval, encoding, and intent-guided decoding; its retriever selects semantically similar examples from an intent-matched corpus.

A. Taxonomy of Comment Intent

  • The comment-intent taxonomy contains six categories: what, why, how-to-use, how-it-is-done, property, and others.The authors treat ambiguous, unspecific “others” examples as noisy and remove identified instances.Source: A. Taxonomy of Comment Intent · p. 2, Table I · p. 2

B. Transformer

  • A Transformer encoder-decoder uses multi-head attention and feed-forward layers, with positional encoding supplying token-order information.Residual connections and layer normalization follow each layer.Source: B. Transformer · p. 2

C. Problem Definition

  • The task is to generate comment y from code x conditioned on intent e, modeling P(y|x,e).At inference, the prediction maximizes conditional likelihood.Source: C. Problem Definition · pp. 2–3

III. APPROACH

  • DOME retrieves an exemplar, encodes code, exemplar, and intent, then decodes a comment using intent-guided selective attention.The decoder selects relevant information from semantic representations to generate an intent-aware comment.Source: §III · p. 3

A. Exemplar Retriever

  • The exemplar retriever filters the corpus to examples with the requested intent, then uses DPR to find a semantically similar comment.DPR represents queries and passages as dense vectors and scores similarity by dot product; intent filtering is described as reducing misleading examples and candidate volume.Source: A. Exemplar Retriever · p. 3

B. The Encoder Layer

The encoder layer combines intent embeddings with separate code and exemplar encoders; the code encoder produces token- and statement-level representations.

B. The Encoder Layer

  • The code and exemplar encoders extract semantic features separately, while the intent embedding captures high-level intent expressions.The code encoder outputs both token- and statement-level representations.Source: B. The Encoder Layer · p. 3

1) Code Encoder:

  • The code encoder flattens statements into one sequence and inserts a [SEP] token after each statement.The authors state this lets statements in the sequence share information and marks statement boundaries.Source: 1) Code Encoder · p. 4
  • The code encoder produces token-level representations and pools them into statement-level representations for intent-guided attention.It applies max pooling within each statement, then concatenates the statement representations.Source: 1) Code Encoder · p. 4

2) Exemplar Encoder:

The decoder uses intent embeddings to select relevant code and exemplar information, then generates a comment conditioned on the intent. The exemplar encoder shares the code encoder's structure but has different parameters.

2) Exemplar Encoder:

  • The exemplar encoder embeds the retrieved exemplar and computes its representation using the same structure as the code encoder, with different parameters.Source: 2) Exemplar Encoder · p. 4

3) Intent Embedding:

  • The intent embedding maps each comment intent to a trainable dense vector that guides selection of intent-relevant encoder information.Source: 3) Intent Embedding · p. 4

C. The Decoder Layer

1) Decoding Process:

  • The gate β represents the degree of source-code/exemplar integration; larger β may signal a semantically different exemplar and a need to attend more to source code.Source: 1) Decoding Process · p. 4
  • At each decoding step, the model projects the decoder state concatenated with the intent embedding, applies Softmax, and uses Argmax to predict the comment token.Source: 1) Decoding Process · p. 4

2) Intent-guided Selective Attention (ISA):

Intent-guided selective attention focuses on information relevant to a requested comment aspect through statement-level selection, token-level selection, and attention combination.

Statement-level Attention Selection

  • Statement-level selection ranks source statements against the intent-conditioned query, retains each row’s k highest scores, and masks the rest before normalization.The selected attention is normalized with Softmax; unrelated source statements receive approximately zero weight.Source: 2) Intent-guided Selective Attention… · p. 5

Token-level Attention Selection

Combining Attentions

  • The method multiplies statement-level and token-level attention element-wise to form the final selective attention matrix.The design reflects a process of first selecting relevant statements, then important tokens within them.Source: 2) Intent-guided Selective Attention… · p. 5

IV. THE COMMENT-INTENT LABELING TOOL

The authors develop COIN to support automatic annotation of comment intents, which they need in large volumes to train and evaluate their approach.

IV. THE COMMENT-INTENT LABELING TOOL

  • The approach requires large volumes of labeled comment-intent data for training and evaluation.Source: §IV · p. 5
  • COIN is a tool for automatic annotation of comment intents.Source: §IV · p. 5
  • The section introduces COIN’s design and analyzes its effectiveness.Source: §IV · p. 5

A. The CodeBERT-based Comment-Intent Classifier

The section describes a CodeBERT-based comment-intent classifier and the manual data preparation used to train it, alongside the classifier baselines. The supplied passages do not report classifier results.

A. The CodeBERT-based Comment-Intent Classifier

  • CodeBERT embeds the concatenated code-comment sequence; a two-layer MLP and softmax predict intent from the final [CLS] embedding.The classifier is fine-tuned with cross-entropy loss on the annotated dataset.Source: A. The CodeBERT-based Comment-Intent… · p. 5

1) Data Preparation:

  • Five developers manually labeled 20K Java code-comment pairs into six intent categories through independent annotation followed by majority-vote conflict resolution.The annotators reached 0.81 Cohen’s Kappa; labeling and conflict resolution took 24.5 days.Source: 1) Data Preparation · pp. 5–6

2) Baselines:

  • The classifier comparison includes LGBM, Random Forest, Decision Tree, BiLSTM, CNN, and BiLSTM+Attention.Random Forest was reported as the strongest classifier in the cited prior work.Source: 2) Baselines · p. 6

3) Evaluation Metrics:

The section reports COIN’s comment-intent classification performance, describes the Funcom and TLC datasets and evaluation metrics, and lists DOME’s implementation settings.

4) Results:

  • 91.1% Macro-Precision, 88.2% Macro-Recall, and 89.6% Macro-F1: COIN outperforms the other classifiers in 10-fold cross-validation.Against CNN, its Macro-Precision, Macro-Recall, and Macro-F1 improve by 7.43%, 24.23%, and 18.99%, respectively.Source: 4) Results · p. 6

A. Dataset

  • Funcom contains 2.1M code-comment pairs, while TLC contains 87,136 pairs; both datasets are preprocessed and automatically intent-labeled before use.The study excludes comments labeled “others” and removes exactly duplicated TLC test pairs.Source: A. Dataset · p. 6, Table IV · p. 6

B. Evaluation Metrics

  • BLEU, ROUGE-L, and METEOR evaluate generated comments using n-gram precision, longest-common-subsequence recall, and a weighted unigram precision-recall harmonic mean, respectively.Metric calculations follow the AST-Trans scripts.Source: B. Evaluation Metrics · pp. 6–7

C. Implementation Details

  • DOME uses dataset-specific batch sizes and comment-length limits, 512-dimensional code and comment embeddings, 128-dimensional intent embeddings, and beam search with size 5.Training uses Adam with learning rate 1e-4 and dropout 0.2.Source: C. Implementation Details · p. 7

VI. RESULTS

The evaluation addresses DOME’s comparison with comment-generation baselines, component contributions, and perceived comment quality. It compares six reproduced baselines and evaluates performance across five intent groups.

VI. RESULTS

  • The evaluation asks how DOME compares with state-of-the-art baselines, how its components contribute, and how people perceive its intent-aware comments.The six baselines were reproduced on the same datasets and checked against results reported in their original papers.Source: §VI · p. 7, 1) Baselines · p. 7

1) Baselines:

  • The six baselines include AST-Trans, CodeTrans, Hybrid-DRL, Re2Com, Rencos, and EditSum, spanning transformer, retrieval, and hybrid approaches.Their described mechanisms include AST structure, copying rare code tokens, retrieval, and retrieve-and-edit generation.Source: 1) Baselines · p. 7

2) Setting:

  • The test set is divided into five intent groups, and all approaches are evaluated on each group and averaged across groups.The study uses BLEU, ROUGE, and METEOR for the intent-category comparisons.Source: 2) Setting · p. 7

3) Results:

DOME outperforms the baselines across intent categories and on aggregate metrics, while component ablations show performance declines when either core component is removed. The section also describes the human-evaluation procedure.

3) Results:

  • 25.66%, 16.59%, and 18.38%: DOME improves BLEU, ROUGE-L, and METEOR over AST-Trans on Funcom; gains over Rencos on TLC are 10.06%, 11.09%, and 14.93%.DOME achieves the best performance on all evaluation metrics, and outperforms baselines for each intent category on both datasets.Source: 3) Results · pp. 7–8, Table V · p. 8

1) Variants:

  • The component variants replace intent-guided selective attention with vanilla cross-attention or remove the exemplar retriever.Both variants use DOME’s experimental setup and are evaluated on Funcom and TLC test sets.Source: 1) Variants · p. 8

2) Results:

  • Removing selective attention lowers average BLEU, ROUGE-L, and METEOR by 3.70%, 4.23%, and 6.94%; removing the retriever lowers them by 6.38%, 6.23%, and 9.97%.The reported results identify positive contributions from both components.Source: 2) Results · p. 9

C. RQ3: Human Evaluation

  • Human evaluation addresses the semantic gap left by lexical metrics by rating generated comments for accuracy, adequacy, and naturalness on a 1–5 scale.Six Java-experienced participants evaluated comments without knowing which approach generated them.Source: C. RQ3: Human Evaluation · p. 9

1) Procedure:

Human evaluation favors DOME across accuracy, adequacy, and naturalness, while qualitative examples show intent-specific comments and selective attention to relevant code. Together, these findings illustrate the approach’s performance and behavior in the reported cases.

2) Results:

  • DOME improves over the best baseline by 14.43% in accuracy, 11.17% in adequacy, and 3.38% in naturalness in human evaluation.Figure 3 presents violin plots for the three aspects; DOME’s average scores are 3.41, 3.44, and 3.91, respectively.Source: 2) Results · p. 9

A. Qualitative Analysis and Attention Visualization

  • In two cases, DOME produces more accurate and fluent comments for different intents, while baseline comments are often short or contain errors.The examples include a DOME “what” comment matching the human-written comment and “why” and usage comments with high semantic similarity to it.Source: A. Qualitative Analysis and Attention… · pp. 9–10
  • Intent-guided selective attention concentrates on code tokens or statements relevant to the requested comment and removes distraction from irrelevant tokens.The examples contrast dispersed vanilla attention and separator-token weights with ISA focusing on the relevant method regions.Source: A. Qualitative Analysis and Attention… · p. 10, Figure 4 · p. 10

B. Threats to Validity

The discussion identifies retrieval, dataset scope, and metric suitability as threats to validity, and describes DOME as addressing intent-aware comment generation through selective attention. It concludes that DOME outperforms baselines in two Java datasets, with human evaluation supporting its practical potential.

B. Threats to Validity

  • DOME’s retrieval assumption is constrained by retriever capability and whether a similar comment exists in the retrieval corpus.The paper uses DPR and gated fusion to mitigate this threat, stating that dissimilar retrieved exemplars will not affect performance.Source: B. Threats to Validity · p. 10
  • Evaluation covers only two Java datasets, and the authors note Java may not represent all programming languages.They argue the datasets are large enough to show effectiveness and suggest language-agnostic features may support broader generalizability.Source: B. Threats to Validity · pp. 10–11
  • Because BLEU’s reliability depends on its use and interpretation, the evaluation also uses ROUGE-L, METEOR, and human assessment.Human evaluation considers accuracy, adequacy, naturalness, and whether comments meet developers’ needs in real-world usage scenarios.Source: B. Threats to Validity · p. 11

A. Automatic Comment Generation

  • Prior automatic comment generation includes template-based, retrieval-based, and learning-based methods, but existing work focuses on general functionality descriptions rather than developer intentions.The paper frames its task as generating intent-aware comments for the same code under different intents.Source: A. Automatic Comment Generation · p. 11

B. Controllable Text Generation

  • DOME applies controllable text generation to code comments, treating generation as one-to-many and using intent to control comment content and style.The related-work discussion contrasts this with controllable generation for attributes, emotions, stories, and task-specific behavior.Source: B. Controllable Text Generation · p. 11

IX. CONCLUSION

  • DOME retrieves an exemplar, encodes the code and exemplar, selects intent-relevant information with selective attention, and fuses features to generate a comment.The conclusion also presents COIN as an automated tool for constructing intent-annotated code-comment datasets and reports evaluation on two Java datasets.Source: §IX · p. 11

Related papers

Loading 2302.07055v1…