Source-linked AI summary
Code Generation as a Dual Task of Code Summarization
Bolin Wei, Ge Li, Xin Xia, Zhiyi Fu, Zhi Jin
TL;DR
CS and CG are important software-development tasks that prior neural approaches largely solved independently, despite their intuitive duality. The paper jointly trains CS and CG models with probability and attention-weight regularization, and reports improved performance on Java and Python GitHub datasets. The results support using their dual relationship as a shared training signal.
Problem
Prior neural approaches solved code summarization and code generation separately, although each task’s input is the other’s output and their relationship had not been exploited.
Method
A dual learning framework jointly trains CS and CG models using probability regularization and a novel attention-weight constraint.
Results
The dual models surpass existing state-of-the-art methods on Java and Python datasets, while dual training improves both CS performance and CG performance over independently trained models.
Takeaways & Limitations
The results support exploiting the duality between CS and CG as a shared training signal for both tasks.
Takeaways & Limitations
On Python code generation, the dual model produces a low percentage of valid code, suggesting that integrating grammar rules could improve validity.
Abstract
from arXiv · showhide
Code summarization (CS) and code generation (CG) are two crucial tasks in the field of automatic software development. Various neural network-based approaches are proposed to solve these two tasks separately. However, there exists a specific intuitive correlation between CS and CG, which have not been exploited in previous work. In this paper, we apply the relations between two tasks to improve the performance of both tasks. In other words, exploiting the duality between the two tasks, we propose a dual training framework to train the two tasks simultaneously. In this framework, we consider the dualities on probability and attention weights, and design corresponding regularization terms to constrain the duality. We evaluate our approach on two datasets collected from GitHub, and experimental results show that our dual framework can improve the performance of CS and CG tasks over baselines.
1 Introduction
CS generates comments from source code, while CG generates code from natural-language intent; both are important but costly to perform manually. Prior work treated them independently, motivating a dual framework that jointly trains both tasks using probability and attention constraints.
- Motivation: CS generates comments from source code, whereas CG generates source code from natural-language intent.Both tasks support software development but manual generation is costly, time-consuming, and error-prone.
- Prior Work: Neural encoder-decoder approaches previously addressed CS and CG as separate tasks, commonly using attention mechanisms.CS studies differed mainly in how they encoded source code, including tokens and AST structures.
- Research Gap: Previous studies had not considered or exploited relations between CS and CG to improve either task.This gap motivates treating the tasks as connected rather than independent.
- Duality: CS and CG are dual because each task’s input is the other task’s output, creating constraints for joint training.Their shared encoder-decoder structure and related language and code understanding requirements support this duality.
- Approach: The proposed framework jointly trains CS and CG while regularizing probability relationships and attention-weight similarity.The attention constraint is designed to strengthen the correspondence between the two models during training.
- Evaluation: Experiments on Java and Python projects collected from GitHub report that the jointly trained models outperform state-of-the-art models.The paper presents this joint model as an initial application of dual learning to automated CS and CG.
2 Related Work
Prior work developed neural models for CS and CG separately, including sequence, structural, and grammar-aware approaches. Dual learning provides a precedent for coupling related tasks, and this paper applies that paradigm to CS and CG with an additional attention-based constraint.
- Code Summarization: CS research commonly used encoder-decoder models with attention to generate summaries from source-code representations.Representations included tokens, serialized AST nodes, and AST structures.
- Code Generation: CG research applied neural sequence-to-sequence and grammar-based models to generate code from natural-language or structured specifications.Grammar rules and structural CNNs were introduced to address syntactic constraints and long dependencies.
- Dual Learning: Dual learning jointly trains a primal task and its dual task, with probabilistic regularization previously applied in several domains.Prior applications included machine translation, sentiment classification, and image recognition.
- Research Gap and Contribution: This work extends dual learning to CS and CG and adds a constraint on attention weights to strengthen their correlation.The paper identifies this as its first dual-learning framework for these two software-development tasks.
3 Proposed Approach
The framework jointly trains code summarization and code generation models, using probabilistic and attention-based dual constraints to connect the two tasks.
- Framework Overview: The framework contains a CS model, a CG model, and dual constraints implemented as regularization terms in the loss.CS maps source code to comments, while CG maps natural-language descriptions to code snippets.
- Probabilistic Duality: The two models are jointly trained because their conditional probabilities connect through the joint probability P(x, y).A Lagrange-multiplier penalty enforces the probabilistic duality during optimization.
- Code Summarization Model: CS uses an attention-based Seq2Seq model with a bidirectional LSTM encoder and an LSTM decoder.The encoder produces contextual token representations, and the decoder predicts comment tokens using attention and negative log-likelihood.
- Code Generation Model: CG is treated as the inverse CS task and uses the same Seq2Seq structure to predict code from natural-language descriptions.Its output layer has more parameters because source-code vocabularies usually contain more identifiers than comment vocabularies.
- Optimization: Algorithm 1 alternates parameter updates for the CS and CG models until convergence, with dual and attention penalties weighted by hyperparameters.The framework has the same complexity as a Seq2Seq neural network.
- Attention Duality: Attention duality aligns corresponding code and comment tokens by comparing CS row distributions with CG column distributions using Jensen–Shannon divergence.The resulting penalties l1 and l2 are summed into the attention regularization term, and one model’s attention weights can serve as soft labels for the other.
4 Experiments
The experiments evaluate code summarization and code generation on Java and Python datasets collected from GitHub, with additional monolingual corpora used for language-model pretraining.
- Datasets: Experiments cover CS and CG using a Java dataset and a Python dataset, with dataset statistics reported in Table 1.The Java methods and descriptions are collected from GitHub, and the Python corpus contains parallel and code-only samples.
- Java Dataset: Java samples pair methods with the first sentence of their Javadoc, which describes the method’s functionality.The Java dataset contains methods extracted from projects from 2015 to 2016.
- Pretraining: The Java language models are pretrained separately on code-only and comment-only corpora drawn from earlier GitHub projects.Java projects from 2009 to 2014 provide the monolingual pretraining data.
- Python Dataset: Python samples are split into training, test, and validation sets by 8:1:1, with code-only samples used to pretrain the source-code language model.Java comment-only data are used as an alternative pretraining corpus for Python comments because sufficient Python monolingual comments were unavailable.
- Training Configuration: The models use warm-start training: CS and CG parameters are first optimized separately with Adam, then dual constraints are applied jointly.The CS model uses 512-dimensional embeddings and LSTM states, while the CG model uses 256-dimensional LSTM states to fit GPU memory.
- Training Configuration: Language models use three LSTM layers, 300-dimensional embeddings and states, batch size 40, dropout 0.3, gradient clipping, and Adam with learning rate 0.002.Language-model parameters remain fixed during dual training, and vocabulary items must have frequency at least 3.
5 Experimental Results
Joint dual training improves both code summarization and code generation, with gains attributed to exploiting task relations through probability and attention constraints. Results also show stronger CS performance from combining regularizers, smoother dual-model attention, and improved code validity, while grammar validity remains a limitation.
- Overall Results: The dual CS model outperforms all baselines simultaneously on BLEU, METEOR, and ROUGE-L, with statistically significant and non-negligible improvements.Wilcoxon Rank Sum test p-values are all less than 0.01, and Cliff’s Delta values indicate non-negligible effect sizes.
- Overall Results: Dual training improves CG performance over the independently trained basic model and increases the percentage of valid generated code on both Java and Python.The CG BLEU scores remain low, indicating that the task is challenging on these datasets.
- Component Analysis: Combining probability and attention regularization further improves CS performance, while attention regularization alone is slightly more effective than probability regularization alone.All WRST p-values for the improvements of two constraints over one on BLEU are less than 0.05, and Cliff’s Delta shows non-negligible improvements.
- Component Analysis: On the Python dataset, joint training reduces the probability regularization value from 129.1 to 125.9 and the attention regularization value from 10.51 to 4.757.The paper interprets these reductions as evidence that the relation between the two models is enhanced after joint training.
- Qualitative Analysis and Visualization: Dual-model attention is smoother than basic-model attention for comment words not aligned with code, producing better code representations for those words.The qualitative analysis attributes this difference to the attention constraint bringing the two models’ attention distributions closer.
- Discussion on Grammar Constraints: The current dual model generates a low percentage of valid code compared with SNM, although SNM’s BLEU score is 8.095 and lower than the basic model on the Python dataset.SNM explicitly constrains grammar when generating ASTs, and all of its predictions are valid.
6 Conclusion
The paper proposes jointly training code summarization and code generation as dual tasks, using probability and attention constraints to strengthen their relationship. Experiments on Java and Python datasets show both models surpass existing state-of-the-art methods.
- The dual learning framework jointly trains code summarization and code generation models to exploit their task relationship.
- The framework constrains both probability relationships and attention mechanisms during joint training.
- Experiments on Java and Python datasets show that dual training enables both models to surpass existing state-of-the-art methods.
- The paper identifies grammar-rule integration as future work for improving generated code in the joint model.