PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep API Learning

Xiaodong Gu, Hongyu Zhang, Dongmei Zhang, Sunghun Kim

arXiv:1605.08535v3cs.SEcs.CLcs.LGcs.NE

TL;DR

Existing API-retrieval methods often use keyword or bag-of-words matching, limiting their handling of query and API sequence semantics. DeepAPI uses an RNN Encoder-Decoder to generate API sequences from natural-language queries, outperforming related approaches in evaluations on millions of annotated code snippets. Its evidence is primarily based on Java/JDK APIs and annotations derived from documentation-comment first sentences.

  • Problem

    Existing information-retrieval approaches match queries and APIs as bags of words, limiting their consideration of natural-language semantics and word/API sequences.

  • Method

    DeepAPI treats API learning as sequence translation, encoding a natural-language query into a fixed-length context vector and generating an API sequence with an RNN Encoder-Decoder.

  • Results

    54.42 average BLEU was achieved by DeepAPI, compared with 19.90 for SWIM and 11.97 for code search, on a corpus of 7 million annotated GitHub code snippets.

  • Takeaways & Limitations

    DeepAPI generates largely accurate API usage sequences and outperforms the related approaches evaluated in the paper.

  • Takeaways & Limitations

    The evaluation studies Java JDK APIs, which may not represent APIs from other libraries or programming languages.

Abstract

from arXiv · show

Developers often wonder how to implement a certain functionality (e.g., how to parse XML files) using APIs. Obtaining an API usage sequence based on an API-related natural language query is very helpful in this regard. Given a query, existing approaches utilize information retrieval models to search for matching API sequences. These approaches treat queries and APIs as bag-of-words (i.e., keyword matching or word-to-word alignment) and lack a deep understanding of the semantics of the query. We propose DeepAPI, a deep learning based approach to generate API usage sequences for a given natural language query. Instead of a bags-of-words assumption, it learns the sequence of words in a query and the sequence of associated APIs. DeepAPI adapts a neural language model named RNN Encoder-Decoder. It encodes a word sequence (user query) into a fixed-length context vector, and generates an API sequence based on the context vector. We also augment the RNN Encoder-Decoder by considering the importance of individual APIs. We empirically evaluate our approach with more than 7 million annotated code snippets collected from GitHub. The results show that our approach generates largely accurate API sequences and outperforms the related approaches.

1. INTRODUCTION

DeepAPI addresses the difficulty of discovering semantically appropriate API usage sequences from natural-language programming queries. It uses deep sequence modeling to generate API sequences and reports higher accuracy than related approaches.

  • Motivation: Developers need API subsets and invocation sequences to implement functionality, but unfamiliar libraries often document usage patterns inadequately.A Microsoft survey reported 67.6% of respondents facing obstacles from inadequate or absent API-learning resources.
  • Motivation: General and code search methods are inefficient because keyword matching overlooks natural-language semantics and forces manual examination of results.SWIM also relies on bag-of-words alignment and cannot distinguish queries such as “convert int to string” and “convert string to int.”
  • Approach: DeepAPI formulates API learning as translating a natural-language query sequence into an API sequence.It treats query words as the source sequence and APIs as the target sequence.
  • Approach: DeepAPI learns word semantics and sequence structure instead of relying on keyword matching or word-to-word alignment.The model embeds words into contextual vectors and learns associated API sequences.
  • Approach: DeepAPI adapts an RNN Encoder-Decoder that encodes annotated query sequences into fixed-length context vectors and decodes API sequences.The model is trained from annotated API-sequence pairs and used to generate sequences for API-related queries.
  • Evaluation: 54.42 average BLEU was achieved by DeepAPI, versus 19.90 for SWIM and 11.97 for code search, using 7 million annotated GitHub code snippets.On 30 real queries, the first relevant result ranked 1.6 on average; 80% of top-five and 78% of top-ten results were relevant.

2. DEEP LEARNING FOR SEQUENCE GENERATION

The paper introduces neural language modeling for sequence generation, moving from recurrent word prediction to an encoder-decoder that maps variable-length source sequences to target sequences.

  • Background: Sequence-to-sequence learning generates one sequence conditioned on another, forming the deep-learning basis adopted by this work.The paper applies these techniques to API usage-sequence generation.
  • 2.1 Language Model: A language model estimates the probability of words in a sequence from preceding words.An n-gram approximation conditions the next word only on the previous n−1 words.
  • 2.2 Neural Language Model: An RNN language model maps each word to a vector, recurrently updates a hidden state, and predicts the following word from that state.Unlike fixed-order n-grams, neural models can use preceding words at longer distances and learn distributed word representations.
  • 2.2 Neural Language Model: The RNN reads words one by one, estimating each next-word probability from the current word representation and hidden state.Training learns network parameters by minimizing prediction error.
  • 2.3 RNN Encoder-Decoder Model: The RNN Encoder-Decoder summarizes a variable-length source sequence into a fixed-length context vector, then generates a target sequence conditioned on that vector and previous outputs.An encoder RNN transforms the source sequence, while a decoder RNN generates the target sequence.
  • 2.3 RNN Encoder-Decoder Model: The encoder-decoder is trained by minimizing negative log likelihood over target words and training instances.The likelihood of each target word is conditioned on the source sequence and model parameters.

3. RNN ENCODER-DECODER MODEL FOR API LEARNING

DeepAPI applies an RNN Encoder-Decoder to translate natural-language queries into API sequences, while attention and IDF-based weighting emphasize relevant query words and important APIs.

  • RNN Encoder-Decoder: The encoder maps query words into a context representation, and the decoder generates APIs sequentially until the end-of-sequence symbol.The model translates queries such as “read text file” into API sequences.
  • Attention: Attention assigns different importance to query parts for each target API, allowing the model to focus on relevant input words.For “save file in default encoding,” “file” is more important than “default” for File.new.
  • API importance: The basic model is augmented because ubiquitous APIs such as Logger.log may not clarify a programming task’s key procedures.The augmentation targets the differing importance of APIs in programming tasks.
  • API importance: IDF-based weighting gives lower weights to ubiquitous APIs and higher weights to less common APIs.The weight uses N, the total number of API sequences, and n_yt, the number containing API y_t.
  • API importance: The API weight is incorporated as a penalty term in the model’s cost function, with λ controlling the penalty empirically.This produces the new cost function for the RNN Encoder-Decoder model.

4. DEEPAPI: DEEP LEARNING FOR API SEQUENCE GENERATION

DeepAPI combines an offline-trained RNN Encoder-Decoder with an online translation stage that ranks API sequences for natural-language queries. It builds training pairs from Java code and uses beam search to produce alternatives, with the paper’s scope limited to the JDK library.

  • 4. DEEPAPI: DEEP LEARNING FOR API SEQUENCE GENERATION: DeepAPI has offline training and online translation stages: annotated API sequences train the model, which generates ranked API sequences for user queries.The workflow is illustrated in Figure 3.
  • Scope: The paper limits its evaluation and implementation scope to APIs in the JDK library, despite stating that the approach could generate APIs in any programming language.The authors identify extension to other libraries and languages as future work.
  • 4.1.1 Extracting API Usage Sequences: API sequences are extracted by traversing method-body ASTs after project-level dependency analysis.The extraction handles constructors, JDK method calls, nested calls, statements, conditionals, and loops.
  • 4.1.2 Extracting Annotations: Method annotations use the first sentence of JavaDoc comments, while methods without comments and irregular annotations are filtered out.Non-words and bracketed words are also filtered from annotations.
  • 4.1 Data construction: The corpus contains 7,519,907 API-sequence and annotation pairs extracted from Java projects collected from GitHub.The projects were selected from 442,928 starred Java projects.
  • 4.2 Training Encoder-Decoder Language Model: The model uses bidirectional encoding and a GRU decoder, with 1000 hidden units and 120-dimensional word embeddings.Training uses minibatch Adadelta, a batch size of 200, top-10,000 vocabularies, one Nvidia K20 GPU, and approximately 240 hours.
  • 4.3 Translation: Beam search retains the n lowest-cost API branches at each time step, prunes the others, and stops branches when they reach the end-of-sequence symbol.DeepAPI outputs n sequences and ranks them by average cost, where n is the beam width.

5. EVALUATION

The evaluation measures generated API-sequence accuracy with BLEU and supplements it with human-evaluation measures. It defines the metrics and frames accuracy, parameter sensitivity, and model enhancements as its research questions.

  • Research questions: The evaluation asks how accurate DeepAPI is, how parameter settings affect accuracy, and whether enhanced RNN Encoder-Decoder models improve accuracy.These are stated as RQ1, RQ2, and RQ3.
  • BLEU: BLEU measures how closely a generated API sequence matches a reference sequence by evaluating candidate n-gram hits.The evaluation treats generated sequences as candidates and code-extracted human sequences as references.
  • BLEU: BLEU uses a brevity penalty and considers n-grams through N=4, with higher scores indicating closer agreement with the reference.A completely matching candidate receives 100% BLEU.
  • Human evaluation: FRank is the rank of the first relevant result, reflecting that users commonly scan result lists from top to bottom.The measure is used alongside relevancy ratio for human evaluation.
  • Human evaluation: Relevancy ratio is the precision of relevant results among selected results, and both human-evaluation measures range from 0 to 100.Higher values are better for both measures.

5.2 Comparison Methods

DeepAPI is compared with code search plus pattern mining and SWIM, using API-sequence retrieval approaches based on the same code corpus and BLEU evaluation.

  • Compared Approaches: The study compares DeepAPI against two state-of-the-art API learning approaches: Code Search with Pattern Mining and SWIM.
  • Code Search with Pattern Mining: Code Search with Pattern Mining combines Lucene code search with UP-Miner API usage pattern mining.Lucene indexes source code as plain text, while UP-Miner clusters extracted API sequences and mines frequent patterns.
  • Evaluation: The comparison uses the same code corpus and compares BLEU scores with DeepAPI.
  • SWIM: SWIM expands query keywords into relevant APIs using statistical word alignment, searches API sequences with Lucene, and then synthesizes code snippets.The evaluation compares DeepAPI with SWIM’s API-learning component rather than its code-synthesis component.

5.3 Accuracy (RQ1)

DeepAPI is evaluated on held-out annotated API-sequence pairs and on manually assessed unseen queries, where it achieves higher intrinsic and extrinsic accuracy than the comparison methods.

  • Intrinsic Evaluation: 7,519,907 annotated API-sequence pairs are split into 10,000 test pairs and the remaining instances for training.BLEU scores are computed on the test set, using the highest score among the top n results for each instance.
  • Intrinsic Evaluation: 54.42 BLEU is achieved by DeepAPI at top 1, versus 19.90 for SWIM and 11.97 for Code Search.The reported improvements are 173% over SWIM and 355% over Code Search; similar results occur for top 5 and top 10.
  • Extrinsic Evaluation: 30 unseen queries are used for human evaluation, including 17 existing queries and 13 longer or semantically varied queries.
  • Extrinsic Evaluation: DeepAPI achieves average FRank 1.6, average top 5 accuracy 80%, and average top 10 accuracy 78%.Two developers independently label returned sequences, reconcile disagreements, and the study applies Wilcoxon signed-rank tests.
  • Extrinsic Evaluation: DeepAPI exceeds SWIM’s average top 5 accuracy of 44% and top 10 accuracy of 47%, with statistically significant comparisons.The reported p-values are 0.01, 0.02, and 0.01; SWIM’s FRank is greater than 4.0 under the conservative treatment of unsuccessful queries.
  • Qualitative Findings: DeepAPI distinguishes reversed word sequences, handles semantically similar queries, and performs well on longer multi-keyword queries.

5.4 Accuracy Under Different Parameter Settings (RQ2)

The parameter study varies word-embedding dimensions and hidden-unit counts to assess their effects on BLEU accuracy, finding hidden units more influential than embedding size.

  • Parameter Study: The study varies word-embedding dimensions and hidden-unit counts and evaluates their impact on BLEU scores.
  • Parameter Effects: Word-embedding dimension makes little difference to DeepAPI’s accuracy.
  • Parameter Effects: DeepAPI’s accuracy depends greatly on the number of hidden units, with an optimum around 1000 hidden units.

5.5 Performance of the Enhanced RNN Encoder-Decoder Models (RQ3)

The study evaluates attention-based and new-cost-function enhancements to the RNN Encoder-Decoder, finding improvements over the basic and attention-based models respectively.

  • Attention Enhancement: The attention-based RNN Encoder-Decoder outperforms the basic model, improving top 1, top 5, and top 10 BLEU by 8%, 5%, and 4%.
  • Cost-Function Enhancement: The new cost-function model improves over the attention-based model by 4%, 2%, and 1% on top 1, top 5, and top 10 BLEU.
  • Cost-Function Enhancement: The enhanced cost-function model has an optimum λ around 0.035 under different parameter settings.

6. DISCUSSION

DeepAPI’s discussion explains its semantic representation, sequence modeling, and trade-off between common and project-specific API usage patterns. It also identifies Java/JDK coverage and annotation quality as validity constraints.

  • Why does DeepAPI work?: DeepAPI embeds semantically similar queries near one another in a continuous space, as illustrated by the t-SNE projection of file-related queries.The projection uses test-set annotations containing “file” and excludes queries longer than eight words.
  • Why does DeepAPI work?: DeepAPI tends to generate common API sequences, whereas information-retrieval approaches can return project-specific sequences.The distinction reflects neural modeling of frequent sequences versus retrieval of individual instances.
  • Why does DeepAPI work?: Query expansion and frequent pattern mining only partially address the limitations of existing approaches.Inappropriate synonym expansion can produce worse results than the original query, and few techniques provide all the identified advantages.
  • Threats to Validity: The evaluation is limited because all studied APIs and related projects are Java APIs from the JDK.The authors plan to extend the model to other libraries and programming languages.
  • Threats to Validity: Annotation quality is constrained by extracting only first sentences from documentation comments, which may omit informative content or contain noise.The authors identify improved natural-language processing for annotation extraction as future work.

7. RELATED WORK

Related work includes code search, API usage-pattern mining, query-to-code generation, and deep learning for source code. DeepAPI differs by generating API sequences without information retrieval and by modeling word and API sequences.

  • Code Search: Code search tools retrieve relevant functions, API usages, or code by matching queries with APIs and documentation.Examples include Portfolio, text-phrase API usage search, and CodeHow’s extended Boolean model and API matching.
  • Mining API Usage Patterns: API usage-pattern miners represent source code as call sequences, cluster similar sequences, and mine frequent API method-call patterns.MAPO and UP-Miner are described as representative approaches in this line of work.
  • Mining API Usage Patterns: Pattern-mining techniques help understand API usage but do not answer which APIs to use, whereas DeepAPI learns usage patterns with a neural language model.This distinction separates mining existing patterns from generating API sequences for a natural-language query.
  • Code Generation from Natural Language: SWIM translates natural-language queries into APIs using Bing search logs and synthesizes idiomatic code, but its API-sequence component differs from DeepAPI.The supplied passage introduces SWIM’s query-to-API and code-synthesis pipeline without detailing all differences.
  • Deep Learning for Source Code: Deep learning has been applied to source-code feature extraction, vector representations, and tree-structured programming-language processing.The passage presents these applications as related work preceding DeepAPI’s API-sequence generation.

8. CONCLUSION

The paper concludes that an RNN Encoder-Decoder can generate API usage sequences for natural-language queries and reports effectiveness in API sequence generation. It identifies broader software-engineering applications and sample-code synthesis as future work.

  • CONCLUSION: DeepAPI applies an RNN Encoder-Decoder to generate API usage sequences from API-related natural-language queries.The conclusion presents this as the paper’s deep-learning approach to API learning.
  • CONCLUSION: The empirical study reports that the proposed approach is effective for API sequence generation.The conclusion characterizes this as an observed effectiveness of deep learning in API learning.
  • CONCLUSION: The model may benefit software-engineering problems such as code search and bug localization, which the authors leave for future exploration.The stated future work also includes synthesizing sample code from generated API sequences.
  • CONCLUSION: An online DeepAPI demo is available on the authors’ website.The conclusion-related materials provide the demo URL.
Loading 1605.08535v3…