Source-linked AI summary
RACK: Automatic API Recommendation using Crowdsourced Knowledge
Mohammad Masudur Rahman, Chanchal K. Roy, David Lo
TL;DR
Natural-language code-search queries often lack the API vocabulary required by keyword-based engines, making effective query preparation difficult. RACK addresses this gap by learning keyword–API associations from Stack Overflow and recommending ranked API classes; on 150 queries, it achieved about 79% Top-10 accuracy and outperformed two state-of-the-art variants across reported ranking metrics.
Problem
Traditional code-search and API-recommendation techniques rely on lexical similarity, making effective API-aware query formulation difficult for developers unfamiliar with target APIs.
Method
RACK learns keyword–API associations from Stack Overflow questions and accepted answers, then translates natural-language code-search queries into ranked API classes.
Results
79% Top-10 accuracy was achieved on 150 code-search queries, while RACK outperformed two state-of-the-art variants in Top-K accuracy, mean average precision, and mean recall.
Takeaways & Limitations
Stack Overflow provides crowdsourced associations that RACK uses to recommend relevant APIs for natural-language code-search queries.
Takeaways & Limitations
The technique might be effective only for popular and frequently used APIs, and fully qualified names are often missing for third-party APIs.
Abstract
from arXiv · showhide
Traditional code search engines often do not perform well with natural language queries since they mostly apply keyword matching. These engines thus need carefully designed queries containing information about programming APIs for code search. Unfortunately, existing studies suggest that preparing an effective code search query is both challenging and time consuming for the developers. In this paper, we propose a novel API recommendation technique--RACK that recommends a list of relevant APIs for a natural language query for code search by exploiting keyword-API associations from the crowdsourced knowledge of Stack Overflow. We first motivate our technique using an exploratory study with 11 core Java packages and 344K Java posts from Stack Overflow. Experiments using 150 code search queries randomly chosen from three Java tutorial sites show that our technique recommends correct API classes within the top 10 results for about 79% of the queries which is highly promising. Comparison with two variants of the state-of-the-art technique also shows that RACK outperforms both of them not only in Top-K accuracy but also in mean average precision and mean recall by a large margin.
I. INTRODUCTION
RACK addresses the difficulty of formulating API-aware code-search queries by using Stack Overflow’s crowdsourced keyword–API associations to translate natural-language queries into ranked API recommendations. Its evaluation reports promising accuracy and substantial gains over two state-of-the-art variants.
- Motivation: Traditional code-search engines rely on lexical similarity and therefore often require carefully prepared queries containing relevant API classes or methods.Existing API-recommendation techniques likewise rely on lexical similarity, requiring developers to know target-API vocabulary.
- Approach: RACK exploits Stack Overflow question–answer associations to translate natural-language code-search queries into ranked lists of relevant API classes.Question keywords represent programming requirements, while APIs extracted from accepted answers represent programming solutions.
- Evidence: 344,086 Java-related Stack Overflow posts indicate that each post uses at least two API classes on average and covers about 65% of classes in each of 11 core Java packages.
- Evaluation: 79% Top-10 accuracy was achieved on 150 code-search queries, and RACK outperformed two state-of-the-art variants in Top-K accuracy, mean average precision, and mean recall.
- Approach: The study constructs a keyword–API mapping database linking 655K question keywords to 551K API classes from Stack Overflow.
II. EXPLORATORY STUDY
The exploratory study tests whether Stack Overflow contains sufficient API and keyword evidence for natural-language API recommendation. It examines accepted answers, core Java-package coverage, question-title keywords, and API extraction across collected Java data.
- Research questions: The study investigates whether accepted Stack Overflow answers refer to standard Java API classes and whether core-package classes are covered.
- Research questions: It also tests whether Stack Overflow question titles contain potential code-search keywords.
- Data collection: 172,043 Java-tagged questions with accepted answers were collected, alongside 2,912 classes from 11 core Java API packages.
- Analysis: API-class usage frequencies in accepted answers were analyzed with probability mass and cumulative density functions.
- Analysis: API elements were extracted from Stack Overflow posts using regular expressions adapted from island-parsing techniques.
C. Answering RQ1: API use in accepted SO answers
Accepted Stack Overflow answers frequently use Java API classes, including classes from the core packages, supporting their use as a source for API recommendation.
- 79.51% of 172,043 accepted answers refer to one or more Java classes.
- Core-package API classes are referred to at least five times on average in each relevant accepted answer.
- At least two distinct core-package API classes are used on average in each accepted answer containing such classes.
- About 65.15% of API classes from each core Java package appear in Stack Overflow answers on average.
- At least 12.22% of answers refer to classes from each individual core API package.
E. Answering RQ3: Search keywords in SO questions
Stack Overflow question titles contain many keywords used in real-world code-search queries, making them a potential source for linking search terms to relevant APIs.
- 1,703 code-search queries contain 1,461 distinct natural-language tokens after selecting queries intended for code-example search.
- Question-title tokens match 66.94% of keywords collected from the code-search queries.
- 73.03% of code-search keywords from each year match Stack Overflow tokens on average, reaching 80% from 2009 to 2011.
- Each Stack Overflow question title contains approximately three code-search keywords on average.
- The technique extracts title tokens and accepted-answer API classes to create semantic token-API associations.
B. API Relevance Ranking & Recommendation
RACK selects candidate APIs by combining keyword–API associations with coherence among query keywords. It then uses these signals to identify APIs that are both likely for individual keywords and functionally consistent across the query.
- Keyword-Keyword Coherence: RACK estimates keyword coherence from co-occurring context words in Stack Overflow question titles.For each keyword pair, it derives context lists, computes semantic similarity, and identifies coherent pairs.
- Keyword-Keyword Coherence: Candidate APIs are selected when they are relevant to both keywords and functionally consistent with coherent keyword pairs.The method intersects APIs associated with each keyword and applies a context-similarity threshold γ.
- Keyword-API Co-occurrence: RACK exploits keyword–API associations from Stack Overflow to identify APIs linked to query keywords.The method captures associations from crowdsourced programming discussions and uses them for candidate selection.
- Keyword-API Co-occurrence: The method retains the top five APIs for each keyword according to association frequency, using δ = 5 as a heuristic threshold.The threshold is selected through iterative experiments on the dataset.
- Algorithm overview: Algorithm 1 organizes RACK around query keyword collection, candidate API collection, relevance estimation, score calculation, and API ranking.The listed stages define the pipeline used to produce recommendations.
3) API Relevance Ranking Algorithm:
RACK preprocesses a natural-language query, gathers candidates using keyword association and coherence heuristics, scores their keyword likelihood and multi-keyword relevance, and returns the highest-ranked APIs.
- Query processing: RACK extracts meaningful query terms with POS tagging and standard natural-language processing before applying its two candidate-selection heuristics.For example, “html parser in Java” becomes the keywords ‘html’, ‘parser’, and ‘java’.
- Scoring: API Likelihood scores how strongly a candidate API co-occurs with an associated query keyword, with values ranging from 0 to 1.The score is based on the API’s rank in the keyword–API co-occurrence list.
- Scoring: API Coherence estimates a candidate API’s relevance to multiple query keywords using semantic similarity between their Stack Overflow-derived context lists.The estimate propagates keyword similarity to APIs that co-occur with both keywords.
- Ranking and recommendation: RACK accumulates API Likelihood and API Coherence scores, ranks candidates by the accumulated scores, and returns the top K APIs.The ranking and recommendation stages produce the final API list.
- Worked example: In the worked example, RACK first collects the top 5 APIs for ‘java’, ‘parser’, and ‘html’, then computes likelihood and coherence for the candidates.The example illustrates the scoring process for the query keywords.
IV. EXPERIMENT
The experiment evaluates RACK’s API recommendations across four research questions covering overall performance, the two heuristics, keyword selection, and comparison with a state-of-the-art technique.
- Research questions: The study evaluates RACK on the relevance of recommended APIs for code search queries.This is the central performance question addressed by the experiment.
- Research questions: The experiment tests how effectively KAC and KKC capture relevant APIs for a query.These are the two proposed heuristics examined in the evaluation.
- Research questions: The study examines whether RACK’s query-keyword selection effectively retrieves relevant APIs.This isolates the contribution of keyword selection from the broader recommendation process.
- Research questions: The evaluation compares whether RACK outperforms a state-of-the-art technique for recommending APIs across query sets.The comparison is framed as a separate research question.
A. Experimental Dataset
The experimental dataset contains 150 natural-language code-search queries drawn from Java tutorial sites, with relevant APIs identified from the tutorials’ explanatory prose and code examples.
- Query collection: The dataset contains 150 code-search queries collected from KodeJava, JavaDB, and Java2s.The sites cover programming tasks involving APIs from standard Java libraries.
- Query collection: Each tutorial task provides a title, one or more code snippets, and prose explaining the code.The title summarizes the programming task in natural language and serves as the search query.
- Query collection: The study uses tutorial titles as code-search queries because they summarize programming tasks with a few natural-language keywords.An example title is “How do I decompress a GZip file in Java?”
- Gold-set development: Relevant APIs are collected from explanatory prose associated with the tutorial code snippets to construct the API-goldset.These APIs are treated as essential for the corresponding programming tasks.
B. Performance Metrics
RACK is evaluated with four metrics spanning recommendation and information retrieval: Top-K Accuracy, MRR@K, MAP@K, and Mean Recall@K.
- Top-K Accuracy: Top-K Accuracy measures the percentage of queries with at least one correctly recommended API within the Top-K results.isCorrect is 1 when a Top-K list contains an API from the API-gold set and 0 otherwise.
- Ranking Metrics: MRR@K averages the reciprocal rank of the first relevant API across all search queries.Reciprocal rank@K is the multiplicative inverse of that API's rank within the Top-K results.
- Information Retrieval Metrics: MAP@K averages query-level Average Precision@K, which averages precision at every relevant API in each ranked list.The metric aggregates precision across relevant APIs and then across queries.
- Information Retrieval Metrics: Mean Recall@K averages the percentage of gold-set APIs correctly recommended within Top-K results across queries.Recall is computed per query before averaging over the dataset.
C. Evaluation of RACK
RACK is evaluated on Java code-search queries using Top-K recommendation against API gold sets, with additional analyses of heuristics and query-term selection. It reaches about 79% Top-10 accuracy, 34.92% mean average precision, and 45.02% mean recall, while combining its heuristics performs best.
- Overall Performance: At K = 10, accuracy, precision, and recall become stationary, supporting the choice of Top-10 result collection.Mean reciprocal rank does not change across the evaluated Top-K recommendations.
- Overall Performance: 79% Top-10 accuracy, 34.92% mean average precision, and 45.02% mean recall are reported for RACK.For Top-5 results, accuracy is 63.00% and precision and recall are close to 33.50%.
- Heuristics: 79.00% accuracy, 34.92% precision, and 45.02% recall are achieved when KAC and KKC are combined for Top-10 results.KAC alone reaches 78.00% accuracy, while KKC alone provides at most 40.00% accuracy.
- Query Term Selection: Important query keywords mainly consist of noun and verb terms, and the paper finds this selection effective for retrieving relevant APIs.The analysis reports weaker performance for some Top-3 and Top-5 settings under alternative query-term selections.
D. Comparison with Existing Techniques
RACK substantially outperforms two variants of a state-of-the-art API recommendation technique. The comparison covers Top-K accuracy, precision, and recall across different K values.
- Comparison Setup: The comparison uses two variants of Thung et al.'s technique as baselines for API recommendation.The variants are based on textual similarity and use API documentation collected for 3,300 Java classes.
- Results: 79% Top-10 accuracy, 34.92% precision, and 45.02% recall for RACK exceed the variants' maximum 49% accuracy, 23.47% precision, and 33.44% recall.These results are reported for Top-10 recommendations.
- Results: RACK's accuracy rises gradually to 79% as K increases, whereas the textual-similarity techniques stop at 49%.RACK also performs significantly better on accuracy, precision, and recall across the comparison.
V. THREATS TO VALIDITY
The evaluation faces threats from subjective API gold-set construction, possible confusion between third-party and Java APIs, and limited testing on standard Java libraries. The authors also discuss metric suitability and related-work scope.
- Internal Validity: API gold sets derived from tutorial code and discussions may involve subjectivity, although the examples are working solutions frequently consulted by developers.The authors contrast this approach with human judgments of API relevance, which they characterize as more subjective.
- Internal Validity: Missing fully qualified names in Stack Overflow text may cause third-party APIs resembling Java API classes to be mistakenly considered.The exploratory findings also suggest the technique may be effective mainly for popular and frequently used APIs.
- External Validity: The technique has been evaluated only with API classes from standard Java libraries, limiting demonstrated generalizability across programming domains.The authors state that its keyword-API co-occurrence basis could be adapted to other domains.
- Construct Validity: The authors identify little or no construct-validity threat because the selected metrics are widely used in recommendation and information-retrieval studies.Top-K Accuracy and Reciprocal Rank represent recommendation metrics, while the remaining metrics come from information retrieval.