PaperlayerView exact revision on arXiv

Source-linked AI summary

Query Expansion Based on Crowd Knowledge for Code Search

Liming Nie, He Jiang, Zhilei Ren, Zeyi Sun, Xiaochen Li

arXiv:1703.01443v1cs.SE

TL;DR

Code search is hindered by vocabulary mismatch between short developer queries and code snippets. The paper proposes QECK, which extracts software-specific terms from high-quality Stack Overflow Q&A feedback, and integrates it with Rocchio as QECKRocchio. QECK improves three code search algorithms by up to 64% in Precision and 35% in NDCG, while QECKRocchio improves over PWordNet by 22% in Precision and 16% in NDCG.

  • Problem

    Vocabulary mismatch between short developer queries and code snippets limits text-retrieval-based code search effectiveness.

  • Method

    QECK retrieves high-quality Stack Overflow Q&A pairs as pseudo-relevance feedback, selects software-specific expansion words, and integrates with Rocchio as QECKRocchio.

  • Results

    64% is the maximum Precision improvement and 35% the maximum NDCG improvement across three algorithms; QECKRocchio improves over PWordNet by 22% in Precision and 16% in NDCG.

  • Takeaways & Limitations

    The reported results support using software-specific words from Stack Overflow crowd knowledge for code search query expansion.

  • Takeaways & Limitations

    The evaluation uses only 20 queries, and title-only queries may not represent actual developer queries.

Abstract

from arXiv · show

As code search is a frequent developer activity in software development practices, improving the performance of code search is a critical task. In the text retrieval based search techniques employed in the code search, the term mismatch problem is a critical language issue for retrieval effectiveness. By reformulating the queries, query expansion provides effective ways to solve the term mismatch problem. In this paper, we propose Query Expansion based on Crowd Knowledge (QECK), a novel technique to improve the performance of code search algorithms. QECK identifies software-specific expansion words from the high quality pseudo relevance feedback question and answer pairs on Stack Overflow to automatically generate the expansion queries. Furthermore, we incorporate QECK in the classic Rocchio's model, and propose QECK based code search method QECKRocchio. We conduct three experiments to evaluate our QECK technique and investigate QECKRocchio in a large-scale corpus containing real-world code snippets and a question and answer pair collection. The results show that QECK improves the performance of three code search algorithms by up to 64 percent in Precision, and 35 percent in NDCG. Meanwhile, compared with the state-of-the-art query expansion method, the improvement of QECK Rocchio is 22 percent in Precision, and 16 percent in NDCG.

1 INTRODUCTION

The paper targets vocabulary mismatch in code search with QECK, which uses Stack Overflow crowd knowledge to expand queries automatically. Experiments report improved retrieval across three algorithms and stronger performance than PWordNet.

  • Motivation: Code search suffers from vocabulary mismatch because short developer queries and code snippets often use different words.The average query length reported for code search is 1.85 words, making effective query formulation difficult.
  • QECK: QECK retrieves relevant Stack Overflow Q&A pairs as pseudo-relevance-feedback documents and extracts software-specific expansion words.It combines textual similarity and question-and-answer quality, then uses TF-IDF to identify expansion words.
  • QECK: QECK automatically generates expansion queries without human intervention while using high-quality Q&A pairs selected from crowd knowledge.The approach differs from general lexical resources by exploiting software-development knowledge in Stack Overflow.
  • Experiments: 64% is the maximum Precision improvement and 35% the maximum NDCG improvement across the three code search algorithms.These results compare recommendation performance before and after applying QECK.
  • Experiments: 22% Precision and 16% NDCG are the reported improvements of QECKRocchio over the state-of-the-art PWordNet method.QECKRocchio incorporates QECK into the classic Rocchio model.
  • Contributions: The paper contributes QECK, evaluations with three code search algorithms, and collections of Stack Overflow Q&A pairs and open-source Android code snippets.The code corpus contains snippets from open-source app projects, while the Q&A collection supplies crowd knowledge.

2 BACKGROUND

The background distinguishes global thesaurus-based expansion from local feedback-based expansion. The paper focuses on a PRF variant that uses Stack Overflow Q&A pairs and their crowd-assigned scores to derive software-specific terms.

  • Query expansion: Query expansion approaches are divided into global methods and local methods.Global methods use external lexical resources, whereas local methods use documents initially matching the query.
  • Global approaches: Global approaches automatically add related words or synonyms from thesauri such as WordNet.WordNet is described as a general-purpose lexical database for computing semantic distance.
  • Global approaches: Software-specific similarity resources address shortcomings of general English resources but may omit the query’s whole context.Examples use source code, method comments, signatures, or Stack Overflow posts.
  • Local approaches: Local approaches expand queries from initially retrieved documents through relevance feedback or pseudo-relevance feedback.PRF automates the manual feedback step by treating initially retrieved documents as relevant.
  • QECK: QECK uses Stack Overflow Q&A pairs rather than code snippets as its initial PRF results and extracts expansion words using crowd-voted post scores.This design combines software-development content with crowd-based quality information.

3 OUR TECHNIQUE

QECKRocchio combines Q&A retrieval, expansion-word selection, and code-snippet retrieval. It ranks high-quality relevant Stack Overflow pairs, selects discriminative terms, and searches an Android code corpus with the expanded query.

  • 3.1 Steps of QECKRocchio: QECKRocchio has three modules: QA Pairs Search Engine, Expansion Words selector, and Code Snippets Search Engine.Its inputs are an original query, a Q&A collection, and a code corpus; its output is a ranked top-k list.
  • 3.1 Steps of QECKRocchio: First-pass retrieval ranks Q&A pairs for the original query and selects the top-m pairs as pseudo-relevance-feedback documents.The paper gives BM25 as an example retrieval model.
  • 3.2.1 Q&A pairs: Q&A pairs are built from Android-tagged Stack Overflow questions combined with an answer, with accepted answers used in the illustrated construction.The example combines the question title and description with the accepted answer.
  • 3.2.1 Q&A pairs: Q&A text is split at camel case and separators, stop words are removed, and the remaining terms are stemmed before indexing.Each indexed document contains the Q&A pair ID and processed words.
  • 3.2.3 Searching Q&A Pairs: Q&A-pair quality combines BM25-based textual similarity with SO scores after min-max normalization.SO score is a weighted mean of the question and answer’s crowd-voted scores.
  • 3.3 Words Selection: Candidate expansion words receive TF-IDF weights, and the top-n words are selected after removing terms appearing in more than 25% of feedback documents.TF-IDF uses term frequency and inverse document frequency to estimate word importance.
  • 3.1 Steps of QECKRocchio: The final QECKRocchio retrieval scores code snippets with BM25 using the expanded query and returns the top-k snippets.The corpus contains 921,713 snippets from 1,538 open-source Android app projects.

4 EXPERIMENTAL SETUP

The evaluation uses three experiments addressing QECK effectiveness, parameter effects, and QECKRocchio’s comparison with a state-of-the-art method. The experiments were implemented on a desktop Java environment.

  • Experimental design: Three experiments correspond to three research questions about QECK effectiveness, parameter influence, and QECKRocchio performance.The setup section states that the experiments also provide dataset and evaluation details.
  • Implementation: The experiments were conducted on a 3.60 GHz Intel i5 PC running Windows 8.1 with 8G memory.QECKRocchio was implemented in Java 1.7.0 using Eclipse.

4.1 Research Questions

The study asks whether QECK improves code-search algorithms, how its parameters affect performance, and whether QECKRocchio outperforms the state-of-the-art PWordNet method.

  • RQ1: The first experiment compares three code-search algorithms before and after applying QECK.The algorithms are IR, Portfolio, and VF.
  • RQ2: QECK performance is studied through two parameters: the number of PRF documents and the number of expansion words.These correspond to feedback Q&A-pair quality and selected expansion terms.
  • RQ2: The parameter study fixes one parameter while varying the other to identify performance trends and optimal settings for each algorithm.The procedure first varies expansion words with five PRF documents, then varies PRF documents after fixing the optimal expansion-word value.
  • RQ3: The third experiment compares QECKRocchio with PWordNet under the same retrieval scenario.Parameters for both approaches are adjusted to achieve their best performance.

4.2 Dataset Collection

The experiments use Android programming tasks, Stack Overflow Q&A data, and a large corpus of Java code snippets collected from real-world projects.

  • Query set: The query set contains 20 real-world Android programming tasks collected from Stack Overflow.The tasks were selected from highly viewed posts and include solutions for judging code-snippet relevance.
  • Query set: Queries are formed by extracting words from each task’s title.Descriptions may contain text and code context, but the experiments use title words as queries.
  • Q&A collection: The Stack Overflow dump yields 5,108,770 Q&A pairs, including 312,941 pairs labeled with the android tag.The dump is the August 2015 public data release.
  • Code corpus: The code corpus contains 921,713 snippets constructed by segmenting Java files from 1,538 Android projects collected from F-Droid.The projects were collected through August 2015.

4.3 Evaluation

Evaluation combines pooled top-10 code-search results, expert relevance judgments, and Precision and NDCG metrics, with statistical comparisons reported for the algorithms.

  • Relevance labeling: The evaluation pools each algorithm’s top-10 results for every query and has two participants assign four-level relevance scores to unique snippets.Relevance judgments are based on whether snippets help solve the programming task.
  • Relevance labeling: Scores 3 and 4 indicate relevant snippets, while scores 1 and 2 indicate irrelevant snippets.Score 4 means highly relevant; score 3 means mostly relevant and reusable with changes.
  • Relevance labeling: Two participants label 3,251 code snippets, agreeing on 2,756 cases for an 84.8% consensus rate.The participants are experienced Java developers with Android development experience.
  • Metrics: Precision measures the proportion of relevant snippets among retrieved results, using 10 as the study’s retrieval count.Relevant snippets are those assigned scores 3 or 4.
  • Metrics: NDCG measures ranking quality by rewarding relevant results appearing higher in the hit list.The study normalizes DCG by ideal DCG and converts scores 1 and 2 to zero before calculating NDCG.

5 RESULTS AND ANALYSIS

The experiments evaluate QECK’s retrieval effectiveness, parameter sensitivity, and QECKRocchio against PWordNet. QECK improves all three code search algorithms, while QECKRocchio outperforms the comparison method.

  • 5.1 RQ1: 64% is the largest Precision improvement, achieved by VF; IR and Portfolio improve by 38% and 33%, respectively.These are improvements in the mean value of Precision after applying QECK.
  • 5.1 RQ1: 35% is the largest NDCG improvement, achieved by VF; IR and Portfolio improve by 20% and 16%, respectively.These are improvements in the mean value of NDCG after applying QECK.
  • 5.1 RQ1: QECK improves the retrieval performance of all three evaluated code search algorithms.The first experiment compares each algorithm before and after QECK, with statistically significant differences reported for Precision and NDCG.
  • 5.2 RQ2: QECK performance generally rises initially as either the number of PRF documents or expansion words increases, then reaches a unique optimum.Too many or too few expansion words are undesirable, and the experiments vary one parameter while fixing the other.
  • 5.3 RQ3: QECKRocchio improves over PWordNet by 22% in Precision and 16% in NDCG, with statistically significant pairwise differences.The reported p-values are below 0.05 for both metrics.

6 THREATS TO VALIDITY

The study identifies threats involving query representativeness, dataset scope, parameter exploration, labeling, comparative methods, and recommendation efficiency. These constraints limit how broadly and confidently the reported code-search improvements can be interpreted.

  • Labeling: Different programming levels among the two labelers could produce inconsistent relevance scores, although an expert arbitrated disagreements.The labeling process involved participants with Android and Java development experience.
  • The query set: 20 queries may not represent actual developer queries because they were extracted only from programming-task titles.The authors plan to use complete task titles and descriptions in future work.
  • The Q&A collection and the Code snippet corpus: Android-focused Q&A and code-snippet datasets are smaller and narrower than other datasets, limiting generalizability.The authors plan to evaluate QECK on larger datasets.
  • The Comparative methods: Reproduced comparative methods differed from their original implementations, potentially because the queries and code corpora were Android-specific.The study compares reproduced algorithms and a comparative method under Android-focused conditions.
  • Parameters: Equal weighting of original and expansion words may overlook performance differences produced by alternative term weights.The experiments varied one parameter at a time and did not automatically tune both parameters and word weights.
  • Efficiency: Average recommendation time was under three seconds, but efficiency remains a practical threat because it depends on code quality, data size, and algorithm.The paper primarily evaluates effectiveness using Precision and NDCG rather than efficiency.

7 RELATED WORK

Prior work covers code search with varied input types, global and local query expansion, and Stack Overflow-based developer support. QECK differs by using software-specific words from Stack Overflow Q&A pairs to expand free-form code-search queries automatically.

  • Code search and query expansion: Code-search studies use test cases, free-form queries, API method names, class-type pairs, desired-code examples, and other inputs.Free-form queries are represented by several words and retrieve ranked code-snippet lists.
  • Code search and query expansion: Global query expansion uses thesauri such as WordNet, whereas local approaches use initially retrieved documents and relevance feedback.Local methods may use user feedback or automatically extracted expansion words.
  • Code search based on query expansion: Existing code-search expansion methods include user-feedback refinement, co-occurrence-based alternatives, and WordNet-based expansion.These approaches differ in automation, input type, and expansion resources.
  • Code search based on query expansion: QECK automatically extracts software-specific expansion words from pseudo-relevance-feedback Q&A pairs on Stack Overflow for free-form queries.This distinguishes it from approaches using test cases or general thesaurus terms.
  • Q&A pairs recommendation on Stack Overflow: Stack Overflow-based tools such as Seahawk and Prompter retrieve discussions for software comprehension using code- or IDE-derived queries.Seahawk uses textual similarity, while Prompter combines several retrieval aspects.

8 CONCLUSION AND FUTURE WORK

The paper concludes that QECK improves code search by addressing the vocabulary problem with software-specific words from Stack Overflow. Future work targets query-feature analysis and better automatic expansion through improved weighting, efficiency, and parameter-selection techniques.

  • Conclusion: QECK improves three code-search algorithms by up to 64% in Precision and 35% in NDCG on Android-platform datasets.The conclusion presents these results as evidence of QECK’s effectiveness for mobile app development.
  • Conclusion: QECKRocchio improves Precision by 22% and NDCG by 16% over the state-of-the-art query expansion method.The comparison supports the effectiveness of software-specific words in QECK.
  • Future work: Future work will analyze query features and recommend reformulation strategies for individual queries.The proposed directions include automatically assessing query performance and selecting reformulation strategies.
  • Future work: Future work will improve automatic query expansion through term-saliency modeling, pseudo-relevance-feedback efficiency, and expanded parameter or weight adjustment.These directions address candidate-term weighting, retrieval efficiency, and optimization of expansion settings.
Loading 1703.01443v1…