Source-linked AI summary
CoSQA: 20,000+ Web Queries for Code Search and Question Answering
Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, Nan Duan
TL;DR
Real-user web queries are underrepresented in existing resources for learning semantic query-code matching. The paper introduces CoSQA and CoCLR, and reports improved code question answering and code search, while its dataset currently focuses on Python.
Problem
Existing resources lack enough real user web queries to support statistical models for semantic matching between natural-language queries and code.
Method
CoSQA provides 20,604 human-annotated query-code pairs, while CoCLR uses code contrastive learning to generate additional training instances.
Results
Training CodeBERT on CoSQA improves code question answering by 5.1%, and incorporating CoCLR adds a further 10.5% improvement; code search shows similar results.
Takeaways & Limitations
CoSQA supports code question answering and code search, and CoCLR improves baseline performance on both tasks.
Takeaways & Limitations
The dataset studies Python and is planned for future extension to more programming languages.
Abstract
from arXiv · showhide
Finding codes given natural language query isb eneficial to the productivity of software developers. Future progress towards better semantic matching between query and code requires richer supervised training resources. To remedy this, we introduce the CoSQA dataset.It includes 20,604 labels for pairs of natural language queries and codes, each annotated by at least 3 human annotators. We further introduce a contrastive learning method dubbed CoCLR to enhance query-code matching, which works as a data augmenter to bring more artificially generated training instances. We show that evaluated on CodeXGLUE with the same CodeBERT model, training on CoSQA improves the accuracy of code question answering by 5.1%, and incorporating CoCLR brings a further improvement of 10.5%.
1 Introduction
Natural-language code search depends on matching real user queries with code, but existing resources contain too few authentic queries for training semantic-matching models. CoSQA addresses this gap with human-annotated query-code pairs and CoCLR augmentation, improving code question answering and code search.
- Natural-language code search aims to retrieve semantically relevant code, making semantic similarity between queries and code a central challenge.
- Existing datasets largely use pseudo queries, while real-query resources provide only 34, 99, and 1,046 test queries.
- CoSQA contains 20,604 web-query and code pairs labeled according to whether the code answers each query.
- The dataset uses Microsoft Bing search logs, GitHub functions, and at least three crowdsourcing annotations per candidate pair.
- CoCLR generates artificially augmented training instances through code contrastive learning for query-code matching.
- 5.1% improvement in code question answering follows CoSQA training with CodeBERT, with a further 10.5% boost from CoCLR; code search shows similar results.
2 Related Work
Related work spans datasets for code search and question answering and two principal model families: information-retrieval methods and deep-learning methods. Existing resources range from large text-code collections to smaller high-quality evaluation sets, while this work focuses on Python.
- Prior datasets support code search and code question answering, including large text-code collections and smaller curated testing sets.
- CoSQA studies Python and plans future extension to additional programming languages.
- Information-retrieval models match query keywords with code sequences, while deep-learning models encode queries and code into vectors for similarity-based retrieval.
- Some approaches improve code-search representations by exploiting code structures.
3 CoSQA Dataset
CoSQA constructs a professionally annotated Python query-code dataset through web-query curation, code selection, candidate filtering, and multi-stage human annotation. The resulting dataset contains 20,604 retained labels with average Krippendorff’s alpha of 0.63.
- Data construction: CoSQA pairs natural-language web queries with complete Python functions and labels whether each function can answer its query.Queries are sourced from Microsoft Bing search logs, while code functions come from the CodeSearchNet Corpus.
- Query curation: 82.40 accuracy and 67.65 F1 were achieved by the keyword-based query-filtering algorithm on a human-annotated intent test set.The test set contains 250 randomly sampled web queries labeled for code-search intent.
- Candidate selection: Candidate pairs are prefiltered with a CodeBERT matching model, retaining the highest-similarity code and removing pairs below similarity 0.5.The candidate process also seeks greater code diversity and controls code frequency.
- Annotation: Annotators first judge code-search intent, then assess whether the code answers the query using its documentation, function header, and body.Answer labels are binary: “1” indicates a correct answer and “0” otherwise.
- Annotation: The dataset’s answer guidelines treat code that fully answers, exceeds, or satisfies a query category as correct, while code satisfying no more than 50% is incorrect.Table 3 provides examples and explanations for these query-code judgments.
- Annotation: 20,604 labels were retained after multi-annotator review, with average Krippendorff’s alpha of 0.63.Each query-code pair was annotated by at least three participants, and pairs with poor agreement or no code-search intent were removed.
4 Tasks
CoSQA supports two query-code matching tasks: retrieving the most likely code from a collection and classifying whether a code answers a query. They use MRR and accuracy, respectively.
- Code search: Natural-language code search is formulated as text retrieval, selecting the most likely code answer from a collection and evaluating it with Mean Reciprocal Rank.The input consists of a query q_i and a code collection C = {c_1, . . . , c_H}.
- Code question answering: Code question answering is formulated as binary classification, predicting whether a code answers a query and evaluating predictions with accuracy.The model outputs label “1” for an answering code and “0” otherwise.
5 Methodology
The methodology combines a CodeBERT-based siamese query-code matcher with CoCLR, which augments training using in-batch and query-rewritten examples. The resulting objective trains the model to distinguish matching and non-matching query-code pairs while adding contrastive signals.
- 5.1 Siamese Network with CodeBERT: The method uses a siamese network with identical subnetworks and shared parameters to model relationships between query and code representations.CodeBERT encodes the query and code separately, producing contextualized pooled representations.
- 5.1 Siamese Network with CodeBERT: Query-code matching combines the query and code embeddings, their element-wise difference, and their element-wise product before classification.A feedforward network produces a relation embedding, and a sigmoid score represents query-code similarity.
- 5.1 Siamese Network with CodeBERT: The base siamese network is trained with binary cross entropy using the label for each query-code pair.The loss distinguishes whether the code can answer or match the query.
- 5.2 Code Contrastive Learning: CoCLR extends the siamese network with contrastive learning over the original pair, in-batch augmented pairs, and query-rewritten pairs.Its overall objective combines the base loss with losses for in-batch and query-rewritten augmentation.
- 5.2 Code Contrastive Learning: In-batch augmentation treats each query paired with other mini-batch codes as dissimilar, while query-rewritten augmentation creates positive examples from lightly modified queries.Queries are rewritten by deleting, switching, or copying words; switching word positions performs best in the reported comparison.
6 Experiments
The experiments evaluate CoSQA and CoCLR for code question answering and code search, showing improvements over baseline CodeBERT and gains from contrastive augmentation. Analyses further examine augmentation choices and code components.
- Experimental Setup: The experiments evaluate code question answering and code search using models trained on CoSQA.Code question answering uses WebQueryTest with 1,046 expert-annotated query-code pairs, while code search uses CoSQA splits and a 6,267-code database.
- Model Comparisons: CodeBERT trained with CoSQA improves performance on both tasks, especially WebQueryTest without direct training data.The authors interpret this as evidence of CoSQA's quality and potential as training data for WebQueryTest.
- Model Comparisons: CoCLR further improves siamese CodeBERT performance on both tasks and raises WebQueryTest performance by 15.6%.The authors report this as a new state-of-the-art result on WebQueryTest.
- Effects of CoCLR: Both in-batch augmentation and query-rewritten augmentation improve code-search performance individually and together.The ablation compares binary cross-entropy alone with additional IBA, QRA, or both losses.
- Effects of CoCLR: Switching word positions in query rewriting performs better than the other tested rewriting methods, regardless of in-batch augmentation.The authors attribute this to web queries not necessarily having accurate grammar, making switched queries better pseudo-positive examples.
- Effects of CoCLR: Adding IBA yields a 1.25% gain versus 9.10% for QRA, but matched one-example comparisons produce an MRR of 55.52%.The authors therefore suggest that high-quality example quantity may matter more than whether examples come from IBA or QRA.
- Effects of Code Components: Removing documentation hurts code-search results more than removing the function header or function body.The analysis links documentation's importance to its shared natural-language modality with queries and its description of code functionality.
7 Conclusion
The paper introduces CoSQA, a large human-annotated dataset of real-world web queries and Python functions, and CoCLR, a contrastive method for generating training instances. The reported models outperform baselines on code question answering and code search, with potential relevance to other aligned text-code tasks.
- Contributions: CoSQA contains 20,604 pairs of real-world web queries and documented Python functions for query-code matching.The dataset is human annotated, with labels indicating whether a function can answer a query.
- Contributions: CoCLR incorporates artificially generated instances into training through code contrastive learning.The method is designed for query-code matching and is evaluated with CoSQA.
- Findings: Models using CoCLR outperform baseline models on code search and code question answering.The conclusion also reports detailed analyses of CoCLR and code components in query-code matching.
- Implications: The CoSQA dataset may support other aligned text-code tasks, including code summarization and code synthesis.
A Heuristics for Query Filtering
The query-filtering heuristics remove queries lacking code-search intent using keyword templates organized into categories. The paper notes that vague queries are excluded from this categorization because they vary morphologically.
- Filtering Rules: The filtering rules use keyword templates derived from six categories of queries without code-search intent.The keywords are presented in Table 8.
- Filtering Rules: Vague queries are ignored because their morphology is variable.