Evaluating Code Recommender Systems: A Review

1 Introduction

• CRSs operate on source artifacts and provide recommendations across software activities, under varied methods, automation levels, and interaction styles.They are also called programming assistants, code assistants, chatbots, or code tools.Source: §1 · p. 1
• Evaluation evidence matters because software engineering is human-centric, and understanding developers’ observations and experiences can inform assessments of CRS quality in use.The review asks how comprehensively evaluation practice investigates developers’ experiences of applying CRSs.Source: §1 · p. 1
• The Eclipse Theia example shows a natural-language prompt yielding code recommendations with explanations, which developers can copy, insert, or refine through further prompting.The depicted system is a built-in chat interface driven by GPT-4o.Source: Figure 1 · p. 1
• The introduction asks whether evaluations reflect that developers spend much working time and perceived workload on activities other than coding during software construction.The authors connect this question to Myths 1 and 2 in.Source: §1 · p. 2
• The review analyzed 92 primary studies from 2017–2024 to characterize evaluation types, software engineering areas, recommender operations, metrics, and reported threats to validity.The studies were selected from an initial set of 4,126 publications using predefined selection and quality criteria.Source: §1 · p. 2
• Earlier reviews addressed general recommender systems, model-driven engineering, or LLM-based code generation and assisted programming, but had not been updated since 2023.This review aims to address the evidence gap between general recommender-system research and narrower CRS subtypes.Source: §1 · p. 2

2 Background

• The review considers CRSs across software processes, developer tasks, and interaction modes, including completion, creation, finding, repair, and reuse.The scope includes reactive, proactive, or mixed operation modes.Source: §2 · p. 2

2.1 Evaluating Code Recommenders

• Offline evaluations assess recommendations without developer-system interaction; they are scalable and repeatable, but cannot capture user-system interactions or all recommender qualities.They commonly use historical or synthetic data and quantitative measures such as precision, recall, and accuracy.Source: §2.1 · pp. 2–3
• User studies observe participants interacting with a recommender on predefined tasks, gathering quantitative and qualitative feedback, but are costly and limited in scale.Small participant and task numbers constrain scalability and generalizability.Source: §2.1 · p. 3
• Online evaluations collect user feedback and interaction data on self-selected tasks in field settings, providing quality-in-use evidence over extended periods but requiring substantial resources.Field deployment can also constrain data collection and pose recruitment or organizational challenges.Source: §2.1 · p. 3

2.2 Literature Reviews

• Systematic mapping studies broadly classify and cluster research, whereas systematic literature reviews seek relevant primary studies and synthesize evidence to answer predefined questions.Both approaches follow systematic principles and established guidelines.Source: §2.2 · p. 3

3 Review Design

• The SLR follows established review guidelines and assesses documentation quality alongside evaluation characteristics to inform understanding of study replicability.The review design is based on Kitchenham et al., Zhang et al., and Wohlin.Source: §3 · p. 3

3.1 Research Questions

• The review examines evaluation prevalence, evaluation types’ use across quality characteristics and software areas, metrics, and reported threats to validity.RQ1 uses the full relevant corpus, while RQ2–RQ4 require detailed reporting and use the highly relevant subset.Source: §3.1 · p. 3

3.2 Review Protocol

• Eligibility required a qualified CRS-and-evaluation contribution; the review included relevant and highly relevant publications, with relevance rated from not relevant to highly relevant.The search was not restricted by CRS technology or interaction mode.Source: §3.2 · p. 4

3.3 Planning the Search

• The pilot search combined automated Google Scholar searching (first 30 pages) with one backward-snowballing iteration, yielding 151 candidates; after duplicate and screening exclusions, 25 remained for the quasi-gold-standard corpus.The reference corpus was used to iteratively define and continuously validate the search string for the main search.Source: §3.3 · p. 5

Quasi-gold standard corpus and search engines

• The quasi-gold standard comprised 18 conference and seven journal articles from 17 venues, and guided selection of four search engines.The selected engines were ACM Digital Library, IEEE Xplore, Scopus, and SpringerLink.Source: Quasi-gold standard corpus and search… · p. 5

Search string

• The final search string contained 27 terms across evaluation and tool-and-action categories, targeting publication titles, abstracts, and keywords.Searches were limited to English full-text articles from 2017–2024.Source: Search string · p. 5

3.4 Main Search

• The main search retrieved 4,101 records; after deduplication and screening, quality assessment left 405 publications in the corpus.The search retrieved 24 of 25 QGS publications, a quasi-sensitivity of 96%; the remaining QGS publication was added separately.Source: §3.4 · pp. 5–6
• Publication numbers rose sharply from 2022 through the end of the study period in 2024.The authors note that evaluative research followed the increase in CRS publications.Source: §3.4 · p. 6

Evaluation types

• 70.0% of evaluations were Offline Evaluations; Evaluations accounted for 17.3%, User Studies 11.1%, and Online Evaluations 1.6%.Offline Evaluations were the most frequently applied type.Source: §4.1 · p. 7, Figure 4 · p. 7

Combined evaluation types

• 94 publications reported two evaluation types, and none reported more than two.The most frequent combination was Offline Evaluation with Human-involved Offline Evaluation (72 cases, 76.6%); in publication families, 75% of later publications retained their predecessor's evaluation type or types.Source: §4.1 · pp. 7–8

Evaluation types over time

• Each additional publication year was associated with an 11.8% decrease in the odds of user-centric evaluation (OR 0.882, 95% CI: 0.791–0.983).For fully user-centric evaluations, the odds decreased by 14.6% per year (OR 0.854, 95% CI: 0.743–0.981).Source: §4.1 · p. 7, Figure 5 · p. 8

Areas and operations

• 61.6% of coded software engineering areas were Software Construction, and Create was the most frequent CRS operation at 53.8%.All evaluation types were primarily associated with Software Construction; Create was most frequent except in Online Evaluations, where Complete was most common.Source: §4.2 · p. 8, §4.4 · p. 10
• Sixteen evaluation-type occurrences were coded as Other, representing six additional operations added to the coding scheme.These included Analyze (5), Diagnose (4), Modify (3), Translate (2), and Optimize (1).Source: §4.2 · p. 8

Metrics and measures

• 393 unique metrics appeared in 556 occurrences, but only 53 metrics were reported in more than one publication.The most prevalent measurement objectives were Functional Suitability (50.7%), Usability (17.4%), and Maintainability (10.8%).Source: §4.3 · pp. 8–9
• Recommendation was the measurement object for 43.9% of metrics, while Developer accounted for 33.3%.Developer impact was assessed directly through measures such as task completion time and acceptance rate, or indirectly through subjective measures of perceptions.Source: §4.3 · p. 9

Threats to validity

• Threats to validity were reported in 79 of 92 publications (85.9%); External threats were the most common coded threat type at 38.8%.Among publications reporting threats, External threats appeared in 94.9%; common examples concerned study materials, while Construct threats concerned operationalization and measurement.Source: §4.4 · p. 9

Formulation of research questions

• Research-question formulation may reflect authors’ subjective judgment and prior knowledge, although closely related literature was consulted and the PICOC model informed question development.The authors describe this as a construct-validity threat.Source: §5 · p. 10

Search process

• No search strategy guarantees exhaustive coverage, and incomplete keyword representation could omit relevant studies despite a QGS-based strategy combining automated search, backward snowballing, and a pilot study.The search process also used backward snowballing and a pilot study.Source: §5 · p. 10
• The search was limited to peer-reviewed venues and conducted in November 2024, excluding gray literature and leaving newer studies able to revise the findings.Excluding non-peer-reviewed literature also provided a baseline quality threshold for included publications.Source: §5 · p. 10

Study selection

• Selection combined automated and manual screening, which can respectively exclude papers with incomplete metadata or introduce subjective judgment and researcher bias.Mitigations included conservative automated exclusion, protocol-based screening, reviewer validation, reliability checks, consensus, and a public artifact for independent assessment.Source: §5 · p. 10

Relevance classification

• Relevance judgments could misclassify publications and bias findings; two reviewers rated them, with κ = 0.75 for N1 and κ = 0.53 for N2, and disagreements were resolved by consensus.RQ1 used 405 relevant or highly relevant publications, while RQ2–RQ4 used 92 highly relevant publications with sufficient detail.Source: §5 · p. 10

Data extraction and analysis

• Manual extraction could introduce interpretation, omission, or ambiguity errors; one reviewer extracted data, 27% was independently double-coded, and remaining issues were discussed collaboratively.Predefined classification schemes informed by SWEBOK and prior secondary studies were used.Source: §5 · p. 10
• Trend findings may not generalize beyond 2017–2024 because observations are concentrated in 2022–2024, and the analysis assumes independence across publication years.Follow-up and extension studies in the corpus were treated as one publication to mitigate possible dependence.Source: §5 · p. 11

Offline evaluations on the rise

• Over time, the odds of studies using user-centric and fully user-centric evaluations declined by 11.8% and 14.6% per year, respectively.The authors offer cost, accessibility of benchmarks, added design demands for nondeterministic LLMs, and the corpus's early coverage of LLM-based CRS research as possible explanations.Source: §6 · p. 11

Offline with humans

• Offline Evaluations account for 70% of reviewed primary studies; Human-Involved Offline Evaluations account for 17%, while User Studies and Online Evaluations together comprise only 1 in 7.Human-Involved Offline Evaluations involve developers judging CRS outputs without direct interaction and require comparatively less effort and resources than User Studies or Online Evaluations.Source: §6 · p. 11

Solitary evaluations

• Only approximately 23% (94) of reviewed publications combine evaluation types, and combinations span at most two types, rather than progressing across Offline Evaluations, User Studies, and Online Evaluations.The discussion describes Offline Evaluations as an initial filter, User Studies as a way to investigate or compare systems, and Online Evaluations as field assessment over time.Source: §6 · pp. 11–12

Normal-angle versus wide-angle lenses

• Among single-type evaluations, 89% of system-centric evaluations address one or two objectives, while 65% of user-centric evaluations cover three or more; 45% of mixed evaluations cover three or more.The discussion cautions that the number of objectives alone does not establish evaluation quality or completeness.Source: §6 · p. 12

Out of the BLEU

• NLP metrics appear in 17.4% of primary studies, and the discussion cautions that token matching can miss functional equivalence and produce inconsistent scores across BLEU variants.The authors recommend prioritizing execution-based metrics such as pass@k, while further research is needed to validate code-specific artifact metrics.Source: §6 · p. 12

Replicability

• Research artifacts were provided by 67% of publications, rising from 27% in 2017 to 72% in 2024; availability was 59% for user-centric, 68% for system-centric, and 71% for combined evaluations.The authors recommend complete replication packages containing materials, infrastructure, and data, subject to ethical or privacy constraints.Source: §6 · pp. 12–13

Validity reporting

• Approximately 86% (79/92) of publications reported validity threats, and material-related threats were most frequent and had the strongest impact on External validity.The discussion links material threats to Offline Evaluations' dependence on models and datasets, and identifies replicability threats specific to LLM-based CRSs.Source: §6 · p. 13
• Threat-category misclassification and uneven reporting remain concerns; the authors recommend reporting threats regardless of outlet constraints and explaining classification choices.Examples of reported confusion include Construct validity being classified as Internal or External validity.Source: §6 · p. 13

Related systematic literature reviews

• Related SLRs cover general recommender evaluation, LLMs in software engineering, and software engineering assistants for broader work tasks.Their reviews catalog topics including datasets, metrics, experimental designs, LLM tasks, assistant types, and evaluation approaches.Source: §7 · pp. 13–14

Related systematic mapping studies

• Related mapping studies address CRSs in model-driven engineering, LLM-based code generation, recommender systems for programmers, and code-comprehension experiments.Their scopes include specific domains, assistance types, code-generation evaluations, or experiment design and reporting.Source: §7 · p. 14
• This review complements prior work by including CRSs regardless of recommendation method or technique and focusing exclusively on their evaluation.Among the related studies discussed, research artifacts are reported as provided only by.Source: §7 · p. 14

Conclusion

• Across 92 primary studies from 2017–2024, system-centric Offline Evaluations predominate, while Human-Involved Offline Evaluations, User Studies, and Online Evaluations remain underrepresented.Most evaluations address Software Construction and Create operations.Source: §8 · p. 14
• Researchers predominantly use information retrieval and NLP-based metrics for Functional Suitability, Usability, and Maintainability; most studies report validity threats, especially Internal and External threats.The review's underlying dataset is publicly available for further analyses.Source: §8 · p. 14
• Offline Evaluations dominate new publications while user-centric evaluations decline; the authors suggest a methodical evaluation framework for more standardized, comparable practices and broader use of evaluation types.The authors state that further investigation is needed to confirm explanations for the trend and identify ways to reverse it.Source: §8 · p. 14

CRediT authorship contribution statement

• Borst contributed investigation, data curation, formal analysis, and visualization; Sobernig contributed supervision and funding acquisition.Both authors contributed to conceptualization and writing.Source: CRediT authorship contribution statement · p. 14

• The full search string applies the evaluation terms AND the topic terms across all databases.Source: A Search String · p. 15
• The evaluation-term group includes “evaluat*,” “assess*,” “study,” and “experiment.”Source: A Search String · p. 15
• The topic group spans code assistance, recommendation, generation, synthesis, and related software-development tasks.Its terms include “github copilot,” “code completion,” “refactoring recommendation,” “bug fixing,” and “code synthesis.”Source: A Search String · p. 15