PaperlayerView exact revision on arXiv

Source-linked AI summary

CL4SE: Benchmarking Context Learning on Software Engineering

Haichuan Hu, Quanjun Zhang, Ye Shang, Guoqing Xie, Chunrong Fang, Zhenyu Chen, Liang Xiao

arXiv:2602.23047v3cs.SE

TL;DR

Existing research lacks a systematic taxonomy and benchmark for understanding how SE-specific context types affect different software engineering tasks. CL4SE addresses this gap with a taxonomy, datasets, and multi-model evaluation, finding a 24.7% average performance improvement and task-specific optimal strategies.

  • Problem

    Existing research lacks systematic categorization of SE-specific context types and benchmarking of their heterogeneous effects across software engineering tasks.

  • Method

    CL4SE defines four SE-oriented context types mapped to representative tasks, constructs datasets with over 13,000 samples from more than 30 open-source projects, and evaluates five LLMs using nine metrics.

  • Results

    24.7% average performance improvement across all tasks, with task-specific strategies including interpretable examples for code generation and project-specific context for summarization.

  • Takeaways & Limitations

    CL4SE provides a standardized evaluation framework, reproducible dataset, and actionable evidence that context engineering requires task-specific design rather than a one-size-fits-all approach.

  • Takeaways & Limitations

    Current context learning methods perform well on static projects but cannot adapt to the evolving nature of software development, motivating dynamic context engineering.

Abstract

from arXiv · show

Context engineering has emerged as a pivotal paradigm for unlocking the potential of Large Language Models (LLMs) in Software Engineering (SE) tasks, enabling performance gains at test time without model fine-tuning. Despite its success, existing research lacks a systematic taxonomy of SE-specific context types and a dedicated benchmark to quantify the heterogeneous effects of different contexts across core SE workflows. To address this gap, we propose CL4SE (Context Learning for Software Engineering), a comprehensive benchmark featuring a fine-grained taxonomy of four SE-oriented context types (interpretable examples, project-specific context, procedural decision-making context, and positive & negative context), each mapped to a representative task (code generation, code summarization, code review, and patch correctness assessment). We construct high-quality datasets comprising over 13,000 samples from more than 30 open-source projects and evaluate five mainstream LLMs across nine metrics. Extensive experiments demonstrate that context learning yields an average performance improvement of 24.7% across all tasks. Specifically, procedural context boosts code review performance by up to 33% (Qwen3-Max), mixed positive-negative context improves patch assessment by 30% (DeepSeek-V3), project-specific context increases code summarization BLEU by 14.78% (GPT-Oss-120B), and interpretable examples enhance code generation PASS@1 by 5.72% (DeepSeek-V3). CL4SE establishes the first standardized evaluation framework for SE context learning, provides actionable empirical insights into task-specific context design, and releases a large-scale dataset to facilitate reproducible research in this domain.

1 Introduction

CL4SE addresses the lack of systematic understanding of SE-specific context types by benchmarking how heterogeneous contexts affect core SE workflows. Its experiments find that task-aligned context improves performance and can shape both model outputs and reasoning.

  • Existing SE research lacks a systematic taxonomy of context types and analysis of how they affect different tasks.
  • CL4SE characterizes four SE-oriented context types and maps them to code generation, code summarization, code review, and patch correctness assessment.
  • 24.7% average performance improvement is achieved across tasks through context learning.
  • Context is most useful when its structure matches the target task’s cognitive demands, rather than being designed task-agnostically.
  • Effective contexts expose reasoning paths, decision procedures, and valid-versus-invalid contrasts, teaching models both what to do and how to do it.
  • CL4SE provides a standardized evaluation framework and actionable guidance for principled context design in SE.

2 Related Work

Prior work spans prompt engineering, context engineering, and in-context learning, but often treats context selection as ad hoc. CL4SE narrows this gap by studying context learning across representative SE workflows and their reasoning demands.

  • 2.1 Prompt/Context Engineering: Prompt engineering designs input queries to elicit desired responses, whereas context engineering retrieves, organizes, manages, and optimizes task-relevant information.
  • 2.1 Prompt/Context Engineering: Context engineering has become important for agents because they require continuous, multi-turn context understanding and management.
  • 2.1 Prompt/Context Engineering: In-context learning enables generalization to unseen tasks without parameter updates and underpins both prompt engineering and context engineering.
  • 2.2 LLM-based Software Engineering: Code review requires complex reasoning and is better modeled as procedural decision-making than as static classification.
  • 2.2 LLM-based Software Engineering: Patch correctness assessment requires distinguishing valid from overfitting patches through comparative reasoning, motivating positive and negative contexts.
  • 2.2 LLM-based Software Engineering: Existing literature lacks systematic categorization of SE-specific contexts and benchmarking of their heterogeneous effects across SE tasks.

3 Benchmark

CL4SE unifies multiple SE tasks, context categories, data granularities, and evaluation settings in a benchmark designed to study how context affects LLM learning. It uses real-project data and supports broad model and language coverage.

  • 3 Benchmark: Prior SE benchmarks are usually task-specific and limited to one granularity level, without fine-grained analysis of context categories.
  • 3 Benchmark: CL4SE unifies four representative SE tasks, spans multiple granularity levels, and analyzes relationships between context types and task-specific learning behavior.
  • 3 Benchmark: The benchmark focuses on context types and organizational structures, rather than merely adding few-shot examples to prompts.
  • 3.1 Overview: CL4SE covers code generation, code summarization, code review, and patch correctness assessment across software lifecycle stages.
  • 3.1 Overview: The benchmark uses substantial data from real software projects, multiple programming languages, various LLMs, and task-specific evaluation metrics.

3.2 Context Taxonomy

CL4SE identifies four SE-oriented context types and examines how they support different forms of model learning. The taxonomy includes interpretable examples, project-specific context, procedural decision-making context, and positive & negative context.

  • Project-specific context: Project-specific context captures information tied to a particular software project and its development practices.
  • Positive & negative context: Positive and negative examples provide contrasting reference frames for learning successful and unsuccessful task handling.The paper notes that using only one type may overemphasize limited patterns or incorrect trajectories.
  • Interpretable examples: Interpretable examples expose reasoning processes and problem-solving strategies alongside task examples.

3.3 Task Selection

CL4SE maps each context type to a representative software engineering task to study context effects in task-specific settings. The mapping links project knowledge, reasoning demonstrations, and contrasting examples to distinct SE workflows.

  • CL4SE maps context types to code generation, code summarization, code review, and patch correctness assessment.The mapping is presented as a one-to-one relationship between context types and representative downstream tasks.
  • Code summarization: Project-specific context is paired with code summarization because project knowledge, naming conventions, and coding styles vary across codebases.
  • Code generation: Interpretable examples are paired with code generation because explicit reasoning demonstrations can guide construction of correct solutions.
  • Patch correctness assessment: Positive and negative examples are paired with patch correctness assessment because distinguishing correct from overfitting patches involves successful and unsuccessful cases.

3.4 Data Collection

CL4SE builds task-specific datasets from real-world and established software engineering sources, applying filtering, balancing, deduplication, and quality checks. The resulting benchmarks cover diverse projects, samples, and problem types.

  • Code generation: 636 code-generation problems form the test set after filtering 2,083 valid problems and selecting the top 30% by difficulty.The remaining 1,447 problems constitute the training set.
  • Code generation: The code-generation test set spans more than 15 categories rather than concentrating on a few problem types.
  • Code summarization: 8,225 code summarization samples were retained from 2,495 source files and 1,197 classes across large Python open-source projects.Samples were parsed at function granularity and manually spot-checked after filtering.
  • Code review: 1,916 code-review samples from 32 repositories comprise 1,191 positive and 725 negative samples.The repositories span Python, JavaScript, and SQL and include varied functionalities.
  • Patch correctness assessment: 2,274 patch-assessment samples contain 1,169 overfitting patches and 1,105 correct patches from Defects4J-based collection and deduplication.The benchmark combines plausible APR patches with developer patches identified as correct.

3.5 Evaluation and Metrics

CL4SE evaluates generation, summarization, review, and patch assessment with task-appropriate metrics. These metrics measure functional success, text overlap or similarity, and classification agreement with ground truth.

  • Code generation: Code generation uses PASS@1 to measure whether the first generated program meets expected functionality.PASS@1 counts first attempts that pass test cases relative to total attempts.
  • Code summarization: Code summarization uses ROUGE, BLEU, METEOR, and BERTScore to assess generated summaries against references.The metrics capture n-gram overlap, precision and recall, synonymy or stemming, fragmentation, and embedding similarity.
  • Code review and patch correctness assessment: Code review and patch correctness assessment use Accuracy, Precision, Recall, and F1-score to compare predictions with ground truth.

4 Experiment

The experiments investigate context learning across software-engineering tasks and models, using five representative LLMs and a broad evaluation design. The study covers task-specific research questions and reports implementation choices intended to support reproducibility.

  • Research questions: The study examines context learning across code generation, code summarization, code review, and patch correctness assessment.These tasks correspond to the paper’s research questions on different context types and workflows.
  • Model selection: Five representative LLMs span general-purpose and software-engineering-specialized models from diverse global manufacturers.The selection includes both closed-source and open-source systems.
  • Reproducibility: The implementation details cover model deployment, parameter configuration, search framework, and technical stack to support reproducibility.Experiments use an 8-card RTX 5880 GPU cluster, while closed-source models are accessed through official APIs.

5 Main Results

Context learning improves performance across the evaluated software-engineering tasks, but its effects vary by task, model, context type, shot count, and problem difficulty. The strongest gains occur in reasoning-intensive workflows, while different contexts exhibit distinct optimal usage patterns.

  • Overall performance: 33% improvement is achieved by Qwen3-Max in code review, the largest task-specific gain reported across the evaluated workflows.Patch correctness assessment follows with a 30% improvement for DeepSeek-V3.
  • Model differences: GPT-Oss-120B and Claude-3.5 show the highest elasticity to context learning, while lower zero-shot baselines generally achieve larger proportional gains.Qwen3-Max still benefits substantially on reasoning-intensive tasks despite strong zero-shot performance.
  • Overall performance: Context learning benefits reasoning-intensive code review and patch correctness assessment more than code generation and code summarization.The results characterize code review and patch assessment as more sensitive to complex logical reasoning and decision-making.
  • Code generation: 1-shot and 2-shot interpretable examples usually maximize code-generation performance, whereas excessive demonstrations can cause performance degradation.DeepSeek-V3 reaches its highest overall PASS@1 rate at 1-shot (77.99%), while GPT-Oss-120B peaks at 4-shot (73.11%).
  • Code generation: Problem difficulty shapes interpretable-example effectiveness, with medium-difficulty tasks benefiting most and easy or hard tasks showing limited gains.On easy problems, several models exhibit ceiling effects; on medium problems, low-shot settings generally perform best.
  • Code summarization: 1-shot project-specific context is optimal for code summarization, producing stronger lexical gains than semantic gains.The study uses same-repository demonstrations to align summaries with project terminology, formatting, and documentation style.
  • Code review: Procedural decision-making context improves code review steadily through 5-shot without redundancy-induced degradation, primarily strengthening recall while stabilizing precision.This pattern contrasts with the shot-count behavior observed for interpretable and project-specific contexts.
  • Patch correctness assessment: 2-shot mixed positive-negative context achieves optimal patch-assessment performance, while positive examples alone outperform negative examples alone.Context mainly improves recall while preserving high precision, with larger gains for general-purpose models.

6 Discussion

The paper identifies deployment barriers that extend beyond benchmark performance: limited contextual data for low-resource languages and static context assumptions for evolving software projects. These boundaries motivate cross-language transfer and dynamic context-engineering research.

  • Low-resource languages: Low-resource programming languages lack sufficient high-quality contextual examples for effective in-context demonstrations.Language-specific semantics and cross-language transfer gaps further constrain generalization, especially in niche industrial domains.
  • Dynamic project contexts: Static context design assumes fixed project conventions, documentation styles, and decision norms, whereas real-world software projects evolve.This mismatch creates practical deployment challenges for maintaining contextual alignment.
  • Dynamic project contexts: Context drift can make static examples outdated or misleading as project contexts change over time.The paper identifies dynamic context engineering as a direction for maintaining alignment with real-time project states.
  • Dynamic project contexts: Long-tail context scarcity leaves emerging edge cases without corresponding historical examples.The absence of relevant examples limits coverage as projects encounter new situations.
  • Dynamic project contexts: Context scalability is difficult because large projects contain fragmented sub-contexts that complicate context construction and maintenance.This is one of three challenges associated with evolving project contexts.

7 Conclusion

CL4SE establishes a standardized benchmark for evaluating SE context learning across four context types, four representative workflows, and large-scale real-world data. Its experiments show substantial but task-dependent gains, while identifying deployment challenges that motivate dynamic and cross-language context engineering.

  • 7 Conclusion: 13,000+ samples from over 30 open-source projects support evaluations of five mainstream LLMs across four core SE tasks.The benchmark covers code generation, code summarization, code review, and patch correctness assessment.
  • 7 Conclusion: 24.7% average performance improvement across all tasks demonstrates context learning’s broad effectiveness for SE workflows.The reported gains are accompanied by task-specific optimal strategies rather than a single universal context configuration.
  • 7 Conclusion: 5-shot procedural context increases Qwen3-Max code review Accuracy and F1 by 18.69% and 22.80%, respectively.Other task-aligned strategies include interpretable examples for code generation, project-specific context for summarization, and mixed positive-negative context for patch assessment.
  • 7 Conclusion: Real-world deployment remains constrained by scarce high-quality examples for low-resource languages and static context designs that cannot adapt to project evolution.The paper identifies context drift, long-tail context scarcity, and context scalability as open challenges.
  • 7 Conclusion: CL4SE releases a standardized evaluation framework, a large-scale dataset, and empirical guidance for principled task-specific context design.These contributions support reproducible research and model comparison beyond ad-hoc prompt crafting.

Declarations

The declarations provide ethical-approval and informed-consent status, followed by author contribution statements.

  • Declarations: Ethical approval and informed consent are both listed as not applicable.The passage then records individual author contributions across conceptualization, methodology, writing, investigation, validation, software, data curation, and formal analysis.
Loading 2602.23047v3…