PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE Context Bench: A Benchmark for Context Learning in Coding

Jiayuan Zhu, Junde Wu, Minhao Hu, Shengda Zhu, Jiazhen Pan, Weixiang Shen, Yijun Yang, Fenglin Liu, Jianye Hao, Yueming Jin, Qirong Ho, Min Xu

arXiv:2602.08316v3cs.SEcs.AI

TL;DR

Existing coding benchmarks largely treat tasks independently, leaving context reuse and its efficiency benefits insufficiently evaluated. The paper introduces SWE-ContextBench, a benchmark linking base and related repository tasks through real issue and pull-request relationships. Results show that compact, correctly selected context improves resolution accuracy and can reduce runtime and cost, whereas irrelevant context may provide limited or negative benefits.

  • Problem

    Existing agent benchmarks focus on solving individual tasks and do not explicitly evaluate how coding agents reuse past experience across related tasks or its effects on efficiency.

  • Method

    SWE-ContextBench contains 1,100 experience tasks and 376 related tasks derived from explicit dependencies and references among GitHub issues and pull requests.

  • Results

    Accurately summarized and retrieved context improves resolution accuracy and reduces runtime and token costs, while unfiltered or incorrectly selected context provides limited or negative benefits.

  • Takeaways & Limitations

    Context learning benefits coding agents when reused context is compact and correctly selected, making retrieval quality central to effective and efficient task solving.

  • Takeaways & Limitations

    The benchmark lacks comprehensive human validation of problem descriptions, test patches, and overall task solvability, so some complex or indirectly linked instances may contain noise.

Abstract

from arXiv · show

Large language models are increasingly used as coding agents for software engineering tasks. Current benchmarks mainly evaluate whether the agent can correctly solve the request or fix the bugs. They largely treat tasks as independent and do not assess whether agents can reuse previous experience across related problems. As a result, the efficiency gains from reusing the previous experience remains difficult to measure. We introduce SWE-ContextBench, a benchmark designed to explicitly evaluate context understanding and retrieval in coding agents. SWE-ContextBench consists of 1,100 base tasks with another 376 related tasks derived from real dependency and reference relationships among GitHub issues and pull requests. SWE-ContextBench groups base tasks and related tasks with shared context across 51 unique repositories and 9 programming languages. The benchmark evaluates how accurately and efficiently agents solve related issues when prior cases are available in context. Using SWE-ContextBench, we study the behavior of multiple coding agents across varying context reuse settings and retrieval strategies. Our results show that accurately summarized and retrieved previous experience can significantly improve resolution accuracy and reduce runtime and token cost, particularly on harder tasks. In contrast, unfiltered or incorrectly selected context provides limited or negative benefits. These findings highlight the importance of context management and retrieval accuracy, and position SWE-ContextBench as a principled benchmark for studying context learning in coding agents.

1 Introduction

Existing coding-agent benchmarks largely treat tasks independently and do not measure reuse of prior experience. SWE-ContextBench addresses this gap by evaluating context learning across related software-engineering tasks and its effects on accuracy, time, and token efficiency.

  • Motivation: Existing agent benchmarks evaluate individual tasks from scratch rather than whether agents reuse past experience across related tasks.They mainly assess knowledge in model weights and short-term input context.
  • Motivation: Real-world software development repeatedly exposes engineers to related problems, successes, and failures that can inform later work.
  • Motivation: Long-context and memory benchmarks assess retrieval or reasoning over information, but do not specifically measure context learning in programming tasks.
  • Benchmark proposal: SWE-ContextBench evaluates context learning through task accuracy variation, context-aware solving time, and token efficiency.These dimensions measure whether agents locate and reuse relevant prior tasks while reducing computational cost.
  • Benchmark proposal: 1,476 tasks across 51 repositories and 9 programming languages comprise SWE-ContextBench, including 1,100 base tasks and 376 related tasks.The benchmark is built from SWE-Bench Lite, SWE-Bench Multilingual, and SWE-Bench Verified, with related instances selected by humans.

2 SWE-ContxetBench

SWE-ContextBench constructs reusable context from real repository tasks and links related issues through explicit dependency and reference relationships. Its structure supports test-based evaluation of context reuse across diverse tasks, repositories, and languages.

  • Evaluation dimensions: The benchmark measures accuracy variation, time efficiency, and cost efficiency when agents solve tasks with prior context.Accuracy variation compares performance with context against a no-context baseline, while time and cost efficiency target faster and less redundant reasoning.
  • Context trajectory collection: 1,100 base instances from 51 repositories and 9 programming languages provide realistic repository-level tasks and reusable solution trajectories.Each base-task trajectory records tool calls, file navigation, and intermediate reasoning for later retrieval.
  • Related task identification: Related tasks are identified by manually analyzing issue and pull-request cross-references, including multi-issue, pull-request-to-issue, pull-request-only, and issue-reference patterns.
  • Related task identification: 319 interdependent task instances are identified across the base instances, with each manually verified as a meaningful task with clear relationships and identifiable resolution.Recursive analysis adds 57 additional interdependent instances.
  • Resulting benchmark tasks: 1,476 total tasks comprise 1,100 experience tasks and 376 related tasks across 51 repositories.The related-task set preserves the experience pool’s language composition; Python accounts for 73.2% versus 70.7% of tasks across the two sets.
  • Resulting benchmark structure: 376 related tasks contain an average of 4.8 FAIL_TO_PASS tests and 211.5 PASS_TO_PASS tests, combining fix validation with regression checks.SWE-ContextBench Lite provides a smaller evaluation set of 300 base and 99 related Python tasks across 12 repositories.

3 Experiment

The experiments compare coding agents across context-reuse settings, showing that concise, accurately selected summaries improve accuracy and can reduce runtime and resource use, especially on difficult tasks.

  • Experimental setup: The benchmark evaluates agents with independently recorded experience trajectories and related-task runs, preventing implicit state carryover while enabling explicit context retrieval.Experience tasks are run in isolated environments, and related tasks receive retrieved context explicitly.
  • Experimental setup: Oracle summaries isolate the effect of high-quality context reuse by supplying gold relevant summaries averaging 217.1 tokens.The setting removes retrieval errors from the comparison.
  • Accuracy: Oracle summaries improve resolution for most agents, with GPT-5.3 Codex reaching 23.94%, Claude Sonnet 4.5 23.40%, and MiniMax 2.7 22.07%.GPT-5.3 Codex also achieves the best file localization at 57.22%.
  • Efficiency: Oracle summaries shift runtime and token distributions toward lower-cost ranges, with the strongest concentration gains for Claude Sonnet 4.5 and GPT-5.3 Codex.GLM-5 and Qwen3.5-plus show greater variability.
  • Context-learning settings: Oracle Summary Learning raises resolution from 26.26% for No-Context to 34.34%, whereas Oracle Context Learning reaches 27.27%.The oracle–free gap is larger for summaries than for full trajectories.
  • Efficiency and retrieval: On harder tasks, oracle-summary runtime falls by more than 60% for the slowest instances, while retrieval and memory methods trade accuracy against runtime and cost.Supermemory reaches 30.30% resolution; OpenViking is fastest at 4.20 minutes and costs $0.53.
  • Retrieval quality: Retrieval quality varies substantially: LangMem reaches 73.34% overall matching, while compact Supermemory contexts achieve 59.60% matching at approximately 30 tokens.Free Context Learning retrieves 136k-token top-1 contexts but matches only 18.18%.

4 Related Work

Existing memory and coding benchmarks evaluate general information handling or repository-level coding, but do not directly test reuse of prior problem-solving trajectories across tasks.

  • General memory benchmarks: General memory benchmarks measure information extraction, multi-session reasoning, and knowledge update over documents, conversations, or synthetic contexts.They focus on general-domain long-context and persistent-memory abilities.
  • Coding benchmarks: Coding benchmarks evaluate repository-level retrieval, code understanding, and bug fixing, but not whether agents learn from prior problem-solving contexts across tasks.Their contexts are tied to the current repository-level task.
  • SWE-ContextBench: SWE-ContextBench introduces trajectory-level memory and directly evaluates cross-task context learning.This distinguishes it from the benchmark categories summarized in Table 6.

5 Conclusion

SWE-ContextBench evaluates context learning in repository-level coding agents and shows that context quality determines whether reuse improves outcomes. Across agents, model capability and reuse strategy jointly shape effectiveness–efficiency trade-offs.

  • SWE-ContextBench evaluates context learning in repository-level coding agents using 1,476 tasks across 51 repositories and 9 programming languages.The benchmark includes 1,100 base tasks and 376 related tasks capturing explicit cross-task dependencies.
  • Context learning benefits agents only when reused context is compact and correctly selected.Free retrieval can provide limited gains or hurt performance when retrieved context is irrelevant.
  • Oracle Summary Learning achieves the highest task resolution, whereas Oracle Context Learning achieves the lowest average runtime and cost.
  • Results across five coding agents show that model capability and reuse strategy jointly shape the trade-off between effectiveness and efficiency.
  • Experiments with memory retrieval frameworks show that stronger retrieval quality improves downstream performance on related tasks.

A.1 Prompt Template Example

The prompt template standardizes repository preparation, task execution, patch generation, and prediction-file output. It also removes repository history and testbed artifacts to prevent leakage and isolate the generated patch.

  • The setup clones each repository, checks out its base commit, and installs dependencies with pip install -e testbed.
  • Repository history is removed and Git is reinitialized before the agent works, preventing solution leakage from prior commits.
  • Each prediction is saved as output/<instance_id>_preds.json with model_name_or_path, instance_id, and model_patch fields.
  • The generated patch is staged and captured with git -C testbed diff --cached.
  • The testbed folder is removed after predictions are saved.
  • Agents navigate to testbed/ and fix the bug described in problem_statement without including tests in the generated patch.

A.2 Implementation Details for Memory Retrieval Frameworks

The implementation describes four memory retrieval frameworks and feeds collected context trajectory files into them as prior context. The frameworks are Mem0, OpenViking, LangMem, and Supermemory.

  • Four memory retrieval frameworks are implemented: Mem0, OpenViking, LangMem, and Supermemory.Collected context trajectory files serve as the input for incorporating prior context.

A.2.1 Mem0

Mem0 ingests prior context trajectories into persistent memory and retrieves relevant contexts through semantic similarity search. Retrieved candidates are ranked by cosine similarity and filtered by context type.

  • Memory Ingestion: Previous JSONL conversation files are parsed into a summary, complete user/assistant history, and repository and instance metadata before storage.The structured representation is committed to a persistent local Mem0 Memory instance through sequential add calls.
  • Memory Retrieval: At inference, a natural-language problem query is embedded with text-embedding-3-small and searched against a Qdrant vector collection.Retrieved candidates are ranked by cosine similarity and filtered by type: past_context.

A.2.2 OpenViking

OpenViking ingests past context trajectories through automatic parsing, chunking, embedding, and indexing, then retrieves relevant contexts with semantic search before fetching their full contents.

  • OpenViking registers each JSONL trajectory as a resource, automatically parsing, chunking, and embedding its contents.Files are processed in batches of 25, with synchronous completion checks after each batch.
  • At inference time, semantic search scans the full resource collection and returns results exceeding a 0.2 relevance threshold.Each result includes a resource URI, relevance score, and generated abstract.
  • The top-k retrieved resources are fetched separately through their full trajectory contents for downstream use.

A.2.3 LangMem

The LangMem and Supermemory configurations ingest past trajectories into persistent or managed memory stores and retrieve them for coding tasks. Results emphasize that useful context requires concise summaries, selective reuse, adaptive retrieval, and interpretation of potentially erroneous prior reasoning.

  • LangMem: LangMem parses each trajectory into ordered conversation messages, normalizes tool calls, and stores extracted memories in a persistent vector-indexed SQLite store.
  • LangMem: 997 memories were extracted from 300 trajectories in approximately 58 minutes at a cost of $0.98.
  • Supermemory: Supermemory uploads formatted trajectories to a managed cloud platform, where content extraction, embedding, and RAG indexing are handled by its closed-source backend.Files are tagged under the past_context container with metadata such as filename, session ID, and message count.
  • Context learning findings: Concise summaries improve reuse because raw trajectory logs are often too verbose and noisy for direct use.The results identify summarization quality as an important factor in context learning.
  • Context learning findings: Naive context learning can raise cost without improving accuracy, while retrieved trajectories may still mislead agents through errors, trial-and-error, or incomplete reasoning.Agents therefore need to decide when to reuse context and interpret retrieved material rather than follow it blindly.
  • Context learning findings: Retrieval frameworks improve performance by finding relevant prior cases, but fixed retrieval counts may not fit tasks requiring either multiple examples or one highly relevant case.The findings motivate adaptive selection of both the number and relevance of retrieved contexts.
Loading 2602.08316v3…