Source-linked AI summary
Compressing Code Context for LLM-based Issue Resolution
Haoxiang Jia, Earl T. Barr, Sergey Mechtaev
TL;DR
Repository-level issue resolution overapproximates code context, increasing token costs and distracting models while existing compressors can damage semantic or repair-relevant information. The paper introduces OCD to distill minimal sufficient contexts and trains SWEzze to compress them at inference time. Across three frontier models, SWEzze improves resolution while maintaining stable compression and broad coverage of solvable instances.
Problem
Existing retrieval returns overapproximate code contexts, raising inference costs and distracting models, while compressors may disrupt semantic links or discard patch ingredients.
Method
OCD uses a hierarchical search combining genetic algorithms and hierarchical delta debugging to produce minimal sufficient contexts for training the SWEzze compression model.
Results
SWEzze maintains stable compression while improving issue-resolution rates and covering 93.8%–99.2% of the union of instances resolved by any baseline across three frontier models.
Takeaways & Limitations
For repository-level issue resolution, compression is more effective when it preserves repair-sufficient context rather than maximizing token reduction or similarity.
Takeaways & Limitations
Minimality is defined relative to the specific LLM and generation parameters used during data construction, so the minimal context may vary across models.
Abstract
from arXiv · showhide
Large Language Models (LLMs) are now capable of resolving real-world GitHub issues. However, current approaches overapproximate the code context and suffer from two compounding problems: the prohibitive cost of processing massive inputs, and low effectiveness as noise floods the context window and distracts the model from the bug-fixing signal. Existing compression techniques fail to resolve this tension: generic compressors compromise the semantic integrity of code, while code-specific tools lack awareness of code structure and task context to preserve essential patch ingredients. To address this, we propose a novel framework consisting of two components. First, Oracle-guided Code Distillation (OCD), a context distillation algorithm that combines genetic search and delta debugging to systematically reduce code contexts to their minimal sufficient subsequence - retaining only the ingredients required for a successful fix. We use this distilled data to fine-tune SWEzze, a lightweight model that learns to compress code context at inference time, filtering noise and combating distraction while preserving fix ingredients. Evaluated on SWE-bench Verified across three frontier LLMs, SWEzze maintains a stable compression rate of about 6 times across models, reduces the total token budget by 51.8%-71.3% relative to the uncompressed setting, improves issue resolution rates by 5.0%-9.2%, and delivers the best overall balance among effectiveness, compression ratio, and latency compared with state-of-the-art context compression baselines.
1 INTRODUCTION
Repository-level issue resolution requires large, scattered code contexts, but overapproximation increases inference cost and distracts models. The paper proposes OCD and SWEzze to preserve repair-sufficient context while reducing noise, improving resolution across frontier models.
- Motivation: Heuristic retrieval often returns hundreds of lines to capture a few relevant segments, creating context overapproximation.Retrieved context can span multiple files, definitions, call chains, hierarchies, modules, and interfaces.
- Motivation: Excessive context raises inference cost and floods the context window with irrelevant code that distracts models from bug-fixing signals.Inference cost scales roughly with provided context size.
- Limitations of prior compression: Generic compressors can disrupt program structure and semantic links, while code-specific heuristics may lack task context needed for patch generation.Relevant links include definition–use relations and type constraints.
- Proposed framework: OCD searches for a minimal sufficient context using genetic algorithms followed by hierarchical delta debugging, retaining fragments functionally necessary for repair.The distilled contexts train SWEzze, a lightweight inference-time compressor.
- Evaluation: SWEzze maintains about 6× compression, reduces total token budget by 51.8%–71.3%, and improves issue resolution rates by 5.0%–9.2% across three frontier models.The evaluation uses SWE-bench Verified within the Agentless workflow.
- Evaluation: SWEzze reaches 93.8%–99.2% of the union of instances solved by any baseline, supporting repair-sufficient context over maximal token reduction or similarity.The paper presents OCD, SWEzze, and benchmark evidence as its main contributions.
2 MOTIVATING EXAMPLE
A Matplotlib DPI bug illustrates why compression must retain project-specific dependencies rather than merely related code. SWEzze preserves the context needed to recompute DPI and produces a correct patch where baseline compressors do not.
- Issue: The Matplotlib issue requires inserting a state update using a project-specific API that is absent from the issue description.The human patch uses state.get('_original_dpi', state['_dpi']) inside figure.py.
- Compression workflow: SWEzze is integrated into Agentless to prune retrieved code to the semantic dependencies needed for issue resolution.The design is intended to be model- and harness-agnostic.
- Patch outcome: SWEzze enables a correct patch for the DPI issue, whereas LongCodeZip and SWE-Pruner produce patches that do not resolve it.Agentless succeeds only with SWEzze because the baselines discard crucial fix-relevant information.
- Context comparison: Figure 2 compares compressed contexts against a delta-debugging-derived minimal sufficient context and highlights retained versus removed segments.SWEzze has the higher correlation with the minimal sufficient context and uniquely contains the ingredients for recomputing DPI.
- Context comparison: The retained scale computation fragment reveals how DPI is computed, allowing the downstream model to generate the correct repair.SWEzze retains this fragment, while LongCodeZip selects irrelevant fragments and SWE-Pruner retains only an insufficient single line.
- Context fidelity: SWEzze obtains BERTScore 0.44 against the minimal context, more than doubling LongCodeZip’s 0.20 and SWE-Pruner’s 0.00.The approach preserves critical components such as the resize method.
3 ORACLE-GUIDED CONTEXT DISTILLATION
OCD formulates context distillation as a functional search for a 1-minimal sufficient subset of retrieved code, using hierarchical structure to preserve well-formed candidates. Its two-phase GA–HDD pipeline first finds repair-enabling contexts and then removes redundant units, producing training data that exposes severe relevance imbalance.
- Minimal Sufficient Context: A sufficient context enables an LLM-generated patch to pass validation, while a minimal sufficient context becomes insufficient after removing any single retained element.This 1-minimal criterion is tractable and requires at most one oracle call per retained element during verification.
- Oracle-Guided Distillation: OCD uses an execution oracle to evaluate candidate contexts by running the repair model and validating generated patches, rather than relying on perplexity or embedding similarity.The oracle-guided formulation defines sufficiency functionally through successful repair.
- Search Space Representation: Hierarchical search decomposes files into file-, function-, and block-level units, with segments serving as the atomic inclusion and exclusion decisions.Omitted units are replaced with placeholders indicating the location and magnitude of the omission, preserving structural awareness.
- Priority-Guided Search: Priority scores use patch overlap to elevate units in files modified by the ground-truth patch and guide search toward likely repair-relevant regions.The score is used to bias genetic initialization and fitness, and to guide HDD toward removing low-priority units first.
- Two-Phase Search Strategy: GA identifies a resolution-enabling subsequence among exponentially many candidates, while HDD hierarchically minimizes it until no redundant units remain.For failed GA configurations, retained-segment priority scores guide the search toward larger, high-priority contexts likely to contain patch-critical information.
- OCD Analysis: Only 8.4% of 49.6 segments per instance are relevant on average, while method-level units comprise 78.0% of segments but have 7.5% relevance density.Removing the genetic algorithm reduces successfully minimized instances by 52.7%, underscoring its role in finding sufficient candidates.
4 SWEZZE COMPRESSION MODEL
SWEzze is a lightweight cross-encoder trained on oracle-distilled minimal sufficient contexts to identify code segments that should remain for patch-sufficient issue resolution.
- Oracle-guided data distillation: OCD produces training examples mapping overapproximate contexts to minimal sufficient subsets, represented as segment-level retention labels.Each example includes an issue, fault location, initial context, and oracle-identified minimal sufficient context.
- Model design: SWEzze fine-tunes Qwen3-Reranker-0.6B as a lightweight cross-encoder that scores whether each code segment should be retained.The model uses a structured query built from the issue description and fault location together with each candidate segment.
- Model design: The segment-level reranking formulation scores a fixed candidate pool instead of generating context from scratch.This matches the supervision, which labels individual segments as retained or discarded.
- Training: LoRA reduces trainable parameters by over an order of magnitude, making fine-tuning feasible on commodity hardware.The paper applies LoRA to the query, key, value, and output projection matrices.
- Training: Class weighting addresses the imbalance caused by oracle retention of far fewer segments than it discards.Without correction, the model could trivially predict that every segment should be discarded.
- Inference: At inference, SWEzze scores segmented contexts, handles oversized segments with overlapping windows, and assembles the result under a token budget.Sliding-window scoring prevents long functions or class bodies from being truncated before selection.
5 EVALUATION
The evaluation compares SWEzze with compression baselines on SWE-bench Verified across multiple downstream LLMs, measuring resolution, compression, cost, latency, and failure causes. SWEzze combines stable compression with lower token usage and stronger issue resolution, while analyses identify distraction, imprecise discrimination, and over-pruning as recurring failure modes.
- Evaluation setup: SWE-bench Verified contains 500 real-world GitHub issues from 12 Python repositories, spanning single-line fixes to multi-file refactorings.Each instance includes an issue description, repository snapshot, ground-truth patch, and regression test suite.
- Evaluation setup: The evaluation compares SWEzze with LLMLingua-2, LongCodeZip, SWE-Pruner, and no compression using Agentless-retrieved contexts.Agentless selects a top-10 ranked set of relevant elements for the initial context.
- Metrics: Resolution rate measures patches passing the full regression suite, while compression rate, token count, and compression time measure efficiency and overhead.Compression rate is |Cinit|/|Ĉ| in tokens, and token count includes compressed prompts plus completions.
- RQ1: Compression Efficiency and Cost: SWEzze maintains approximately 6× compression across all three LLMs, with rates of 6.03×, 5.95×, and 6.55×.Its compression is more stable than the baselines, whose rates vary more or remain below 3×.
- RQ1: Compression Efficiency and Cost: 51.8%–71.3% lower total token budgets and 4.0× faster compression than LongCodeZip show SWEzze’s efficiency trade-off.SWEzze is not the cheapest compressor in absolute terms but balances compression rate, stability, and latency.
- RQ2: End-to-End Issue Resolution: SWEzze covers 93.8%–99.2% of the union of resolved instances and adds cases beyond the strongest baseline for every model.The added instances are 16, 11, and 5 for DeepSeek-V3.2, Qwen3-Coder-Next, and GPT-5.2.
- RQ2: End-to-End Issue Resolution: SWEzze achieves the best issue resolution across all three LLMs, improving rates by 5.0%–13.0% over uncompressed and no-context settings.Relative to no compression, the reported improvements are 9.2%, 5.0%, and 8.6% for DeepSeek-V3.2, Qwen3-Coder-Next, and GPT-5.2.
- RQ3: Failure Analysis: Manual failure analysis attributes most failures to distracting retained context, imprecise discrimination among similar segments, and syntactic damage from aggressive compression.These failure modes correspond to retaining irrelevant logic, confusing similar implementations, or discarding necessary surrounding context.
6 THREATS TO VALIDITY & DISCUSSION
The evaluation transfers across model families, but OCD depends on information unavailable at inference time and defines minimality relative to the construction model and decoding parameters.
- Generalization: SWEzze generalizes across GPT-5.2, DeepSeek-V3.2, and Qwen3-Coder-Next, none of which generated the OCD training data.This cross-model setup suggests the distilled fix ingredients capture semantic dependencies rather than model-specific decoding artifacts.
- Limitations: Ground-truth patches and test coverage used during OCD search are unavailable at inference time.The deployment model must approximate oracle judgments from the issue description and retrieved code alone.
- Illustrative evidence: SWEzze removes a distracting fragment retained by SWE-Pruner, enabling correct resolution in the illustrated case.The figure uses the same visual notation as Figure 2.
- Limitations: OCD training and deployment therefore have a distribution gap between oracle inputs and the information available to the compressor.The proposed training formulation predicts which units the oracle would retain using issue and retrieved-code information.
- Limitations: Minimality is defined relative to the specific LLM and generation parameters used during data construction, so the minimal context may vary across models.The search targets minimal sufficient contexts to provide cleaner supervision with less noise.
7 RELATED WORK
Prior work retrieves and compresses context using heuristics, similarity, or general prompt-compression methods, whereas SWEzze reframes compression as oracle-guided distillation of repair-sufficient code.
- LLM Context: Long-context LLMs remain vulnerable to distraction from irrelevant context, despite partial progress on long-context handling.This weakness motivates reducing irrelevant code around issue-resolution tasks.
- Context Retrieval: Issue-resolution systems retrieve code through heuristics, embeddings, BM25, retrieval-augmented generation, or agentic code search.These mechanisms aim to provide project-specific dependencies needed for repository-level repair.
- Prompt Compression: Task-agnostic and task-specific natural-language prompt compressors remain ineffective at selecting relevant code context for issue resolution.Their compression objectives do not directly identify repair-critical code fragments.
- LLM-based Issue Resolution & Program Repair: Agentless provides the canonical issue-resolution workflow used here to evaluate frontier LLMs.The paper positions Agentless as a widely used evaluation workflow.
- Code Compression & Information Selection: Earlier code-selection methods lack sufficient fine-grained precision or are not code-specific, unlike SWEzze.SWEzze explicitly searches for context configurations that enable correct patch generation.
- Code Compression & Information Selection: SWEzze reframes compression from relevance selection to oracle-guided distillation of the minimal context sufficient for a correct patch.This approach uses oracle feedback rather than statistical proxies such as similarity.
8 CONCLUSION
The paper presents SWEzze, trained on OCD-generated minimal sufficient contexts, and reports stable compression with improved issue-resolution rates across three frontier LLMs.
- Conclusion: SWEzze combines OCD’s genetic algorithms and hierarchical delta debugging to identify minimal sufficient contexts for repair.The resulting contexts preserve ingredients required for issue resolution rather than surface-level relatedness.
- Conclusion: SWEzze achieves stable compression while improving resolution rates over previous compression baselines on SWE-bench Verified.The evaluation uses three frontier LLMs.