Source-linked AI summary
SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents
Xin He, Yanlin Wang, Mingwei Liu, Jiachi Chen, Hongyu Zhang, Guanbin Li
TL;DR
Existing repository-level benchmarks mainly test functional correctness and omit review-derived acceptance constraints that affect whether patches are acceptable. SWE-Gate derives such constraints from real reviews, builds repository-level tasks with separate tests, and evaluates both dimensions. Across its benchmark, many functionally successful repairs fail constraint validation, showing that functional-only evaluation overestimates complete repair capability.
Problem
Existing repository-level benchmarks largely measure functional test success, but not whether repairs satisfy additional review-derived acceptance constraints.
Method
SWE-Gate derives constraints from maintainer reviews and constructs repair instances with separate functional and constraint tests, non-compliant patches, and gold patches.
Results
Across four models, 644 patches pass functional tests, but only 423 pass both suites, leaving 221 hidden failures.
Takeaways & Limitations
Functional success alone does not establish that an agent-generated repair satisfies the full repository requirements or is acceptable for integration.
Takeaways & Limitations
FSR does not improve with explicit constraint guidance and decreases by 3.3–9.9 percentage points for the other models; these results are an observed controlled-ablation trade-off.
Abstract
from arXiv · showhide
Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived acceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software development. We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation between issue resolution capability and review constraint compliance. We construct SWE-Gate with 303 repository-level repair instances spanning 75 open-source Python repositories across diverse software domains. Experiments with four LLM backends spanning different capability levels under a common coding-agent scaffold reveal a substantial gap between functional success and success under the complete repair specification: among 644 repairs that pass the functional tests, 221 fail to satisfy the provided review constraints. These findings show that functional-only evaluation overestimates agents' ability to satisfy the full requirements of repository-level repair tasks. The replication package including code, data, and experimental results is available at https://github.com/DeepSoftwareAnalytics/SWE-Gate.
Introduction
Existing repository-level benchmarks mainly test functional correctness, although real code review also imposes objectively testable acceptance constraints. SWE-Gate addresses this gap by evaluating both dimensions separately and showing that functional success can overstate complete repair success.
- Motivation: Repository-level benchmarks typically treat passing functional tests as repair success, even though accepted patches may need additional review-derived requirements.Examples include backward compatibility, exception semantics, and repository-specific implementation conventions.
- Motivation: A Pydantic repair illustrates the distinction: the initial implementation delivered conditional serialization but changed a public interface, prompting a backward-compatible redesign.Separate functional and compatibility tests can distinguish behavior added from preservation of the existing schema.
- SWE-Gate: SWE-Gate evaluates review constraint compliance alongside functional correctness using separate executable tests derived from maintainer reviews.Each instance includes a functional-test-passing non-compliant patch and a gold patch passing both test suites.
- Implication: Functional success does not establish that an agent-generated repair satisfies the full set of repository requirements or is acceptable for integration.The benchmark is designed to expose this gap through explicit joint evaluation.
- SWE-Gate: 303 repair instances span 75 repositories and six software domains, supporting evaluation across diverse repository contexts.The benchmark uses a common Mini-SWE-Agent scaffold to characterize constraint following across four LLM backends.
Related Work
Prior software-engineering benchmarks expanded from function-level tests to repository-level repair and broader settings, but they still primarily evaluate tested functional behavior. Related studies also show that test weakness and contextual assumptions limit what passing tests establish.
- Benchmark evolution: Early benchmarks evaluated function-level programs, while RepoBench, RepoCoder, CrossCodeEval, and SWE-bench incorporated repository context or real GitHub issue repair.Later work extended languages, enterprise projects, repository evolution, multimodal issues, freshness, contamination control, and security.
- Benchmark evolution: SWE-bench motivated repository-level coding agents by formulating repair as resolving real GitHub issues with executable tests.Examples of subsequent agents include SWE-agent, OpenHands, and Agentless.
- Evaluation limits: Passing available tests may not establish patch correctness because weak test suites can admit overfitting repairs.This issue has also been observed in SWE-bench evaluations.
- Evaluation limits: Code reviews address maintainability, consistency, compatibility, and other engineering concerns beyond functional defects.These concerns motivate evaluating acceptance constraints in addition to functional behavior.
SWE-Gate Benchmark
SWE-Gate constructs repository-level repair instances by transferring review-derived constraint seeds into compatible code contexts and validating functional and constraint behavior separately. Its schema and validation artifacts make constraint violation, separability, and satisfiability explicit across diverse repositories and engineering concerns.
- Construction strategy: SWE-Gate uses a constraint-first strategy that abstracts engineering intent from real reviews and instantiates it in compatible repository contexts rather than appending rules to arbitrary issues.Seed repositories provide review knowledge, while instance repositories provide executable target contexts.
- Constraint extraction: The pipeline reconstructs issue–pull request artifacts, extracts atomic review suggestions, selects verifiable constraint seeds, transfers them, and validates instances through executable tests and quality assurance.The extraction process preserves review, issue, code, and diff evidence while avoiding unstated rules.
- Benchmark instance: Each benchmark instance combines an issue description, mutant patch, functional test, constraint description, constraint test, non-compliant patch, and gold patch.The non-compliant patch passes the functional test but violates the constraint, whereas the gold patch passes both.
- Validation: Validation requires the original repository to pass F, the mutant to fail F, the non-compliant repair to pass F and fail C, and the gold repair to pass both tests.This matrix establishes that functional repair and constraint compliance are separable yet jointly satisfiable.
- Instance synthesis: Candidate synthesis anchors are rejected when the constraint lacks natural motivation, every functional repair would satisfy it, or the two dimensions cannot be validated separately.The agent first explores source files, tests, abstractions, and implementation conventions to locate a compatible anchor.
- Dataset characteristics: 303 instances cover 75 open-source Python repositories and multiple software domains, while the taxonomy includes error semantics, schema or metadata, compatibility, performance, idempotence, and resource lifecycle requirements.Error Semantics occurs in 152 instances (50.2%), and Schema / Metadata / Typing occurs in 143 (47.2%).
Evaluation
SWE-Gate evaluates repository-level repairs with separate functional and review-constraint tests, measuring both issue resolution and compliance with additional acceptance requirements. Across controlled evaluations, functional success consistently exceeded joint success, while explicit constraint guidance improved compliance and joint outcomes but did not uniformly improve functional repair.
- Evaluation design: 303 repository-level instances across 75 repositories were evaluated with functional and constraint suites under a common agent scaffold.The study used four LLM backends and discarded generated test-file changes before evaluation.
- Evaluation design: SWE-Gate reports Functional Success Rate, Constraint Following Rate, and Joint Success Rate to separate issue resolution from review-constraint compliance.FSR measures functional passes over all instances, CFR measures constraint passes among functional successes, and JSR requires both.
- Overall performance: 74.9% was GPT-5.5’s highest Functional Success Rate, while GPT-4o-mini resolved 9.2% of instances.DeepSeek-V4-Flash reached 66.7% and GPT-5.4-mini 61.7%.
- Overall performance: 221 of 644 functional successes failed constraint validation, producing a 34.3% Hidden Failure Rate across models.Only 423 patches passed both suites; the Hidden Failure Rate ranged from 29.5% for GPT-5.5 to 53.6% for GPT-4o-mini.
- Constraint guidance: Providing the constraint increased Joint Success Rate for every model, with GPT-5.5 rising from 41.3% to 52.8%.Across all four models, joint successes increased from 360 to 423.
- Constraint guidance: Explicit constraint guidance improved Constraint Following Rate by 10.2–25.6 percentage points, but functional success declined for three models and slightly decreased for GPT-5.5.The authors characterize this as an observed trade-off under a one-generation-per-condition ablation rather than a general causal claim.
- Constraint categories: Constraint-following difficulty varied by category: Scope Generalization, Lifecycle Cleanup/Resource, Encoding/Escaping/Quoting, and Schema/Metadata/Typing had lower CFRs than Sentinel Distinction and Ordering/Argument Preservation.Category comparisons are descriptive because categories overlap and some are small.
Conclusion
SWE-Gate evaluates review-constraint compliance separately from functional correctness using tasks derived from real pull-request reviews. Across 303 instances from 75 Python repositories, functional-only evaluation overestimates agents’ ability to satisfy complete repair requirements, while future extensions must address broader languages and non-executable review requirements.
- SWE-Gate derives review constraints from real pull-request reviews and evaluates their compliance separately from functional correctness.
- 221 of 644 functionally successful repairs fail to satisfy the provided constraints.
- The benchmark covers 303 repository-level instances from 75 Python repositories.
- Future work should extend SWE-Gate beyond Python and develop reliable methods for review requirements that cannot yet be expressed as executable tests.