PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Pass Rate Tell the Whole Story? Evaluating Design Constraint Compliance in LLM-based Issue Resolution

Kai Yu, Zhenhao Zhou, Junhao Zeng, Ying Wang, Xueying Du, Zhiqiang Yuan, Junwei Liu, Ziyu Zhou, Yujia Wang, Chong Wang, Xin Peng

arXiv:2604.05955v1cs.SEcs.AI

TL;DR

Issue-resolution benchmarks commonly measure success through tests, but acceptable patches must also comply with design constraints that tests may not encode. SWE-Shield extracts and links constraints from development history, then evaluates patch compliance; experiments find that test success often does not correspond to design satisfaction.

  • Problem

    Issue-resolution evaluations primarily measure test-based functional correctness, although patch acceptance also depends on design constraints that may be implicit in project history.

    Source: §1 · p. 2

  • Method

    SWE-Shield extracts design constraints from pull requests, links them to issue instances, and uses an LLM-based verifier to assess patch compliance.

    Source: §1 · p. 2

  • Results

    Across most settings, functional correctness has negligible association with design compliance (Cramér’s V≤0.11), and many test-passing patches still violate constraints.

    Source: §1 · p. 2, §9 · p. 13

  • Takeaways & Limitations

    SWE-Shield supports systematic study of design alignment and fine-grained diagnosis of design failures beyond test-passing evaluation.

    Source: §9 · p. 13

    DSR and DVR approximate design satisfaction through explicit instance-level constraints and may not capture tacit or undocumented design decisions.

    Source: §7 · p. 12

Abstract

from arXiv · show

Repository-level issue resolution benchmarks have become a standard testbed for evaluating LLM-based agents, yet success is still predominantly measured by test pass rates. In practice, however, acceptable patches must also comply with project-specific design constraints, such as architectural conventions, error-handling policies, and maintainability requirements, which are rarely encoded in tests and are often documented only implicitly in code review discussions. This paper introduces \textit{design-aware issue resolution} and presents \bench{}, a benchmark that makes such implicit design constraints explicit and measurable. \bench{} is constructed by mining and validating design constraints from real-world pull requests, linking them to issue instances, and automatically checking patch compliance using an LLM-based verifier, yielding 495 issues and 1,787 validated constraints across six repositories, aligned with SWE-bench-Verified and SWE-bench-Pro. Experiments with state-of-the-art agents show that test-based correctness substantially overestimates patch quality: fewer than half of resolved issues are fully design-satisfying, design violations are widespread, and functional correctness exhibits negligible statistical association with design satisfaction. While providing issue-specific design guidance reduces violations, substantial non-compliance remains, highlighting a fundamental gap in current agent capabilities and motivating design-aware evaluation beyond functional correctness.

1 Introduction

The introduction argues that test-based issue-resolution benchmarks miss design constraints that can affect patch acceptance. It presents SWE-Shield for evaluating design compliance alongside functional correctness and reports that test success often fails to predict design satisfaction.

1 Introduction

  • Existing benchmarks primarily judge resolution by test pass rates, although real-world patch acceptance also depends on project-specific design constraints.Violating a constraint can lead to rejection even when all tests pass.Source: §1 · p. 2
  • Design-aware evaluation must recover implicit constraints from project history, link them to issues, and assess compliance in generated patches and reasoning traces.Constraints may be intertwined across review comments or distributed across multiple pull requests.Source: §1 · p. 2
  • DesignHunter extracts constraints from pull-request discussions, which SWE-Shield links to issues and checks with an LLM-based patch verifier.The benchmark has variants derived from SWE-bench-Verified and SWE-bench Pro.Source: §1 · p. 2
  • Pass Rate reaches 70.25%–75.95% on SWE-Shield_verified and up to 42.69% on SWE-Shield_pro, while DSR remains 32.64%–50.20% and violations reach 45.85%.Functional correctness has negligible association with design compliance in most settings (Cramér’s V≤0.11); issue-specific guidance reduces DVR by up to 6.35 percentage points, but residual rates remain above 30%.Source: §1 · p. 2
  • SWE-Shield combines functional correctness and design-constraint compliance to evaluate LLM-based issue resolution.The benchmark contains 495 issues from six projects associated with 1,787 design constraints.Source: §1 · p. 2

2 Motivation

The motivation section illustrates why issue resolution requires more than passing tests: patches must also respect constraints embedded in project context and review history. It identifies challenges in extracting these implicit constraints and verifying whether patches comply.

2.1 The Necessity of Mastering Design Constraints in Issue Resolution

  • A patch can pass the tests yet fail in a relevant deployment context: Django’s distinct() solution mishandled duplicates with PostgreSQL and was later replaced by Exists().The issue remained open for nearly ten years before the final pull request resolved it.Source: §2.1 · p. 3
  • The example shows that issue resolution depends on design considerations beyond the stated functional requirements, including latent dependencies and reliability concerns.Ignoring these constraints risks introducing long-term technical debt.Source: §2.1 · p. 3

2.2 The Pitfall of Evaluating AI Assistants Solely Based on Test Cases

  • Test-only leaderboards can count a patch as successful even when review evidence indicates it would not be accepted in real-world development.The SWE-bench issue example’s distinct() patch passed the provided tests but conflicted with the earlier review discussion.Source: §2.2 · p. 3
  • Design constraints span project conventions, context-dependent decisions, trade-offs, cross-cutting concerns, and implementation style, which test execution alone may not capture.The paper leaves systematic categorization of these constraints for future research.Source: §2.2 · pp. 3–4

2.3 The Challenges of Acquiring and Verifying Implicit Design Constraints

  • Constraint extraction is difficult because review discussions entangle multiple considerations and scatter a single concern across comments or pull requests.A proposed solution may also lack an explicit rationale in the comment where it appears.Source: §2.3 · p. 4
  • Design constraints are non-executable and context-dependent, so verification requires semantic comparison rather than a traditional test oracle.The same design decision may be realized through different code structures.Source: §2.3 · p. 4

3 Construction of SWE-Shield

SWE-Shield turns implicit design reasoning in code reviews into structured constraints linked to issue-resolution tasks, then evaluates patches for compliance. DesignHunter extracts and aggregates the constraints, while separate association channels link them to issues and an LLM judge assesses patches.

3 Construction of SWE-Shield

  • SWE-Shield represents design guidance as problems, alternative options, rationales, applicable conditions, and code references, covering project conventions and contextual trade-offs rather than pure bugs or style rules.Examples include error-handling protocols and API consistency.Source: §3 · p. 4

3.1 DesignHunter: A Two-stage Design Constraint Extraction Approach

  • DesignHunter decomposes review threads into atomic suggestions, then recomposes related suggestions into generalized constraints grounded in recurring design reasoning.The two-stage process is intended to retain context while moving from individual comments to higher-level guidance.Source: §3.1 · p. 4

3.1.1 Stage I: Atomic Design Suggestion Extraction.

  • DesignHunter uses sliding comment windows and checks suggestions against pull-request changes to identify adopted guidance and preserve reference code and applicable conditions.Non-adopted suggestions remain available as evidence for alternative options, trade-offs, or conditions in later synthesis.Source: §3.1.1 · pp. 4–6

3.1.2 Stage II: Hierarchical Design Constraint Aggregation.

  • DesignHunter hierarchically clusters suggestions using semantic and development-history similarity, then merges or splits child constraints while preserving their meaning and traceability.Semantic similarity receives greater weight than structural dependency, and synthesized options must trace back to original suggestions and code.Source: §3.1.2 · pp. 6–7

3.2 Issue–Constraint Association Identification

  • Issue–constraint association combines explicit links from an issue’s resolving pull request with semantic matching against repository-wide constraints when direct traceability is unavailable.Semantic matching compares constraint descriptions with LLM-generated change intents from the resolving patch.Source: §3.2 · p. 7

3.3 An Accompanying Patch Verifier: LLMs-as-Judge for Design Satisfaction

  • A three-model LLM judge labels patch–constraint pairs as Satisfied, Violated, or Neutral after checking applicability, with majority voting determining the final label.Neutral captures constraints considered but not applicable to the concrete patch changes.Source: §3.3 · p. 7

4 Implementation

The implementation builds SWE-Shield from SWE-bench-Verified and SWE-bench-Pro, filtering and validating issue–constraint links. The resulting benchmark covers 495 issues and 1,787 high-quality constraints; manual checks assess the extractor and patch verifier, and the two variants differ in patch size and language scope.

4.1 Construction Pipeline

  • 618 candidate issues from SWE-bench-Verified and SWE-bench-Pro were selected from repositories with more than 40 associated issues, then augmented with constraints extracted from related pull requests.The initial extraction yielded 10,885 constraints: 4,695 from Verified and 6,190 from Pro.Source: §4.1 · p. 8

4.2 Manual Validation

  • 90.4% of 374 sampled extracted constraints were verified as valid, with Cohen’s kappa of 0.74; association annotators achieved kappa of 0.7783.The association review retained 1,787 high-quality issue–constraint associations.Source: §4.2 · p. 8
  • 495 issues and 1,787 high-quality design constraints remain in SWE-Shield after manual review; the preceding association stage linked 2,458 constraints to 648 issues.Issues without valid constraints were excluded.Source: §4.2 · p. 8, §4.1 · p. 8, §4.3 · p. 8
  • Three LLM judges agreed on at least two votes in 95.25% of cases, while verifier judgments matched human annotations at 80.8% with Cohen’s kappa of 0.7934.The human comparison sampled 318 patches from 1,842 agent-generated patches.Source: §4.2 · p. 8

4.3 Benchmark Characteristics.

  • SWE-bench-Verified averages 12.99 changed lines and is Python-only, whereas SWE-bench-Pro averages 197.92 changed lines and spans multiple languages.Verified has 303 of 306 issues within 0–99 changed lines; Pro has a maximum patch size of 2,028 lines.Source: §4.3 · p. 8

5 Empirical Study

The study evaluates state-of-the-art issue-resolution agents on functional correctness and design satisfaction, then examines their association, model differences, and constraint-guided refinement. It measures correctness with Pass Rate and design satisfaction with DSR, DVR, and DNR.

5.1.1 Studied LLM-based Agents.; 5.1.2 Evaluation Metrics.

  • The study evaluates SWE-agent, Live-SWE-agent, Lingxi-v1.5, and Sonar Foundation Agent, using Pass Rate for correctness and DSR, DVR, and DNR for design outcomes.The agents use competitive frontier models including Kimi-K2, GPT-5, Claude-Sonnet-4.5, Gemini-2.5-Pro, and Gemini-3.0-Pro; DNR marks cases where no associated constraint applies.Source: §5.1.1 · p. 9, §5.1.2 · p. 9

5.2 RQ1: Effectiveness in Design-Aware Resolution

  • DSR remains low despite strong Pass Rates: 39.13%–50.20% on SWE-Shield_pro and 32.64%–42.80% on SWE-Shield_verified, while Pass Rates reach 42.69% and 75.95%, respectively.The study reports a substantial drop from Pass Rate to patches that both pass tests and satisfy design constraints.Source: §5.2 · p. 9, Table 2 · p. 10

5.3 RQ2: Correlation between Correctness and Satisfaction

  • Test outcomes and design judgments show negligible association (Cramér’s V≤0.1157), with chi-square tests not significant in most settings.Many patches pass tests while violating applicable constraints, including security, error-handling, and maintainability requirements.Source: Table 3 · p. 10, §5.3 · p. 10

5.4 RQ3: Comparison across Foundation Models

  • Claude-Sonnet-4.5 has the highest DSR (50.20%) and lowest DVR (37.15%); other models have DSR of 38.34%–41.50% and DVR of 39.92%–45.85%.Pass Rates vary from 13.83%–43.87% across models on SWE-Shield_pro, while DSR differences are modest.Source: §5.4 · p. 10
  • Across four models, 35 of 377 distinct violated constraints (9.3%) are shared, while the largest model-specific set contains 48 constraints (12.7%).The shared example in Django #13410 is a constraint to catch only BlockingIOError in lock(), rather than broad OSError exceptions.Source: §5.4 · pp. 10–11, Figure 3 · p. 10, Figure 4 · p. 11

5.5 RQ4: Investigation on Design Satisfaction Improvement

  • DVR reductions vary by model: Claude-Sonnet-4.5 falls from 37.15% to 30.80%, while Gemini-2.5-Pro falls from 39.92% to 35.27%.The authors report that models differ in how effectively they internalize and operationalize identical design guidance.Source: §5.5 · p. 11
  • Only GPT-5 slightly improves Pass Rate after refinement; the other models regress relative to their initial rates.The authors describe a trade-off between design compliance and functional correctness during refinement.Source: §5.5 · p. 11
  • Explicit design-constraint guidance reduces DVR across evaluated agents, but post-refinement DVR remains above 30%.In the Django #50909 example, the refined patch recognizes existing min/max implementations but still sometimes reimplements logic locally instead of following the repository-preferred choice.Source: Figure 5 · p. 11, §5.5 · p. 11, Figure 6 · p. 12

6 Discussion

The discussion identifies limits in scaling constraint extraction, in the design quality of gold patches, and in LLM-based verification, then outlines requirements for design-aware resolution.

Scalability Challenges in Constraint Extraction for Large Codebases

  • Large repositories’ extensive histories make constraint extraction costly and traceability difficult, potentially requiring aggregation, incremental extraction, or selective sampling.The challenge includes thousands of pull requests, long review threads, and evolving conventions in enterprise-scale codebases.Source: §6 · p. 12

Limitations in “Gold” Patches Regarding Design Considerations

  • Some developer-approved gold patches pass functional tests without fully adhering to relevant design constraints.This exposes a limitation of benchmarks that derive correctness primarily from tests based on gold patches.Source: §6 · p. 12

Opportunities for Formal Verification of Design Compliance

  • LLM-based design-satisfaction verification could be complemented by formal methods for stronger guarantees on certain constraint classes.Proposed complements include static checkers synthesized with LLM support, type-based reasoning, and model checking, especially in high-assurance contexts.Source: §6 · p. 12

Requirements for Effective Design-Aware Issue Resolution Techniques

  • Design-aware resolution systems must reason about design rationale, alternative solutions, and context-specific applicability conditions.The discussion calls for combining semantic reasoning over constraints with structured knowledge representations to guide patch generation.Source: §6 · p. 12

7 Threats to Validity

The study’s validity is constrained by stochastic and LLM-dependent judgments, limited experimental coverage, and the fact that explicit constraints only approximate design satisfaction.

Internal Validity

  • LLM output stochasticity and using LLMs for both constraint extraction and satisfaction judgment threaten internal validity.The study fixes decoding temperature at zero and manually inspects extracted constraints and a sample of judgments; multiple annotators, guidelines, and agreement measures support consistent human evaluation.Source: §7 · p. 12

External Validity

  • Experiments cover two datasets, two repositories, and three representative agents built on three state-of-the-art foundation models, limiting generalization evidence.The authors note that evaluation across more datasets, repositories, and model or agent variants would provide stronger evidence.Source: §7 · p. 12

Construct Validity

  • DSR and DVR approximate design satisfaction because explicit constraints from discussions and related artifacts may miss tacit or undocumented decisions.The paper nevertheless treats compliance with explicit project-specific constraints as a necessary condition for high-quality patches and a meaningful indicator beyond functional correctness.Source: §7 · p. 12

8 Related Work

Prior work mines design knowledge and expands software-engineering benchmarks, but the paper distinguishes its approach by extracting project-native constraints and evaluating compliance with them.

8.1 Design Knowledge Extraction

  • Prior mining approaches recover design decisions and rationales [29] [30], while DesignHunter models constraints, including when they apply, by consolidating fragmented signals into code-grounded representations.This supports traceability between design reasoning and implementation evidence.Source: §8.1 · pp. 12–13

8.2 LLM Benchmarking in Software Engineering

  • Repository-level benchmarks have grown more realistic, yet evaluation still centers on test pass rates; checklist-based quality signals often miss project-specific decisions.This work instead mines constraints from code review discussions and assesses whether agents recognize and comply with them.Source: §8.2 · p. 13

9 Conclusion

The paper presents SWE-Shield to evaluate design-aware issue resolution by making implicit constraints explicit, traceable, and measurable. Its results show that test success and design alignment can diverge, with violations persisting despite design guidance.

9 Conclusion

  • SWE-Shield extracts constraints from historical pull requests and code reviews, grounds them in issue-resolution tasks, and evaluates design-aware resolution.The benchmark makes implicit, project-specific design constraints explicit, traceable, and measurable.Source: §9 · p. 13
  • State-of-the-art agents often pass all tests while violating design constraints, and functional correctness shows little statistical dependence on design compliance.Surfacing relevant design knowledge can reduce violations, but many remain.Source: §9 · p. 13
  • The findings point to a need for design-aware evaluation and techniques that help models operationalize implicit design rationale.The paper frames design-respecting behavior as a first-class objective beyond test passing.Source: §9 · p. 13

Related papers

Loading 2604.05955v1…