Source-linked AI summary
RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests
Gyuhyeong Kim, Hyojung Gwon, Jeonghyeon Kim, Kyuhong Shim, Sunjae Lee
TL;DR
Coding-agent benchmarks use detailed issue descriptions, while real requests are shorter, sparser, and more casual, creating a benchmark–reality gap. REALSWE measures this gap and builds controlled task variants; realistic inputs reduce resolution rates by 6.4 percentage points on average, while Desired Behavior and Motivation are especially valuable signals. The study’s conclusions are bounded by its model coverage and single-turn evaluation setting.
Problem
Real user requests differ from the long, structured, information-rich prompts commonly used by SWE-BENCH-style evaluations, limiting evidence about performance under realistic communication.
Method
REALSWE applies an information taxonomy and linguistic dimensions to real and benchmark requests, then constructs 381 multi-variant task families with controlled compositions and styles.
Results
Realistic inputs reduce resolution rates by 6.4 percentage points on average, can change model rankings, and show particularly strong effects from Desired Behavior and Motivation.
Takeaways & Limitations
Users should specify Desired Behavior for bug fixes and Motivation for feature requests, while agent interfaces can ask targeted clarification questions.
Takeaways & Limitations
The evaluation excludes several frontier coding models and uses single-turn tasks, so findings may not fully generalize to the strongest systems or interactive clarification.
Abstract
from arXiv · showhide
Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues--long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts from SWE-chat and problem statements from SWE-bench Verified and Pro. We find that requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but just 7% of benchmark problems. Furthermore, 87% of real prompts are casually written whereas 94% of benchmark problems are formal. Guided by these observations, we introduce sys, 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within each family share the same underlying task and gold patch while differing only in information composition and linguistic style. Evaluating seven contemporary LLMs with sys, we find that i) realistic inputs reduce resolution rates by 6.4 pp on average and can change model rankings. Controlled analysis further shows that ii) including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit; iii) linguistic style has only small, model-dependent effects. These findings provide actionable guidance for users and agents: explicitly stating the desired behavior and motivation--which most real prompts omit--substantially improves the LLM's software engineering performance.
1 Introduction
SWE-BENCH provides a standard for repository-level coding-agent evaluation, but its detailed, formal issue descriptions differ substantially from short, sparse real-user requests. REALSWE characterizes and benchmarks this gap, finding that realistic inputs lower resolution rates and that information content matters unevenly.
- SWE-BENCH tasks are built from executable GitHub issues and their fixes, making leaderboard scores a standard for comparing coding capabilities.
- Real user requests are typically short, informal, and sparse compared with detailed, structured benchmark issues.A request may reduce the same intent to “this crashes on empty input—fix it,” leaving agents to infer requirements.
- REALSWE introduces 381 multi-variant task families whose variants share an underlying task while varying information composition and linguistic style.It is released as REALSWE-BENCH, a fixed realistic evaluation set, and REALSWE-FRAMEWORK, a configurable framework for custom configurations and ablations.
- 6.4 percentage points: resolution rates drop on average under realistic inputs, and model rankings can change.Controlled analysis also finds uneven value across information components and only small, model-dependent effects from linguistic style.
- Desired Behavior and Motivation are valuable signals, yet only 5% of real user prompts state them.The paper presents explicitly stating these signals as actionable guidance for improving coding-agent performance.
2 Related Work
Prior benchmarks increasingly incorporate realistic developer requests, but existing approaches often lack openness or independent control over communication factors. REALSWE addresses this by varying information composition and linguistic style while holding the underlying software task constant.
- Conventional coding-agent benchmarks pair repository tasks with a single canonical issue description, leaving communication effects largely unexamined.
- Real-session benchmarks provide authentic inputs but can confound communication with task, repository, and difficulty differences.
- SAVING SWE-BENCH transforms repository tasks into user-style inputs but jointly changes multiple specification properties, limiting attribution to individual factors.
- REALSWE instead represents each task as a multi-variant family grounded in observed information compositions and linguistic dimensions.This design independently controls communication properties for more targeted evaluation.
3 Method
REALSWE constructs realistic, configurable evaluations by measuring how real and benchmark requests differ, transforming eligible benchmark problems into controlled variants, and validating the resulting pipeline. The analysis finds sharp differences in information composition and linguistic style, while the final benchmark matches real-request length.
- Construction pipeline: REALSWE’s four-stage construction defines task-specific taxonomies and linguistic dimensions, measures real–benchmark differences, creates multi-variant families, and validates each LLM-driven stage.
- Request characterization: The taxonomy describes requests by information types grounded in GitHub issue templates, while linguistic style uses Formality, Sentence type, Certainty, and Perspective.
- Benchmark–reality mismatch: 88% of real prompts contain only [P] or [PA], compared with 7% of SWE-BENCH VERIFIED and PRO problems.Real requests are therefore compositionally sparse, whereas benchmark problems commonly include reproduction, environment, and contextual information.
- Benchmark–reality mismatch: 86.8% of real requests are casual and 51.3% imperative, whereas 84.8% of VERIFIED and 100% of PRO prompts are formal.Approximately 89% of prompts in both benchmarks are declarative; certainty and perspective show no consistent separation.
- Variant construction and validation: 403 benchmark problems initially contain every required taxonomy field, and quality filtering leaves 381 task families: 192 bug fixes and 189 feature requests.The LLM judge agrees highly with human judgments for decomposition and rephrasing, with reported accuracy of 0.97 and 0.99 respectively.
- Quality and realism: 1,417 characters: REALSWE-BENCH’s average task-description length closely matches SWE-CHAT’s 1,427 characters and is below conventional benchmarks’ 1,672–2,776 characters.
4 Experiments
The experiments show that realistic, sparse requests reduce coding-agent success, alter model comparisons, and shift the value of information toward Desired Behavior and Motivation rather than linguistic style or residual context. Controlled evaluations across benchmark variants identify which request components most affect resolution and cost.
- 4.2 Main Results: The Benchmark–Reality Gap: 6.4 pp: resolution rates decrease on average when seven LLMs receive REALSWE-BENCH inputs instead of original problem statements.Absolute drops range from 4.0 pp to 8.0 pp, while six of seven models also incur higher average cost under realistic inputs.
- 4.2 Main Results: The Benchmark–Reality Gap: Model rankings are not preserved under realistic inputs, with MiMo V2.5 Pro rising from fourth to second place.Its shift relative to Qwen3.7 Plus is +3.7 pp, with a 95% CI of [+0.7, +7.3].
- 4.2 Main Results: The Benchmark–Reality Gap: Realistic inputs widen separation among top models while narrowing the overall strongest-to-weakest performance range.The top-cluster range grows from 1.7 to 3.3 pp, whereas the overall range contracts from 19.8 to 15.8 pp.
- 4.2 Main Results: The Benchmark–Reality Gap: Bug-fix resolution falls by 9.1 pp on average, compared with 3.7 pp for feature-request resolution.The analysis tests whether missing precise intended behavior explains the larger bug-fix gap.
- 4.3 Controlled Analysis of REALSWE Results: Linguistic style has only small, model-dependent effects, with no significant style contrast among the eight comparisons.Bug-fix changes average 0.0 pp and feature-request changes average −1.8 pp; Holm-adjusted p≥.35.
- 4.3 Controlled Analysis of REALSWE Results: Desired Behavior is consistently valuable for bug fixes, whereas Reproduction Steps, Environment Information, and other residual fields show little average effect.Removing Desired Behavior lowers resolution by 8.0 pp on average, while removing [A], [E], and [R] changes resolution by only 1.8 pp on average; Motivation lowers feature-request resolution by 3.4 pp on average.
5 Discussion and Limitations
REALSWE translates its findings into guidance for handling sparse requests while noting limitations in model coverage and single-turn evaluation.
- Actionable guidance for information-sparse prompts: Users should specify Desired Behavior for bug fixes and Motivation for feature requests rather than routinely writing benchmark-style issue reports.Interfaces can ask targeted clarification questions, or agents can infer missing information from the request and repository context.
- Actionable guidance for information-sparse prompts: Clarification or inference mechanisms could help agents handle information-sparse requests and improve users’ experienced performance.
- Limitations: The evaluation excludes frontier systems such as GPT-5.6, Opus 5, and Kimi K3, limiting generalization to the strongest available coding models.The evaluated DeepSeek V4 Pro and MiMo V2.5 Pro are trillion-parameter-scale models, but the paper does not include the listed frontier systems.
- Limitations: Because evaluation is single-turn, it does not capture whether iterative clarification mitigates sparse or ambiguous requests.The authors identify controlled conversational evaluation as future work.
6 Conclusion
The paper introduces REALSWE to evaluate coding agents under realistic user requests and concludes that information composition materially affects measured performance. Desired Behavior and Motivation are particularly valuable signals, supporting more realistic evaluation and better-informed interfaces.
- REALSWE is a benchmark and configurable framework for evaluating coding agents under realistic user requests.
- Realistic inputs can substantially change measured performance, while Desired Behavior and Motivation are particularly valuable signals.
- The authors hope REALSWE supports more realistic evaluation and better-informed coding-agent interfaces.
A.1 Source Benchmarks
REALSWE-BENCH combines SWE-BENCH VERIFIED and PRO tasks, selecting 381 candidates that span repositories and languages without evidence of being easier, smaller, or more concentrated than excluded tasks.
- Source benchmarks: REALSWE-BENCH derives its tasks from SWE-BENCH VERIFIED and SWE-BENCH PRO, which provide widely used repository-level coding-agent evaluations.
- Source benchmarks: SWE-BENCH VERIFIED contributes 500 human-validated tasks from Python repositories.Ninety-three professional Python developers reviewed 1,699 tasks and retained 500 well-specified tasks with appropriately scoped tests.
- Source benchmarks: SWE-BENCH PRO contributes 731 more difficult, longer-horizon tasks from larger and more complex repositories, including languages beyond Python.
- Coverage and task characteristics: REALSWE-BENCH spans 21 repositories and four programming languages, with Python accounting for 55.1% of tasks.All tasks in the three non-Python languages come from SWE-BENCH PRO.
- Coverage and task characteristics: Gold patches touch 4 files and edit 134 lines on average; feature requests average 5 files and 187 lines versus 3 files and 82 lines for bug fixes.Every evaluated model has a lower resolution rate on feature requests than on bug fixes.
- Task selection: 381 of 1,229 original tasks contain all required information categories after excluding 22 low-scoring candidates.The resulting benchmark contains 381 selected tasks and 848 excluded tasks for comparison.
- Selection bias: Selected tasks are harder (53.9% against 63.0%), larger (134 lines against 94), and more evenly distributed (13.9 effective repositories against 11.0) than excluded tasks.No repository exceeds 12.3% of selected tasks, whereas django/django accounts for 22.5% of excluded tasks.
A.5 Example Tasks
The examples illustrate REALSWE-BENCH’s construction pipeline and its coverage of task types and information compositions.
- Construction pipeline: Figure 7 depicts the input and output of the construction pipeline for one task, including an all-field variant that preserves information while changing linguistic style.The original contains all five bug-fix fields, while the [PDREA] variant uses casual, imperative, confident, non-first-person language.
- Example tasks: Figure 8 presents four tasks spanning bug fixes and feature requests crossed with [P] and [PA] information compositions.
- Construction pipeline: The construction pipeline is described in §3.3, with Figure 3 showing the whole pipeline and Appendix F validating its steps and task quality.
B.1 Task-Type Classification
The paper classifies benchmark tasks primarily as bug fixes or feature requests, then prepares them for information-field decomposition and realistic variant construction.
- Task-type distribution: 86.4% of SWE-BENCH VERIFIED tasks are bug fixes, compared with 56.5% of SWE-BENCH PRO tasks.
- Task-type distribution: Only two of 1,231 benchmark tasks are classified as other, and both are refactorings excluded from subsequent processing.
- Task-type distribution: The remaining 1,229 tasks are carried forward because bug fixes and feature requests cover essentially all benchmark tasks.
- Information preparation: The pipeline redistributes problem-statement text across taxonomy fields without rewriting its information content.
- Information preparation: For SWE-BENCH PRO, Requirements are merged with the Problem Statement, while Interface metadata is supplied separately when available.
D.1 Information Composition
Benchmark problems contain substantially more supporting information than real requests, especially Desired Behavior and Motivation, while benchmark language is also more formal and declarative.
- Information composition: 88% of real prompts contain only [P] or [PA], compared with 7% of benchmark problems.
- Information composition: Real prompts contain 1.4 information fields on average versus 2.9 for benchmark problems, and 64.2% contain only [P].
- Information composition: Desired Behavior appears in 5.4% of real requests versus 73.5% of benchmark problems, while Motivation appears in 8.9% versus 96.1%.
- Linguistic style: 86.8% of real requests are casual and 51.3% imperative, whereas 84.8% of VERIFIED and 100% of PRO prompts are formal and approximately 89% declarative.
- Analysis: Matched comparisons use the same tasks across input conditions, with task resolution defined from three repeated runs and uncertainty assessed using repository-aware procedures.
E.1.2 Robustness Across Information Contexts
Additional matched-context analyses show that Desired Behavior consistently helps, while Motivation generally helps but is more sensitive to model and context.
- Desired Behavior: Desired Behavior has a positive point estimate in every surrounding information context examined.
- Desired Behavior: Desired Behavior’s benefit is not confined to the cumulative ablation path, although individual comparison strengths vary after multiplicity correction.
- Motivation: Motivation retains a positive direction with and without [A], but shows greater uncertainty and model variation.
- Overall robustness: Across contexts, both effects preserve their direction while their magnitude and precision remain context- and model-dependent.
F.1 Pipeline Validation and Task Quality
Validation supports the construction pipeline, the resulting 381 task families, and the linguistic contrast used to model real-user requests.
- Pipeline validation: Quality-control validation removes 22 critical decomposition failures and no rephrasing failures, leaving 381 released task families.
- Pipeline validation: The pipeline tests classification, artifact cleaning, field assignment, and rephrasing for their intended transformations.
- Linguistic validation: Pairwise judgments favor rephrased requests on formality and sentence type, while certainty and perspective show weaker effects.
- Linguistic validation: Human validation preserves the primary contrast that SWE-CHAT is predominantly casual and imperative while SWE-BENCH is predominantly formal and declarative.
G Experimental Details
The experimental setup documents the models and serving configurations, then specifies LLM-based procedures for classifying, cleaning, decomposing, rephrasing, and judging GitHub issue content and prompt style.
- Model Configuration: Table 16 records each model’s snapshot, serving provider, and inference setting, including reasoning-effort or thinking-token configurations.OpenRouter-routed models carry the openrouter/ prefix; other models were accessed directly.
- LLM Prompts: The classification prompt assigns each GitHub issue exactly one type: bug, feature, or other.It prioritizes bug or feature when either classification reasonably applies.
- LLM Prompts: The cleaning prompt removes automatically inserted GitHub-template artifacts while preserving user-authored content verbatim.If no template artifacts are present, the issue is returned unchanged.
- LLM Prompts: Line-level decomposition assigns every non-blank line exactly once to predefined bug-report or feature-request fields while preserving line content.Bug reports use five fields, whereas feature requests use desired_solution, problem_motivation, and additional_context.
- LLM Prompts: Rephrasing converts structured issue fields into casual requests while preserving technical content, meaning, and intent.The instructions prefer imperative sentences and non-first-person perspectives when natural.
- LLM Prompts: The decomposition judge scores whether content belongs in its assigned field, without evaluating factual correctness, detail, writing quality, or usefulness.Non-null fields receive scores from 1 to 3 based on how consistently their content matches the field definition.