Source-linked AI summary
What Makes Software Issue Resolution Tasks Difficult for Agents?
Ebtesam Al-Haque, Brittany Johnson
TL;DR
Benchmark solve rates do not explain why software issue-resolution tasks differ in difficulty. This paper measures static features of patches, repositories, and prompts, finding that task difficulty is predictable and that structural composition can support difficulty-aware evaluation.
Problem
Aggregate agent solve rates do not explain why software issue-resolution tasks are difficult, and task difficulty has not been systematically characterized from static properties.
Method
The study extracts deterministic features from issue patches, repositories, and prompts in CoderForge-Preview to relate task structure to agent outcomes.
Results
Static task structure predicts agent success, with prompt linguistic features contributing among top predictors for mid-band tasks.
Takeaways & Limitations
Making benchmark structure observable supports comparisons of measured capabilities and difficulty-stratified evaluation of agent progress.
Takeaways & Limitations
All trajectories come from one model and scaffold, so the identified difficulty predictors may not generalize to agents with different architectures or strategies.
Abstract
from arXiv · showhide
Background. Advances in agentic systems are simultaneously, and rapidly, saturating benchmarks. Despite this often discussed phenomena, benchmark scores remain difficult to interpret due to the lack of control and characterization of task difficulty. More specifically, we currently have little understanding of what makes one task harder than another, and to what extent task difficulty is predictable from static task properties. Aims. We propose a measurement framework to investigate and systematically quantify what structural properties of software tasks correspond to agent success rates for issue resolution tasks. Method. We conducted a large scale empirical study on CoderForge-Preview, the largest open dataset of coding agent trajectories to date, by extracting features across task patch, repository and prompt. We evaluated the predictive power of each feature against task outcomes using ensemble methods, SHAP attribution, and effect size analysis. Results We found that task difficulty is substantially predictable from static features (AU C = 0.863) and is largely driven by patch fragmentation and repository scale. Prompt linguistic features become visible among top contributors for tasks in the mid-band, revealing a layered structure of difficulty. Conclusion. The difficulty of an issue resolution task is encoded in its structure. This enables static, pre-hoc difficulty estimation and lays the groundwork for difficulty-controlled benchmark construction for evaluation of agents.
1 Introduction
The paper addresses the gap between aggregate coding-agent solve rates and explanations of task difficulty. It introduces a deterministic framework using static patch, repository, and prompt properties to predict success and identify influential factors across difficulty levels.
- Motivation: Aggregate agent solve rates do not explain why individual issue-resolution tasks are difficult.The paper motivates understanding task difficulty as agentic decision-making systems become more widely deployed.
- Research questions: The study asks whether static task features predict agent success and which feature groups and individual features account for that predictive performance.These questions are evaluated through a large-scale empirical study on CoderForge-Preview.
- Measurement framework: Task difficulty is represented as agent success rate and characterized using features from the gold patch, repository, and natural-language prompt.The feature design draws on prior work concerning code change complexity, software navigation, and requirements engineering.
- Analysis approach: The authors train predictive models, ablate feature groups, and use SHAP values to determine which static features drive task difficulty.These analyses support the framework’s characterization of issue-resolution difficulty from static properties.
- Contributions: The paper provides large-scale evidence that agent success is predictable from static task structure and identifies factors associated with success across difficulty levels.The stated contributions include discussing how influential factors vary at different levels of difficulty.
2 Related Work
Related work frames issue-resolution evaluation around benchmark validity, task-difficulty modeling, and established software-engineering predictors of change complexity and requirement ambiguity. This work distinguishes itself by using deterministic structural measures rather than LLM-derived encodings.
- Benchmark evaluation: SWE-bench established repository-and-issue patch generation as the standard task for evaluating LLM-based agents.SWEBench+ and UTBoost found solution leakage or weak tests in many resolved instances, raising concerns about aggregate solve rates.
- Difficulty modeling: Item Response Theory is widely used to model task-level difficulty and reduce evaluation cost through adaptive task selection.Recent work applies IRT to agentic coding by augmenting the Rasch model with LLM embeddings and judge rubric scores across several benchmarks.
- Difficulty modeling: The proposed features are deterministic measures grounded in empirical software-engineering constructs, enabling reproducible difficulty scores without model inference.These structural features could provide cost-effective, interpretable inputs to an IRT difficulty predictor.
- Software-change complexity: Prior software-engineering research links code churn to defect density and change diffusion to whether commits are defect-inducing or clean.Diffusion refers to how scattered a change is across files and hunks, rather than raw change volume alone.
- Requirement ambiguity: Natural-language prompts function as software requirements, with coordination, referential, and attachment ambiguity identified as consequential sources of misinterpretation.Recent empirical work also reports that LLMs struggle to reliably identify ambiguous requirements.
3 Method · 3.1 Dataset
The study uses CoderForge-Preview, a large open dataset of test-verified coding-agent trajectories spanning tens of thousands of tasks and repositories. Its tasks come from three established sources, and trajectories were generated with Qwen3-Coder-480B in an OpenHands scaffold.
- 3.1 Dataset: 258K test-verified trajectories span 51K tasks across 1,655 repositories in CoderForge-Preview.The dataset is described as the largest open dataset of coding-agent trajectories to date.
- 3.1 Dataset: CoderForge-Preview draws tasks from R2E-Gym, SWE-Smith, and SWE-Rebench.The sources contribute 4,216, 37,221, and 9,764 tasks, respectively.
- 3.1 Dataset: SWE-Smith contributes 37,221 tasks, making it the largest of the three listed sources.R2E-Gym contributes 4,216 tasks, while SWE-Rebench contributes 9,764 tasks.
- 3.1 Dataset: R2E-Gym contributes 4,216 tasks and SWE-Rebench contributes 9,764 tasks to the dataset.Together, these sources complement the larger SWE-Smith collection.
- 3.1 Dataset: Trajectories were generated using Qwen3-Coder-480B, identified as one of the best-performing open-source models on agentic coding benchmarks.The model operated within an OpenHands v0.52.1 scaffold.
- 3.1 Dataset: The trajectory-generation setup used an OpenHands v0.52.1 scaffold.The supplied passage specifies the scaffold version but ends before describing the remainder of the setup.
3.2 Feature Engineering
Feature engineering models issue-resolution difficulty through three dimensions: solution complexity in the patch, navigation difficulty in the repository, and specification interpretability in the prompt. The resulting taxonomy retains 54 features for all models and includes structural, ambiguity, cohesion, testing, and documentation signals.
- Feature set: 54 retained features are used in all models after features marked for removal are dropped during feature selection.The three feature groups characterize the solution, repository environment, and natural-language specification, respectively.
- Patch features: Patch features capture solution complexity: the amount an agent must find, understand, and change to resolve an issue.These features draw on just-in-time defect-prediction constructs.
- Repository features: Repository features capture navigation difficulty by representing codebase scale, directory structure, naming ambiguity, test infrastructure, and documentation.Fault localization is required before patch generation because the agent must identify relevant files and lines.
- Prompt features: Prompt features capture specification interpretability through component load, coordination, attachment, referential ambiguity, and cohesion.The ambiguity constructs target natural-language defects that can cause misinterpretation, while cohesion measures lexical and semantic connectivity between adjacent sentences.
3.3 Feature Selection
Feature selection used iterative VIF filtering to reduce multicollinearity, producing a common feature set for all three models.
- 3.3 Feature Selection: A VIF threshold of 10 guided iterative filtering, which removed the highest-VIF feature at each step until all scores fell below the threshold.The procedure followed the cited thresholding rule throughout feature selection.
- 3.3 Feature Selection: 9 features were dropped, leaving 54 features with a maximum VIF of 7.59 for all three models.The removed features are marked † in Tables 2–4, and the retained feature set is shared across models.
3.4 Predictive Modeling
Predictive modeling used three task-level outcomes and held-out evaluation to assess issue-resolution difficulty. The analysis combined tuned classification and regression models with feature-group ablation and SHAP explanations to measure predictive signal and feature associations.
- Outcome variables: The study modeled pass_rate, any_success, and maj_success as continuous or binary task outcomes.pass_rate is the fraction of runs passing all tests, while any_success and maj_success indicate at least one successful run and majority success, respectively.
- Modeling and tuning: Models were tuned with RandomizedSearchCV using 4-fold inner cross-validation and 28 iterations per model group.ROC-AUC guided classification tuning, while negative mean squared error guided regression tuning.
- Evaluation: Held-out evaluation used AUC–ROC, PR–AUC, Brier score, F1, MCC, RMSE, MAE, and R2, with 10-fold XGBoost cross-validation used to assess stability.Classification metrics were reported separately from regression metrics, and XGBoost results included mean ± SD on the training-set cross-validation.
- Feature analysis: SHAP values ranked features by mean absolute contribution while quantifying the direction and magnitude of associations with task success.The explanations were computed for the best-performing XGBoost model and used to interpret its non-linear predictions.
- Feature analysis: Component ablation estimated the predictive signal from patch complexity, repository structure, and issue text across individual, pairwise, and full feature combinations.The ablation reported test AUC and R2 for each feature-group subset.
4 Results
Static patch, repository, and prompt features predict agent success strongly, with ensemble models outperforming linear models and explaining 40.8% of pass-rate variance. Patch complexity and repository structure provide nearly all predictive signal overall, while prompt features become more prominent for near-baseline tasks.
- Feature-group ablation: Prompt features add negligible signal after patch and repository features, with ∆AUC ≤0.002 and ∆R2 < 0.01 across outcome definitions.Prompt-only performance is near chance for binary outcomes and explains little continuous variance, with AUC = 0.599, AUC = 0.593, and R2 = 0.025.
- Feature-group ablation: Patch complexity and repository structure together achieve AUC = 0.861 and R2 = 0.401 with 27 features, within 0.002 AUC of the full model.Patch-only features reach AUC = 0.846 and R2 = 0.375, while repository-only features reach AUC = 0.839 and R2 = 0.350.
- Feature attribution: Patch deletion, hunk gaps, and repository structure dominate SHAP importance, with patch_lines_deleted having mean absolute SHAP 0.406 and the top three features explaining 29%.High feature values generally push predictions toward failure; patch × repository pairs form the leading feature interactions.
- Difficulty-band analysis: Prompt features rank among the five largest contributors for 70.3% of near-baseline tasks, compared with 26.8% of easy and 6.8% of hard tasks.Frequent contributors include pronoun density at 11.1%, conjunction-chain length at 10.1%, and competing dependents per head at 9.4%.
5 Discussion
The discussion shows how structural difficulty scores can support benchmark auditing, difficulty-stratified evaluation, and calibrated developer reliance while remaining deterministic and reference-agent-specific. These applications make benchmark composition, agent progress, and task-specific expectations more observable.
- Benchmark structural composition becomes directly observable, enabling comparisons of whether benchmarks measure the same thing and whether solve-rate gains reflect targeted capability improvements.
- Difficulty scores can audit whether a benchmark’s task distribution is structurally balanced or concentrated in a narrow region of task space.Benchmarks dominated by compact, single-hunk patches in shallow repositories under-represent broader navigation and multi-site coordination.
- Difficulty-stratified evaluation distinguishes gains on easy tasks from gains on structurally difficult tasks by reporting performance within difficulty quartiles.The model’s well-calibrated probabilities can be used as continuous difficulty estimates.
- The required features are entirely deterministic, avoiding the LLM embeddings and per-task rubric scoring required by IRT-based difficulty estimation.
- Difficulty patterns are relative to a reference agent, motivating separate per-agent difficulty profiles because agents may find different structural features difficult.Different localization strategies can change difficulty, such as navigating wide directory structures.
- Difficulty bands give developers a grounded basis for calibrating reliance by estimating how well an agent performs on tasks like the one at hand.
6 Threats to Validity
The study’s difficulty estimates face threats from pretraining overlap, stochastic outcomes, limited agent and language coverage, oracle-patch ground truth, and prompt-feature ambiguity. These limitations constrain how broadly the reported structural predictors can be interpreted.
- Agent pretraining on benchmark repositories: Pretraining overlap may inflate task success independently of structural features when benchmark repositories were included in Qwen3-Coder-480B’s public-code pretraining.CoderForge-Preview draws from public GitHub repositories, creating potential leakage between pretraining data and evaluated tasks.
- Outcome reliability and agent stochasticity: Nonzero-temperature trajectories conflate genuine task difficulty with decoding variance, which static features cannot predict.This stochastic component contributes to irreducible variance in the regression target.
- Single model group: Because all trajectories use Qwen3-Coder-480B within OpenHands, predictor importance may not generalize across agent architectures, context windows, or scaffolding strategies.Repository scale and directory breadth may partly reflect this agent’s navigation strategy.
- Python-centric sample: The predominantly Python-centric sample limits direct transfer of absolute feature importance across programming languages, despite language-agnostic underlying constructs.Python file counts and pytest-based test organization are ecosystem-specific, whereas test density, repository scale, and edit fragmentation are broader constructs.
- Gold patch as ground truth: Gold-patch features may misrepresent agent difficulty because generated patches can differ structurally from the oracle patch used as ground truth.Simple gold patches may require complex reasoning, and complex gold patches may not, although the proxy is consistent with prior defect-prediction work.
- Prompt features: Prompt features capture structural conditions for ambiguity but cannot distinguish ambiguity resolved by context from ambiguity yielding genuinely different implementations.Thus, syntactically complex issue descriptions may not correspond directly to multiple valid interpretations.
7 Conclusion
Software issue resolution difficulty is encoded in task structure: static features from the gold patch, repository, and prompt predict agent success without model inference. This supports pre-hoc difficulty estimation and structurally controlled evaluation and benchmark construction.
- 7 Conclusion: AUC = 0.863: Static, deterministic features from the gold patch, repository, and issue prompt predict agent success without model inference.The features enable difficulty estimation before task execution.
- 7 Conclusion: The framework enables pre-hoc difficulty estimation, difficulty-stratified evaluation, and structurally controlled benchmark construction.It also supports more grounded developer reliance on agent tools.