PaperlayerView exact revision on arXiv

Source-linked AI summary

Well-Specified and Well-Tested? Assessing SWE-bench-Style Datasets with a Self-Evolving Agent

Zhonghao Jiang, Xuyang Zheng, Yanfu Yan, Zhongxin Liu, Lingfeng Bao, Shuiguang Deng

arXiv:2610.39417v1cs.SEcs.AIcs.CL

TL;DR

SWE-dataset quality assessment is costly, and static rubrics struggle with nuanced cases. ExLabel evolves and reuses experience for clarity and coverage judgments; it achieves the best F1 across all 12 SWE-TQ settings.

  • Problem

    SWE-dataset quality depends on task clarity and test coverage, but manual assessment is costly and fixed LLM rubrics struggle with nuanced cases and accumulated context.

    Source: §1 · p. 2

  • Method

    ExLabel accumulates annotated-instance knowledge into an evolving rubric and retrieves relevant experience to assess task clarity and test coverage; SWE-TQ supports evaluation across eight languages.

    Source: §1 · p. 2

  • Results

    ExLabel achieves the best F1 in all 12 settings, with relative average-F1 gains of 44.2%/49.9% for task clarity and 24.0%/18.4% for test coverage on Python/multilingual subsets.

    Source: §1 · p. 3

  • Takeaways & Limitations

    On SWE-bench Verified, a model fine-tuned on ExLabel-filtered SWE-rebench data resolves 66.2% of issues, outperforming the best-baseline-filtered model by 14.5%.

    Source: §1 · p. 3

    The evaluation focuses on issue resolution, although that setting spans bug fixing, feature implementation, and feature optimization.

    Source: §7.2 · p. 15

Abstract

from arXiv · show

Large language models (LLMs) have shown strong performance on software engineering (SE) tasks, driving the need for high-quality datasets for reliable training and evaluation. SWE-bench-style datasets (SWE datasets) are widely used to evaluate LLM-based coding agents on real-world software issues, where task clarity and test coverage directly affect whether instances are solvable and verifiable. However, quality issues persist in existing SWE datasets, and manual annotation is costly and labor-intensive, motivating LLM-based annotation. Existing approaches typically rely on static, overly general rubrics, limiting their ability to handle diverse and nuanced scenarios and often yielding high false positive rates. In contrast, human annotators draw on accumulated experience to make context-aware judgments. Inspired by this, we propose ExLabel, an LLM-based, self-evolving, experience-guided assistant for task clarity and test coverage assessment in SWE datasets. ExLabel maintains an evolving experience bank that distills cross-instance knowledge into stable yet adaptive judgment criteria and reuses relevant experience to guide the assessment of new instances. For systematic evaluation, we construct SWE-TQ, a new benchmark spanning eight programming languages, by cleaning human annotations from SWE-bench and Multi-SWE-bench. Experiments with three backbone LLMs show that ExLabel achieves the best F1 across all settings, with relative improvements of 44.2%/49.9% and 24.0%/18.4% over the best baseline for task clarity and test coverage assessment on the Python/Multilingual subsets, respectively. Moreover, a coding model fine-tuned on SWE-rebench data filtered by ExLabel resolves 66.2% of issues on SWE-bench Verified, outperforming the model trained on data filtered by the best baseline by 14.5%.

1 Introduction

SWE-dataset quality assessment is important but costly, and fixed rubrics can struggle with nuanced cases. ExLabel accumulates and reuses experience; on SWE-TQ it outperforms the best baseline across all settings and also improves downstream training results.

1 Introduction

  • Task-quality defects can undermine dataset reliability; manual annotation is costly, while fixed generic rubrics struggle with nuanced cases and instance-level rules risk fragmentation.Source: §1 · p. 2
  • ExLabel accumulates cross-instance knowledge into an evolving judgment rubric, then retrieves relevant experience to guide new task-clarity and test-coverage assessments.Its rollout, reflection, and update cycle distills patterns from comparisons between predicted and golden annotations.Source: §1 · pp. 2–3
  • SWE-TQ retains unanimously annotated instances, adds manual cleaning, and covers eight languages across Python and multilingual subsets.It contains 1,076/1,907 task-clarity instances and 833/1,739 test-coverage instances in the Python/multilingual subsets, respectively.Source: §1 · p. 3
  • ExLabel achieves the best F1 in all 12 settings; relative average-F1 gains over the best baseline are 44.2%/49.9% for task clarity and 24.0%/18.4% for test coverage on Python/multilingual subsets.Source: §1 · p. 3
  • A model fine-tuned on ExLabel-filtered SWE-rebench data resolves 66.2% of SWE-bench Verified issues, outperforming models trained on best-baseline-filtered and unfiltered data by 14.5% and 21.7%.Source: §1 · p. 3

2 Background and Motivation

The section defines task quality through two judgments, then uses django-15869 to show how relevant experience can change an LLM’s test-coverage assessment.

2.1 Problem Definition

  • Task clarity assesses whether the description sufficiently specifies a solution, while test coverage assesses whether the suite adequately verifies correctness.Both assessments use Boolean labels and explanatory notes; task clarity cannot rely on details available only in the patches.Source: §2.1 · p. 4

2.2 Motivating Example

  • For django-15869, a test requiring an exact deprecation-warning message is inadequate because the task never specifies that wording.Without relevant prior experience, the LLM predicts adequate coverage; retrieved experience helps it identify the test’s over-specification.Source: §2.2 · p. 4

3 Approach

EXLABEL accumulates experience by comparing predictions with golden annotations and refining an experience bank. It reuses that cross-instance experience to assess new SWE instances.

3 Approach

  • EXLABEL builds an experience bank through iterative reflection on predictions against golden annotations, then reuses accumulated cross-instance experience to label new SWE instances.Source: §3 · pp. 4–5

3.1 Experience Accumulation

  • Experience accumulation uses a rollout–reflection–update cycle to extract patterns, successful strategies, and errors from labeled instances.Source: §3.1 · p. 5

3.1.1 Rollout

  • Rollout summarizes task type, task explanation, and test explanation, retrieves and reranks relevant experience with hybrid sparse BM25 and dense embeddings, then predicts labels and notes.Source: §3.1.1 · pp. 5–6

3.1.2 Reflection

  • Reflection compares predicted and golden annotations to distill transferable experience as Understanding, Successes, and Mistakes, with Mistakes reserved for failed predictions.Source: §3.1.2 · p. 7

3.1.3 Experience Update

  • An LLM maintenance procedure inserts, modifies, or deletes experience; shared IDs support deduplication, and the refreshed bank feeds the next rollout.Source: §3.1.3 · pp. 7–8

3.2 Experience Reuse

  • For new unlabeled instances, EXLABEL retrieves and reranks relevant experience from three summaries, then predicts labels and notes using selected items as context.Source: §3.2 · p. 8

4 SWE-TQ: A Benchmark for Task Clarity and Test Coverage Assessment

SWE-TQ provides human-annotated task clarity and test coverage assessments across Python and seven other languages. Its construction filters for agreement, separates accumulation and test data, and manually cleans annotations.

4.1 Benchmark Construction

  • SWE-TQ combines SWE-bench and Multi-SWE-bench human annotations into Python and multilingual subsets covering seven additional programming languages.Source: §4.1 · p. 8
  • Coverage uses the most recent 20% of instances as a temporal test set, while clarity uses a random 30% split to retain enough underspecified test instances.Source: §4.1 · p. 8
  • Manual cleaning removes instances with external-link-dependent clarity labels or tests requiring an unspecified exact error message; unresolved disagreements are excluded.The two authors achieved Cohen’s kappa of 0.928 on cleaning decisions.Source: §4.1 · pp. 8–9

4.2 Benchmark Characteristics

  • SWE-TQ contains 1,076/1,907 clarity instances and 833/1,739 coverage instances in Python/Multilingual; positive cases are especially rare for clarity.Only 6.0% and 4.5% of clarity test instances are underspecified in Python and Multilingual, respectively.Source: §4.2 · p. 9, Table 2 · p. 9
  • Single surface characteristics yield AUCs of 0.50–0.77, while all four test sets exceed their stated minimum sample sizes for 95% confidence and a 10% margin of error.Source: §4.2 · p. 9

5 Experiment Setup

The experiments compare EXLABEL with automated annotation methods and coding agents across two assessment tasks and two SWE-TQ subsets. They also evaluate filtering’s downstream training setup using SWE-bench Verified.

5.1 Research Questions

  • The four research questions cover comparative effectiveness, component ablations, downstream training effects, and how experience helps EXLABEL.Source: §5.1 · p. 9

5.2 Baselines

  • Baselines include SPICE, direct prompting, SimpleRAG, and the repository-exploring coding agents Codex and Claude Code.Source: §5.2 · pp. 9–10

5.3 Evaluation Metrics

  • F1 is the primary metric for identifying unsolvable or unverifiable instances, while FPR measures valid instances mistakenly filtered out; token use and cost measure efficiency.Source: §5.3 · p. 10

5.4 Implementation Details

  • All methods use three LLM backbones on both SWE-TQ subsets; EXLABEL retrieves ten candidates and reranks to three, and Table 3 reports overall assessment performance.Source: §5.4 · p. 10, Table 3 · p. 11

5.5 Downstream Effectiveness Analysis

  • The downstream study filters 5,000 SWE-rebench instances with each technique, fine-tunes Qwen3.5-9B-Base on retained resolved trajectories, and evaluates resolution on SWE-bench Verified.Source: §5.5 · p. 11

6 Experimental Results

EXLABEL outperforms the baselines on task clarity and test coverage, and its filtered training data improves downstream issue resolution. Ablations and a case study examine how its experience bank and retrieved guidance contribute.

6.1 RQ1: Effectiveness

  • Across all 12 task, subset, and backbone settings, EXLABEL has the best F1; gains are 44.2%/49.9% for clarity and 24.0%/18.4% for coverage on Python/Multilingual.The gains are relative to the best-performing baseline and are averaged across three backbones.Source: §6.1 · pp. 11–12
  • EXLABEL uses 6.0k–21.0k tokens per instance versus 25.6k–265.5k for Codex and Claude Code, but still removes some valid instances (FPR: 0.7%–6.6% for clarity; 6.4%–30.1% for coverage).EXLABEL’s F1 exceeds SPICE’s in all 12 settings, with lower FPR in 11.Source: §6.1 · p. 12

6.2 RQ2: Ablation

  • Removing the experience bank cuts F1 by 25.5%–35.7% for task clarity and 49.3%–84.0% for test coverage.The RQ2 summary reports a 48.6% average F1 reduction when the bank is removed.Source: §6.2 · pp. 12–13

6.3 RQ3: Downstream Effectiveness

  • EXLABEL-filtered training data yields a 66.2% SWE-bench Verified resolution rate, 14.5% above the best baseline, with 1,087 retained resolved trajectories.It improves resolution by 21.7% over Full Data while using fewer than half of Full Data’s 2,309 trajectories; cost per resolved trajectory is $0.46.Source: Table 5 · p. 13, §6.3 · p. 13

6.2 RQ2: Ablation

  • Removing reranking lowers F1 by 20.1%–34.2% and 4.3%–6.7% across the two tasks; Successes and Mistakes provide complementary guidance.Removing both fields reduces F1 by 26.5%–29.7% and 7.3%–9.3% for the two tasks, respectively.Source: §6.2 · p. 13

6.4 RQ4: Case Study

  • In the deprecation-filter case, EXLABEL’s historical guidance flags an exact-warning-message test as over-specified, unlike approaches relying on code context or repository conventions.The test could reject a valid solution that raises the same warning with a different message.Source: Figure 5 · p. 14, §6.4 · p. 14

7 Discussion

The discussion highlights recurring errors in task-clarity and test-coverage labels, then describes annotation and evaluation boundaries that qualify the benchmark’s validity.

7.1 Typical Annotation Errors in SWE Datasets

  • Task descriptions relying on inaccessible external links can be mislabeled well-specified, while tests asserting an error message absent from the task can be mislabeled adequate.The authors identify these patterns during manual cleaning and use them to construct SWE-TQ as a benchmark for automated annotation evaluation.Source: §7.1 · pp. 14–15

7.2 Threats to Validity

  • SWE-TQ retains unanimously annotated instances and receives author review, but manual labels may still contain errors; the review achieves Cohen’s kappa of 0.928.The evaluation focuses on issue resolution, which includes bug fixing, feature implementation, and feature optimization; the authors describe extension to other task settings as feasible.Source: §7.2 · p. 15

8 Related Work

Related work situates SWE-bench-style datasets as both benchmarks and training resources, and positions EXLABEL against manual and automated quality-assessment approaches.

8.1 SWE-bench-style Datasets

  • SWE-bench-style datasets use executable environments for issue-resolution evaluation; benchmark datasets draw on real repositories, while training datasets also use automated or synthetic construction.Examples include SWE-bench’s 2,294 instances from 12 repositories and synthetic-data approaches such as SWE-Smith, SWE-Synth, and BugPilot.Source: §8.1 · pp. 15–16

8.2 Quality Assessment of SWE Datasets

  • Quality concerns—including underspecified tasks and inadequate tests—persist even in manually verified data, motivating systematic assessment before datasets are used for training or evaluation.Manual annotation is common, while SPICE uses predefined criteria or instance-specific context; EXLABEL instead accumulates and retrieves cross-instance judgment experience.Source: §8.1 · p. 16, §8.2 · p. 16

9 Conclusion

The conclusion presents EXLABEL and SWE-TQ as contributions to SWE dataset quality assessment. It reports stronger F1 across settings and improved downstream resolution after filtering training data.

9 Conclusion

  • EXLABEL assesses task clarity and test coverage by extracting patterns and error patterns from historical annotations through a self-evolving loop.The authors construct SWE-TQ from cleaned SWE-bench and Multi-SWE-bench annotations, spanning eight programming languages.Source: §9 · p. 16
  • On SWE-TQ, EXLABEL achieves the best F1 in all settings, with relative gains of 44.2%/49.9% for clarity and 24.0%/18.4% for coverage on Python/Multilingual.The comparisons are against the best-performing baseline.Source: §9 · p. 16
  • EXLABEL-filtered training data improves the fine-tuned coding model’s SWE-bench Verified resolution rate by 14.5% over the best-performing baseline.Source: §9 · pp. 16–17

A Detailed Results of RQ1

EXLABEL balances precision and recall better than the compared techniques, while baseline approaches tend to be either conservative or imprecise.

A Detailed Results of RQ1

  • 52.7% average precision and 52.0% average recall let EXLABEL balance the two metrics, with recall second only to SPICE.Table 6 reports precision and recall for all techniques.Source: A Detailed Results of RQ1 · p. 23, Table 6 · p. 23
  • 14.8 percentage points is EXLABEL’s average precision–recall gap, versus 29.4–42.5 percentage points for the baselines.Source: A Detailed Results of RQ1 · p. 23
  • SPICE has the highest recall in 9 of 12 combinations and 67.4% average recall, but its 30.8% average precision is lowest.Source: A Detailed Results of RQ1 · p. 23
  • Vanilla, SimpleRAG, Codex, and Claude Code average 22.3%–36.6% recall, while EXLABEL can remove more low-quality instances and discard fewer valid ones than SPICE.Source: A Detailed Results of RQ1 · p. 23

B Impact of Historical Data Size

Experience-bank gains appear with limited historical data, while the trends differ between task clarity and test coverage; the request also includes the RQ3 training recipe.

B Impact of Historical Data Size

  • Using about 400 annotated historical instances (20%), EXLABEL improves F1 by 2.7–67.3 percentage points over w/o Experience Bank across all four task and subset combinations.Source: B Impact of Historical Data Size · p. 24
  • For task clarity, F1 shows no clear increase beyond 20% historical data; underspecified instances were 4.9% and often involved missing task-description information.Source: B Impact of Historical Data Size · p. 24
  • For test coverage, F1 rises from 77.5% to 79.4% on SWE-TQ-PYTHON and 37.3% to 40.0% on SWE-TQ-MULTILINGUAL as historical data grows from 40% to 100%.Source: B Impact of Historical Data Size · p. 24

C Training Recipe of RQ3

  • RQ3 fine-tunes Qwen3.5-9B-Base on four H100 GPUs for three epochs, with batch size 8, learning rate 5 × 10−5, and 131,072-token context.Source: C Training Recipe of RQ3 · p. 25

D Case Study for Ablation

The ablation case concerns a test patch that depends on an error string absent from the task description; EXLABEL identifies this hidden dependency.

D Case Study for Ablation

  • EXLABEL identifies the hidden error-message dependency and generates the correct prediction, using a prior mistake about error-message validation.Source: D Case Study for Ablation · p. 25
  • Without reranking, retrieved experience redirects attention from implementation dependence toward behavioral consistency and semantic correctness.Source: D Case Study for Ablation · p. 25
  • Without the Mistakes field, the model checks core behavior but concludes there is no implementation-detail dependency.Source: D Case Study for Ablation · p. 25
  • Without Successes, or with both fields removed, retrieval returns an unrelated mock-argument issue and the model misses the specific error-message dependency.Source: D Case Study for Ablation · p. 25

Related papers

Loading 2610.39417v1…