PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction

Hexuan Deng, Yue Wang, Wenyu Jiang, Cheng Yang, Haolin Yang, Zhaohua Zhang, Chenchen Zhao, Beiduo Chen, Muxi Chen, Sa Zhu, Geyuan Zhu, Jianhuan Zhuo, Qiuyong Xiao, Tianwen Jiang, Jihong Zhang, Xuebo Liu

arXiv:2610.11559v1cs.CLcs.AIcs.MAcs.SE

TL;DR

Existing coding-assistant benchmarks incompletely cover long-horizon repository work and realistic multi-turn interaction. SWE-JOURNEY synthesizes long tasks and uses persona-conditioned simulation; performance varies sharply by persona, with Non-coder results far below Software Architect results. The authors identify asking right, finding right, and fixing right as key interaction capabilities.

  • Problem

    Existing benchmarks only partially evaluate long-horizon repository development together with multi-turn interactions that adapt to incrementally disclosed requirements and user feedback.

    Source: §1 · p. 1

  • Method

    SWE-JOURNEY synthesizes long-horizon coding tasks with a weak-to-strong pipeline and conditions a user-simulation agent on four personas mined from real interaction data.

    Source: §1 · p. 2

  • Results

    Mean Final F2P is 78.50% for Software Architects and 23.00% for Non-coders, a gap of 55.50 percentage points.

    Source: §4.1 · p. 8

  • Takeaways & Limitations

    Asking right, finding right, and fixing right are key capabilities for effective coding assistance in realistic interactions.

    Source: §5 · p. 10

    The user simulator approximates rather than fully captures human complexity, and task synthesis plus main experiments cost $16k, excluding pilot runs and analysis.

    Source: §5 · p. 10

Abstract

from arXiv · show

Coding assistants such as Claude Code and Codex have become a major application of LLM agents, yet existing benchmarks remain far from real-world use, particularly in task horizon and interaction length. Code assistants require completing long chains of development work in continuously evolving repositories, while repeatedly clarifying requirements and adapting implementations through multi-turn interaction. To address these gaps, we introduce SWE-Journey, a benchmark for more realistic evaluation of coding assistants. To address the task-horizon gap, we propose a weak-to-strong synthesis pipeline that automatically constructs long-horizon coding tasks. To address the interaction gap, we mine four representative user personas from real interaction data and build a user-simulation agent to reproduce realistic code-assistance interactions. On average, models pass over 75% of tests for requested functionality with software architects, but fewer than 25% with non-coders. These results show that current coding assistants still fall short of enabling reliable coding for non-coders. We further analyze the reasons for this gap and identify asking right, finding right, and fixing right as key capabilities during interaction.

1 INTRODUCTION

Realistic coding-assistant evaluation must combine long-horizon repository work with multi-turn interaction that adapts to users’ changing requirements and feedback. SWE-JOURNEY targets this joint gap with synthesized tasks and persona-conditioned simulation.

1 INTRODUCTION

  • Long-horizon coding work spans substantial sequences of changes across files and modules, with later steps building on earlier ones.Users also disclose requirements incrementally, give feedback, leave requests underspecified, and revise goals in response to intermediate results.Source: §1 · p. 1
  • Existing benchmarks cover the two demands separately: long-horizon tasks often reveal full requirements upfront, while interactive benchmarks often use shorter sequences and coarser user models.Their joint evaluation of stateful repository development and realistic multi-turn interaction remains underexplored.Source: §1 · p. 1
  • SWE-JOURNEY uses a scalable weak-to-strong pipeline to synthesize long-horizon tasks and mines four recurring personas from real interactions to condition its user simulator.The pipeline generates follow-up requirements beyond public seeds to reduce direct solution leakage.Source: §1 · p. 2
  • Models pass fewer than 25% of functionality tests on average with Non-coder users, and fine-grained analyses identify asking right, finding right, and fixing right as key capabilities.The benchmark combines long-horizon tasks and realistic multi-turn interaction in a single conversation.Source: §1 · p. 2

2 RELATED WORK

Prior work extends coding benchmarks along either interaction length or task horizon, but a coding-assistance-specific persona taxonomy and sufficiently long interaction horizons remain missing. Constructing long-horizon tasks from scratch is also costly and difficult to scale.

Multi-turn coding benchmarks

  • Interactive coding benchmarks range from scripted rounds and progressively revealed tasks to adaptive feedback and reactive or simulated users, but often lack sufficiently long horizons and coding-specific persona definitions.The authors address that gap by mining real interactions for four representative personas and synthesizing longer tasks that elicit more turns.Source: §2 · p. 3

Long-horizon coding benchmarks

  • Long-horizon benchmarks cover feature development, software evolution, chained changes, and project-scale implementation, but do not by themselves supply realistic multi-turn interaction.The cited work spans repository-level features, upgrades, regression risks, and full-program reimplementation.Source: §2 · p. 3
  • Constructing long-horizon tasks from scratch typically requires substantial expert involvement, motivating automated synthesis to lower cost and reduce leakage risk.Even labeling a short SWE-Bench-style instance is estimated to cost approximately $100 per instance.Source: §2 · p. 3

3 METHOD

SWE-JOURNEY combines automated long-horizon task synthesis, persona-driven user simulation, and online testing in a persistent environment. Its method builds tasks and interactions through staged generation, then evaluates functionality and regression behavior across task stages and at session end.

3.1 DESIGN PRINCIPLES

  • Weak-to-strong synthesis decomposes long-horizon task construction into smaller subtasks and accumulates reusable module-level skills, lowering synthesis cost from roughly $100 for manual labeling to $56.64 per task.The strategy is intended to make each synthesis step manageable while generating follow-up requirements rather than reusing public issues.Source: §3.1 · p. 3
  • The design models user variation through mined persona dimensions and evaluates coding assistants online, reflecting sequential edits to a working directory rather than offline patch submission.Online evaluation also aims to avoid destructive operations or substantial environmental changes.Source: §3.1 · pp. 3–4

3.2 LONG-HORIZON CODING TASK SYNTHESIS

  • A stronger model plans repository-grounded subtasks, and a weaker model implements each through request and acceptance-criteria generation, reference patches, F2P tests, and P2P regression tests.The plan must exceed its seed task on at least two of four dimensions: context scope, change scope, requirement depth, and knowledge breadth.Source: §3.2 · pp. 4–5
  • Three progressively harder plan stages extend each task, while rubric-based semantic checks and execution checks gate generation and trigger retries or a return to planning.F2P tests must fail before and pass after a patch; P2P tests must pass both before and after.Source: §3.2 · p. 5

3.3 MULTI-TURN INTERACTION VIA PERSONA-DRIVEN USER SIMULATION

  • Mining real interaction logs yields five dimensions—expertise, detail preference, control style, goal stability, and verification style—and four representative groups: Non-coders, Product Managers, New Developers, and Software Architects.The four groups are selected by programming expertise and interaction categories more common than the population average.Source: §3.3 · p. 5
  • The persona-conditioned simulator uses repository and test tools, tracks per-subtask status and user confidence, and scores and revises messages for naturalness, disclosure pacing, coherence, and role boundaries.A confidence score reaching zero ends the session.Source: §3.3 · p. 6

3.4 REALISTIC EVALUATION DURING EXECUTION

  • Testing runs in a persistent Docker container during interaction and after the session; Proc. F2P, Final F2P, and Final P2P measure stage completion, final functionality, and retained functionality.Turn/Session and Cost/Session describe interaction length and estimated assistant cost.Source: §3.4 · p. 6
  • The benchmark contains 30 instances and 594 subtasks, with 6,332 F2P and 4,166 P2P nodes; its correctness checks assess test solvability, task coverage and authenticity, and interaction authenticity.Reference patches average 10.2 files and 768.3 changed lines, while evaluation averages $56.64 per instance.Source: §3.4 · pp. 6–7

4 EXPERIMENTS

SWE-JOURNEY evaluation finds low functional completion overall and a sharp performance gap across user personas. The analysis distinguishes asking, locating, and implementing fixes as separate interaction capabilities.

4.1 RESULTS

  • Sessions average 18.28 turns and 19.8 subtasks, whose patches touch 10.2 files and change 768.3 lines on average.The Software Architect elicits 25.92 turns per session.Source: §4.1 · p. 7
  • All models score below 50% on Proc. F2P, and the highest Final F2P is 52.22%, while Final F2P exceeds Proc. F2P by 3.53 percentage points on average.Stronger models average above 90% on functionality retention, but some models remain below 80%.Source: §4.1 · p. 8
  • Mean Final F2P reaches 78.50% with the Software Architect but only 23.00% with Non-coders, a 55.50-percentage-point gap on the same instances.For each of eleven models, single-shot Final F2P is below the Software Architect result but above the other three persona results.Source: §4.1 · p. 8
  • GLM-5.2 and KIMI-K3 approach the strongest proprietary models, while HY3 scores 54.33 at $0.65 per session, approximately 1/65 of CLAUDE-OPUS-5’s cost.HY4-PREVIEW ranks third with an Average of 63.29 at an estimated $5.46 per session.Source: §4.1 · p. 8

4.2 FINE-GRAINED CAPABILITY BREAKDOWN

  • Most models solicit user input on fewer than 20% of turns, while HY4-PREVIEW and CLAUDE-OPUS-5 ask more often and achieve higher test-node pass rates.Ask% measures turns soliciting input; Rel% measures turns with questions relevant to the task’s main line.Source: §4.2 · p. 8
  • Stronger models tend to rank higher across reported interaction metrics and elicit higher user confidence.The analysis tracks eliciting requirements, locating code, and implementing changes, using an LLM as a judge because manual annotation is costly and expertise-intensive.Source: §4.2 · p. 8, Table 3 · p. 9
  • The weakest models approach 60% off-target localization turns, and weaker models may grow more confident in a wrong direction.Finding Right improves as personas become more cooperative, suggesting ambiguous requests make localization harder.Source: §4.2 · p. 9
  • QWEN and DEEPSEEK skip problems on 16–17% of turns and critical problems on 13–14%; closed-source models are excluded because reasoning traces are inaccessible.The skip metrics capture dismissed problems without grounds and, for Crit%, only those that would break a task requirement.Source: §4.2 · p. 9

5 CONCLUSION

SWE-JOURNEY combines automatically synthesized long-horizon tasks with persona-based user simulation for more realistic coding-assistant evaluation. The paper reports that assistants struggle particularly with users of limited technical expertise and identifies three key interaction capabilities, while noting cost and simulation limitations.

5 CONCLUSION

  • SWE-JOURNEY evaluates coding assistants in long-horizon, multi-turn settings using weak-to-strong task synthesis and persona modeling with user simulation.The conclusion identifies asking right, finding right, and fixing right as key capabilities for effective coding assistance.Source: §5 · pp. 9–10
  • Task synthesis and main experiments cost $16k, and persona-conditioned simulation cannot fully capture human complexity.Generated tasks may also inherit biases from underlying repositories and synthesis models.Source: §5 · p. 10

A.1 AGENTIC CODING DATA SYNTHESIS.

  • Existing synthesis methods mostly produce isolated bug fixes or one-shot repository construction, while SWE-JOURNEY generates follow-up requirements without copying public issues or fixes.Earlier approaches relying on public artifacts limit diversity and raise leakage risk.Source: §A.1 · p. 16

A.2 TASK DIMENSION, KNOWLEDGE COVERAGE, AND TASK ANGLES

The appendix characterizes task difficulty and coverage with four task dimensions and develops taxonomies for knowledge domains and task angles. It also defines four data-informed simulated-user personas across five interaction dimensions.

A.2 TASK DIMENSION, KNOWLEDGE COVERAGE, AND TASK ANGLES

  • Tasks are characterized by context scope, change scope, requirement depth, and knowledge breadth, which guide difficulty escalation.Context and change scope each use five ordered levels; requirements can carry multiple labels, and knowledge breadth counts distinct domains per task.Source: §A.2 · p. 16
  • Knowledge-domain and task-angle taxonomies are derived from benchmark content through LLM labeling, balanced clustering, deduplication, and human confirmation.Their categories provide a source-grounded basis for measuring coverage against the source pool.Source: §A.2 · pp. 16, 18
  • Construction promotes coverage through greedy set-cover seed selection and planning-stage sampling from a frozen 20-angle distribution.The seed selection favors rare labels and limits each repository to three instances; the sampler caps defect-fixing weight and floors the two rarest angles.Source: §A.2 · p. 18
  • The 30 synthesized cases cover 48 of 56 source-attested knowledge domains (85.7%).They preserve most technical coverage across the source benchmarks.Source: §A.2 · p. 18
  • The 30 cases span all task-angle categories except Environment setup and Repository collaboration, which are outside the primary focus.The resulting mix extends beyond bug fixing to feature addition, interface-contract adjustment, data processing, and observability.Source: §A.2 · pp. 18–19

A.3 USER PERSONA CONSTRUCTION AND STATISTICS

  • The persona taxonomy covers expertise, detail preference, control style, goal stability, and verification; four representative personas are extracted from real interaction logs.Their profiles capture distinct tendencies in the log population, including control and verification styles.Source: §A.3 · p. 19
  • The four simulated configurations are Non-coder, Product Manager, New Developer, and Software Architect, designed to expose distinct collaboration conditions.The Non-coder’s goal stability is adjusted to pivoting to avoid duplicating the Product Manager condition; these profiles are not intended to reproduce real-world population frequencies, and comparisons concern complete collaboration conditions rather than individual dimensions.Source: §A.3 · pp. 19–20

A.4 BENCHMARK POST-PROCESSING

The post-processing checks whether dependent subtasks are solvable and revises requirements judged under-specified. It also reclassifies generated test nodes for whole-task evaluation.

A.4 BENCHMARK POST-PROCESSING

  • 77.7% of subtasks had a passing run; an audit revised 8.2% judged under-specified, while 14.1% of failures were attributed to model limitations and left unchanged.Among previously unsolved subtasks covered by interactive experiments, all had a successful run with the Software Architect persona, and 93.33% were completed by at least three distinct coding-assistant models.Source: §A.4 · pp. 21–22
  • Whole-task evaluation retains 6,332 F2P and 4,166 P2P nodes after excluding unusable nodes and reclassifying some declared P2P nodes as F2P.The final set contains 10,498 nodes in total.Source: §A.4 · p. 22

A.5 TASK SCALE, COVERAGE, AND COMPOSITION

SWE-JOURNEY combines larger reference-patch scale with broad task composition, and its synthesized tasks and interactions receive favorable authenticity comparisons on reported dimensions. Interaction-authenticity is assessed against real logs, while task-authenticity comparisons include benchmark datasets.

A.5 TASK SCALE, COVERAGE, AND COMPOSITION

  • 10.2 files and 768.3 changed lines per instance are roughly 8× the files and over 50× the lines of SWE-bench Verified, and 2× the files and over 4× the lines of SWE-bench Pro.SWE-EVO touches more files, while its changed-line counts are comparable; the corresponding scale is presented in Table 9.Source: §A.5 · p. 22
  • SWE-JOURNEY leads on change scope, requirement depth, and new task angles, and is comparable to the strongest source on context scope and knowledge breadth.Its five reported values are 4.20 for context scope, 4.77 for change scope, 3.00 requirement labels per case, 8.70 distinct domains per case, and 3.80 distinct angles per case; Figure 3 depicts the five-axis comparison.Source: §A.5 · pp. 22–23, Figure 3 · p. 22

A.6 AUTHENTICITY SCORES

  • The synthesized set scores highly on repository-role fit and problem clarity, ranks highest on cross-file collaboration, and is second only to SWE-Marathon on skill compositionality.These comparisons are part of the task-authenticity assessment in Figure 4a.Source: §A.6 · p. 23, Figure 4 · p. 23
  • Synthesized interactions attain high naturalness and context coherence, with scores comparable to genuine coding-assistant logs.Figure 4b reports 1–5 mean scores across five interaction-authenticity dimensions for real logs and SWE-JOURNEY interactions.Source: §A.6 · pp. 23–24, Figure 4 · p. 23

B.1 EXPERIMENTAL SETUP

The evaluation uses a common closed-loop harness to compare coding assistants across four persona-conditioned user interactions. Sessions run in a persistent environment and end through task completion, user abandonment, or a shared turn-limit rule.

B.1 EXPERIMENTAL SETUP

  • Thirteen proprietary and open coding assistants are evaluated with the same harness and execution-limit policy, using greedy decoding and a 32,768-token generation cap.The fixed GLM-5.2 persona-conditioned simulator is not scored; using the same model for generation and evaluation may introduce bias, which the authors mitigate for closed-source evaluations by using different versions.Source: §B.1 · p. 24
  • The GLM-5.2 simulator conducts interactions under four personas and gradually discloses requirements using a hidden subtask checklist.It can inspect the repository, run hidden tests, mark subtask state, and decide what information to provide next.Source: §B.1 · p. 24
  • The coding assistant uses a persistent-container shell loop with a reply-to-user action, allowing user turns to alternate with up to 250 autonomous command steps.Message and history truncation preserve user turns and recent context; overflow triggers one retry after forcing history to the low watermark.Source: §B.1 · p. 25
  • A session is one model–instance–persona interaction and ends when all subtasks resolve, confidence reaches zero, or the shared turn limit is reached.Turn counts are realized interaction lengths rather than preset budgets, and longer sessions can permit more cumulative execution; reported session cost includes coding-assistant tokens but excludes simulator tokens.Source: §B.1 · p. 25

B.2 SINGLE-SHOT LONG-HORIZON TEST-NODE PASS RATES

Single-shot performance falls below the Software Architect persona but exceeds the other persona results; the section also defines per-turn metrics for evaluating interaction capabilities. The single-shot and interactive rankings are strongly associated, while capability metrics use LLM judges with manual reliability checks.

B.2 SINGLE-SHOT LONG-HORIZON TEST-NODE PASS RATES

  • Single-shot evaluation gives the entire synthesized instance in one mega-prompt, with no simulated user, evolving requests, or mid-course feedback.It scores only the final repository state, reporting Final F2P, Final P2P, and their Average.Source: §B.2 · p. 25, Table 10 · p. 26
  • 60.31% for single-shot Average versus 75.94% for Software Architect Average across the same eleven models; single-shot exceeds each of the other three persona means.The mean paired difference between Software Architect and single-shot is 15.62 percentage points; single-shot means for the other profiles are 21.82%, 28.41%, and 34.12%.Source: §B.2 · pp. 25–26, Table 10 · p. 26
  • Spearman ρ = 0.873 (p < 0.001, n = 11) for Average scores, with local rank changes rather than a wholly different ordering.HY4-PREVIEW ranks first in both settings; KIMI-K3 and GLM-5.2 reverse order.Source: §B.2 · p. 26

B.3 FINE-GRAINED CAPABILITY METRICS

  • The seven per-turn capability metrics are distinct from final test-node pass rates, and LLM-judge accuracy exceeded 90% in manual checks of 50 instances per metric.User Confidence aggregates simulator self-reports; the other six metrics use three LLM-judged stages.Source: §B.3 · pp. 26–27
  • Asking Right judges whether user-facing replies solicit information, decisions, or approval and whether those questions address open, on-plan needs.Ask% is the rate of soliciting turns; Rel% is the rate of on-plan turns and cannot exceed Ask%.Source: §B.3 · p. 27
  • Finding Right classifies committed localization against user guidance; Miss% counts off-locus leaves, while no-locus turns are excluded from both Hit% and Miss%.The overconfident-wrong flag is separate from Miss leaves, so a turn may count as either, both, or neither.Source: §B.3 · p. 27
  • Fixing Right uses visible reasoning to assess recognized problems: turns are labeled skipped, grounded, or clean, and skipped problems are rated for criticality.Models without usable reasoning text are shown as unavailable for these measures.Source: §B.3 · p. 27

B.4 PER-PERSONA RESULTS

Persona choice changes interaction length, clarification behavior, localization errors, and critical omissions. More technical profiles are associated with lower localization misses and critical-skip rates, while similar session lengths can accompany substantially different completion rates.

Per-persona results

  • The appendix reports four persona-specific result tables, an aggregate persona overview, and four persona-specific capability breakdowns.The breakdowns cover confidence, asking, localization, and grounding measures.Source: §B.4 · pp. 27–28, Table 15 · p. 29, Table 16 · p. 30
  • 35.57% Final F2P for New Developer versus 78.50% for Software Architect, despite nearly identical mean session lengths of 25.8 and 25.9 turns.Product Manager sessions average 7.8 turns, the shortest of the four profiles.Source: §B.4 · p. 28, Table 12 · p. 28, Table 13 · p. 29
  • Ask is 19.59% for Product Manager and 40.27% for Software Architect, whose on-plan Rel rate is 37.58%.Clarification shifts from product-level choices toward code-level questions and evidence-backed architectural proposals as user expertise increases.Source: §B.4 · p. 28, Table 12 · p. 28, Table 14 · p. 29
  • Miss falls from 72.18% for Non-coder to 40.68% for New Developer and 22.52% for Software Architect, averaged over models.The decline appears across all thirteen models; the text attributes better localization to code-level feedback, including a New Developer correction that redirects GLM-5.2 to timezone-conversion functions.Source: §B.4 · pp. 28–29, Table 11 · p. 28, Table 13 · p. 29
  • Critical Skip is 12.69% for Product Manager and 4.64% for New Developer; New Developer and Software Architect rates are both below those of the less technical profiles.The reduction from Product Manager to New Developer appears for all eleven models with a Fix score.Source: Table 12 · p. 28, §B.4 · p. 29, Table 13 · p. 29

B.5 ANSWER-SEEKING RETRIEVAL AND TASK COMPLETION

This section examines how answer-seeking retrieval relates to task completion, using SWE-Journey’s retrieval analysis and cases of asking, finding, and fixing requirements. The cases show that retrieved answers do not ensure generated requirements are completed, and that correcting mistaken premises and preserving existing contracts can avoid regressions and test-loading failures.

B.5 ANSWER-SEEKING RETRIEVAL AND TASK COMPLETION

  • GPT-5.6-SOL averages 7.30 retrieval turns per session and reaches target-repository code in 28.75% of turns; four frequent searchers gain 23–40 points on seed tasks.Weaker models usually average below one retrieval turn and rarely expose future code-bearing material.Source: §B.5 · p. 31, Table 20 · p. 32
  • Strong retrievers perform significantly better on searchable seed tasks but not synthesized tasks, while weaker retrievers show no gap or lower seed accuracy, indicating synthesized tasks mitigate retrieval leakage.In the Matplotlib case, retrieved code supplied only font-family handling; the model still had to add required interfaces and defaults.Source: §B.5 · pp. 31, 33, Table 20 · p. 32
  • Local git history is most common (0.75 turns/session; 1.60% of turns reach code), while clone/fetch reaches more code (2.08% vs. 1.69%); code-reaching and passing turns are at most 0.22%.Among reached turns, all-F2P-passing sessions account for 12.33% for history, 10.56% for clone/fetch, and 9.44% for GitHub search; package upgrades are rare.Source: Table 21 · p. 32, §B.5 · p. 33

C.1 CASE: COMPLETE CONVERSATIONS WITH A NON-CODER

  • A complete Non-coder conversation shows requests and corrections evolving coherently while expertise, explanation preferences, control style, goals, and verification style remain consistent.The user planned to try the plots later; no user-side test invocation was recorded.Source: C CASE STUDY · p. 33, §C.1 · p. 36

C.2.1 CASE A1: THE DIAGNOSTIC CONTRAST—ONE CLARIFYING QUESTION

  • Clarification matters when ambiguity could lead to the wrong interface: CLAUDE-OPUS-5 surfaced incompatible path interpretations and obtained the user’s correction, while GPT-5.6-SOL rewrote uploads without asking.The case contrasts preserving the canonical-path contract with following an under-specified request literally.Source: §C.2.1 · pp. 36–37, §C.2 · p. 36

C.3.1 CASE F1: FOLLOWING A WRONG INSTRUCTION

  • Finding right requires staying with load-bearing code as requirements change; in Case F1, both assistants received instructions conflicting with the Sunday-first convention for ISO weekday queries.The section distinguishes turns that stray from relevant code (Miss%) from overconfident turns in a wrong direction (OC%).Source: §C.3 · p. 37, §C.3.1 · p. 38
  • After initially requesting an in-place change to Monday=1, the user reversed course and asked to restore Sunday=1 and add a separate ISO weekday expression across all four backends.The user said changing the existing lookup risked regressions for code relying on its Sunday=1 convention.Source: §C.3.1 · p. 38
  • Claude-Opus-5 flagged that the existing weekday lookup is Sunday=1, not Monday=1, implemented true ISO numbering, and the user accepted the correction.The new expression was registered on DateField.Source: §C.3.1 · p. 38
  • DeepSeek-V4-Pro changed the existing weekday behavior, then added ExtractIsoWeekDay without public-interface integration; tests could not load it, while Claude-Opus-5 preserved the old behavior and integrated the new expression.The original lookup used Sunday=1 across all four backends; Claude corrected the user's mistaken premise and registered ExtractIsoWeekDay on DateField.Source: §C.3.1 · p. 38

C.3.2 CASE F2: OVERGENERALIZING FROM A PASSING SELF-TEST

In the Product Manager case, repeated localization mistakes culminate in a passing self-test being treated as proof that batch functionality needs no changes. The test covers a mixed-refraction batch, but the paper cautions that its result neither establishes full hidden-test correctness nor identifies the source of remaining failures.

C.3.2 CASE F2: OVERGENERALIZING FROM A PASSING SELF-TEST

  • The assistant declares batch support complete without edits after earlier corrections repeatedly put the reference location on the wrong interface.The user had to correct placements on both the sky-position interface and object setup.Source: §C.3.2 · p. 39
  • A four-target test with refraction off for two targets and on for two reports 0.00 km round-trip error for all four.The assistant presents this model-chosen test as evidence that the batch case works.Source: §C.3.2 · p. 39
  • The round-trip self-test does not establish full hidden-test correctness, and the aggregate failures do not identify which batch requirement is responsible.Most hidden checks still fail, while the user ultimately accepts the assistant's claim.Source: §C.3.2 · p. 39

C.3.3 CASE F3: ACCURATE LOCALIZATION UNDER CHANGING REQUIREMENTS

The cases show that following changing requirements and grounding claimed fixes are distinct challenges: models may adapt accurately, but can also resist corrective evidence or overstate verification.

C.3.3 CASE F3: ACCURATE LOCALIZATION UNDER CHANGING REQUIREMENTS

  • GPT-5.6-SOL's raw case diagnostic favors it over CLAUDE-OPUS-5, with mean 34.47 versus 7.11.GPT-5.6-SOL locates the relevant code and adapts edits as requirements change, while the separate dialogues do not establish that asking less is better.Source: §C.3.3 · p. 40

C.4.1 CASE G2: DEFENDING A WRONG CONCLUSION AGAINST CORRECTIVE EVIDENCE

  • In G2, HY3 and Qwen3.5-397B-A17B defend incorrect conclusions despite corrective evidence, and the failures remain in final evaluation.The final F2P scores are 4/115 for HY3 and 7/70 for Qwen3.5-397B-A17B.Source: QWEN3.5-397B-A17B · p. 41, §C.4.1 · p. 41

QWEN3.5-397B-A17B

  • In G2, a roughly twelve-degree hour-angle error persisted despite the model's claims that its matrix was proven and the requirements incompatible.The user's coordinate-convention correction was followed by abandoned direct-transform work; route-comparison and refraction tests remained failed.Source: QWEN3.5-397B-A17B · p. 41, §C.4.1 · p. 41

QWEN3.5-397B-A17B — New Developer

  • The New Developer case shows that revisions can correct implementation details, including retaining live deduplication state while persisting a backup for reloads.The assistant removes in-memory clears after the user corrects that requirement.Source: QWEN3.5-397B-A17B — New Developer · p. 42

C.4.2 CASE G3: CONFIDENCE IS NOT CORRECTNESS

  • The Domain Expert dialogue ends at confidence 8 and F2P 84/103, versus the New Developer dialogue's confidence 73 and F2P 12/103.The contrast separates confidence from measured correctness but does not establish that verification alone caused the score gap.Source: QWEN3.5-397B-A17B — New Developer · p. 43

C.5 CASE STUDY: REACHING THE ANSWER IS NOT SOLVING IT—WEB RETRIEVAL AND HISTORY MINING

  • The retrieval case distinguishes finding a published seed solution from using related upstream code to implement generated requirements.GPT-5.6-SOL retrieves the Ansible module directly, while the Matplotlib case uses upstream code as a reference.Source: §C.5 · p. 43

C.5.1 CASE R1: COPYING A PUBLISHED SEED SOLUTION

In the Ansible case, GPT-5.6-SOL retrieves a published module and writes it into the working tree. The seed task finishes at 3/3 Fail-to-Pass, illustrating direct reuse of a public solution.

C.5.1 CASE R1: COPYING A PUBLISHED SEED SOLUTION

  • For repeatable user operations on Pluribus Networks switches, GPT-5.6-SOL finds PR #51428 and commit 5e88cd9972 on GitHub.The task covers user creation, password updates, and deletion.Source: §C.5.1 · p. 43
  • 3/3 Fail-to-Pass: the seed task finishes after GPT-5.6-SOL retrieves and reuses the published Ansible module.Matching SHA256 values confirm that the retrieved artifact was written into the checkout; generated follow-up requirements remain separate.Source: §C.5.1 · pp. 43–44
  • The retrieved module directly supplies the seed implementation, rather than requiring the model to reconstruct it.The analysis separates this public seed solution from generated follow-up requirements.Source: §C.5.1 · p. 44

C.5.2 CASE R2: UPSTREAM CODE AS A REFERENCE FOR GENERATED REQUIREMENTS

The Matplotlib case uses upstream code as an integration reference, not as a ready-made solution to generated requirements. After user feedback and corrections, the seven typography subtasks finish at 44/44 Fail-to-Pass.

C.5.2 CASE R2: UPSTREAM CODE AS A REFERENCE FOR GENERATED REQUIREMENTS

  • The inspected upstream releases and main branch do not provide the generated interface and configuration behavior as a ready-made implementation.The task extends beyond the original shared-axis bug.Source: §C.5.2 · pp. 44–45
  • User feedback also exposes a readback issue: an explicit font-family string must return as a one-element list.The model normalizes the effective family to a new list while copying existing sequences.Source: §C.5.2 · p. 45
  • 44/44 Fail-to-Pass: the seven typography subtasks pass after corrections to family readback and validation.The generated task requires weight and style controls, setter/getter support, and new-plot defaults without changing existing appearance.Source: §C.5.2 · pp. 44–45
  • GPT-5.6-SOL traces upstream labelfontfamily handling as a reference, then implements missing weight/style counterparts and six tick-specific defaults.The recorded edits extend axis.py, rcsetup.py, and matplotlibrc.Source: §C.5.2 · p. 45

Related papers

Loading 2610.11559v1…