PaperlayerView exact revision on arXiv

Source-linked AI summary

Coding-Agent Benchmarks Should Match Their Users' Task Flows

Igor Slinko, Yaroslav Golubev, Sergey Titov

arXiv:2610.09633v1cs.LGcs.AIcs.CLcs.SE

TL;DR

Issue-derived benchmarks do not capture the mix and switching of user requests in real coding-agent sessions, and interaction distributions differ across settings. SWE-TaskFlow transforms verified tasks with prompt splitting and repository QA, selecting trajectories against a target Task Flow. In a 700-task pilot, splitting approximately doubled agent cost without a stable resolve-rate change.

  • Problem

    Issue-derived benchmarks miss the broader mix and within-session switching of user requests, while measured Task Flows differ across settings.

    Source: §1 · p. 2

  • Method

    SWE-TaskFlow preserves verified tasks while using prompt splitting and repository QA to generate trajectories selected for alignment with a target Task Flow.

    Source: §3 · p. 3

  • Results

    Split prompts approximately doubled agent cost, while resolve-rate shifts were mixed across models and largely comparable to run-to-run variation.

    Source: §4 · p. 3

  • Takeaways & Limitations

    Benchmarks should name a target use case and calibrate their interaction layer to its measured Task Flow.

    Source: §1 · p. 2, §6 · p. 4

    Long Production Sessions represent only one use case, and pilot comparisons used three-run min–max ranges without significance tests.

    Source: §5 · p. 4

Abstract

from arXiv · show

The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories. In a pilot on 700 SWE-Bench Pro tasks, solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate: the interaction protocol itself is an important dimension of evaluation.

1 Introduction

Long Production Sessions contain a broader mix of user requests than issue-derived benchmarks, and users switch task types within sessions. SWE-TaskFlow transforms verified tasks toward a measured target Task Flow, while the paper argues that no single interaction distribution is universally realistic.

1 Introduction

  • Long Production Sessions comprise sessions with at least three user messages and account for 33% of the 4,782 Production Sessions.The study describes their Task Flow through session lengths, user-message task types, and transitions between adjacent types.Source: §1 · p. 2
  • Issue-derived benchmarks mostly present feature requests and bug fixes, whereas Production Sessions also include code questions, planning, review, refactoring, and execution.Users may switch among these task types during a session, unlike a single detailed issue scored on its final patch.Source: §1 · p. 2
  • Task Flows differ across Production Sessions and three public interaction corpora, so benchmarks should target and measure a specific use case.The paper argues that a distribution measured on one product does not transfer to another.Source: §1 · p. 2
  • SWE-TaskFlow transforms verified SWE tasks into replayable multi-turn trajectory candidates while preserving their original requirements and tests.Its contributions also include per-turn evaluation signals and a pilot on 700 SWE-Bench Pro tasks.Source: §1 · p. 2

2 Task Flows of Production Sessions

The study classifies user messages in JetBrains IDE sessions and characterizes Task Flow by session lengths, task types, and transitions. Long Production Sessions include substantial code explanation and question requests, while public interaction corpora have distinct Task Flows.

2 Task Flows of Production Sessions

  • The study classifies roughly 14,000 user messages into 16 task types, with three classifier models agreeing on 76.1% of labels.The messages come from 4,782 JetBrains IDE sessions; the same classification pipeline is applied to production sessions, public corpora, and benchmark prompts.Source: §2 · p. 2
  • A corpus-level Task Flow comprises distributions of session lengths, message task types, and directed transitions between adjacent messages.The transition distribution preserves ordering information that a task-type histogram alone would lose.Source: §2 · p. 2
  • Of 4,782 Production Sessions, 33% contain at least three user messages; bug fixes (25.9%) and new features (18.9%) lead but do not dominate the long-session slice.The 75th and 90th percentiles are three and eight messages, respectively; subsequent Task-Flow analysis conditions on this slice.Source: §2 · p. 2
  • EXPLAIN requests account for 6.1% of long-session messages and general programming QUESTIONs for another 9.7%.Explanation is an intent issue-derived benchmarks lack; it remains common at every position and motivates repository QA.Source: §2 · pp. 2–3
  • Long-session samples from SWE-chat, SpecStory, and DataClaw differ from Production Sessions and from one another in intents and transitions.The paper concludes that the target interaction distribution must be measured in its own setting.Source: §2 · p. 3

3 SWE-TaskFlow: Transforming Verified Tasks

SWE-TaskFlow transforms existing issue-derived benchmarks while retaining their verified tasks and tests. Prompt splitting and repository QA generate candidate interactions, which TFAS selects against a measured target Task Flow.

3 SWE-TaskFlow: Transforming Verified Tasks

  • SWE-TaskFlow keeps the repository, requested change, environment, and final tests fixed while regenerating the interaction layer of existing issue-derived tasks.Unlike related transformations that choose trajectory shape ad hoc, it calibrates trajectory shape to a measured Task Flow.Source: §3 · p. 3
  • Prompt splitting turns an issue into three ordered, actionable user turns while preserving its requirements; repository QA adds zero to three verifiable behavior questions.Issues failing checks remain single-turn; 700 of 731 SWE-Bench Pro tasks are retained, and candidates contain one to six turns.Source: §3 · p. 3
  • Repository questions are mined from a solved-task trajectory and live repository state, then checked for answerability, validity before and after the patch, and solution leakage.Each candidate also receives a hidden reference answer and must pass an executable proof script in a clean repository copy.Source: §3 · p. 3
  • TFAS selects candidate trajectories by maximizing alignment with target distributions for session lengths, task types, and directed transitions.It combines one minus base-2 Jensen–Shannon divergence for each distribution using a geometric mean scaled to 0–100; 100 denotes an exact match.Source: §3 · p. 3

4 Pilot: The Interaction Layer Changes the Measurement

In the 700-task pilot, splitting prompts approximately doubled agent cost without a stable resolve-rate change. Adding TFAS-calibrated QA improved Task Flow alignment, while resolve-rate rankings remained unstable across models.

4 Pilot: The Interaction Layer Changes the Measurement

  • Split prompts approximately doubled agent cost, while resolve-rate shifts were mixed across models and largely comparable to run-to-run variation.These runs establish no success ordering, but show that progressive disclosure changes agent effort.Source: §4 · p. 3
  • TFAS rose from 59.6 for pure Split to 77.8 with calibrated QA, with moderate added cost and no stable resolve-rate ranking across models.Figure 2 plots resolve rate against mean agent cost per issue on the 700 tasks; its whiskers show min–max across three runs.Source: §4 · pp. 3–4, Figure 2 · p. 4
  • Split+QA trajectories produce a test-suite verdict after each issue turn and a judged answer for each QA turn, unlike the single binary signal from Single trajectories.The denser verified signals support finer-grained diagnosis and ranking-based training.Source: §4 · p. 4

5 Discussion: Toward Richer Task Flows

SWE-TaskFlow covers prompt splitting for implementation intents and repository QA for EXPLAIN, while most other observed intents still lack augmentations. Static replay and reactive interaction evaluate complementary regimes, and pilot resolve-rate readings are directional because comparisons were not significance-tested.

5 Discussion: Toward Richer Task Flows

  • Most user intents still lack augmentations; planning and code-review turns are especially difficult to verify automatically.SWE-TaskFlow currently covers prompt splitting for implementation intents and repository QA for EXPLAIN.Source: §5 · p. 4
  • Static replay measures progressive disclosure, memory, and adaptation, but not feedback that reacts to agent behavior.Reactive benchmarks measure that feedback at the cost of simulator assumptions and trajectory variance.Source: §5 · p. 4
  • Three pilot runs use min–max ranges rather than confidence intervals, with no significance tests, so resolve-rate readings are directional only.Long Production Sessions define only one use case; classifier accuracy lacks a human-labeled test set.Source: §5 · p. 4

6 Conclusion

Interactive benchmarks should state the user interaction distribution they represent, since real users mix and switch task types in ways issue-derived benchmarks miss and public corpora differ. SWE-TaskFlow adapts verified tasks into calibrated, replayable multi-turn evaluations.

6 Conclusion

  • Interactive benchmarks should name their target interaction distribution because real users mix and switch task types in ways issue-derived benchmarks miss, and public corpora differ.SWE-TaskFlow preserves verified tasks while using splitting, verifiable QA, and TFAS-selected placement to create calibrated, replayable multi-turn evaluations.Source: §6 · p. 4

A Task Taxonomy and Classification

The appendix describes a shared pipeline for classifying user messages into a 16-label task taxonomy and reports inter-model robustness. It also summarizes the classified session sample and its message-count distribution.

A Task Taxonomy and Classification

B Task Flows in Public Interaction Corpora

Long-session flows differ across public coding-agent corpora, so calibration should begin with a named target and its measured Task Flow. The procedure then compares a verified source benchmark with that target and augments gaps while preserving verifiability.

B Task Flows in Public Interaction Corpora

  • SpecStory, DataClaw, and SWE-chat differ from Production Sessions and from one another in intents and transitions; none is ground truth.SpecStory is closest to Production but has more planning; DataClaw is execution-heavy with short operational turns, while execution and planning dominate SWE-chat.Source: B Task Flows in Public Interaction… · p. 8

C Calibrating to a New Target

  • Define the target by user population, product surface, session filter, and measurement window; if production data is unavailable, a public corpus is only a provisional substitute.The filter and measurement date matter because task flows differ across products and can drift as products and models change.Source: C Calibrating to a New Target · pp. 10–11
  • Sample user-side messages across complete sessions, then classify each with a fixed taxonomy and check inter-model agreement before fixing the target.If the transition distribution is unstable, enlarge the sample or coarsen the taxonomy; low classifier agreement calls for revising label definitions and rerunning the check.Source: C Calibrating to a New Target · p. 10
  • Measure and publish session-length, task-type, and directed-transition distributions together so others can recompute TFAS and its components.Transitions include explicit START and END edges after non-instantiable messages are filtered.Source: C Calibrating to a New Target · p. 10
  • Compare the source benchmark flow with the target and augment frequent target intents or session lengths that the benchmark lacks, using verifiable turns.SWE-Bench Pro’s length gap is addressed by prompt splitting and its EXPLAIN gap by repository QA; QUESTION, PLANNING, CODE-REVIEW, REFACTOR, EXECUTE, and DEBUG remain uncovered.Source: C Calibrating to a New Target · p. 10
  • Generate admissible augmentation combinations and select one candidate per task to maximize TFAS, checking that optimization does not change task difficulty.The paper uses randomized local search and reports component scores alongside the aggregate; its QA-placement ablation checks whether placement changes difficulty.Source: C Calibrating to a New Target · pp. 10–11
  • Evaluate rare but consequential intents as separately reported stress-test slices because frequency-based alignment can push them toward zero.The paper names destructive EXECUTE commands and security-relevant CODE-REVIEW as examples.Source: C Calibrating to a New Target · p. 11

D Related Transformations of Verified Tasks

SWE-TaskFlow differs from earlier verified-task transformations by calibrating fixed, replayable trajectories to a measured interaction distribution. Its pipeline splits prompts and inserts repository-grounded questions, with verification and inference handled separately.

D Related Transformations of Verified Tasks

E Generation and Verification Pipeline

  • The splitter accepts an issue only when it can produce ordered actionable turns while preserving all original requirements and technical artifacts without adding requirements.Atomic issues remain single-turn; a final model-based completeness check follows decomposition.Source: E Generation and Verification Pipeline · pp. 11–12
  • Question generation mines prior solving trajectories and live repository state, then retains only questions with reference answers and executable proof scripts that pass in a clean environment.Questions must remain valid before and after the patch, avoid revealing the solution, and provide observable evidence; this excludes questions such as architectural rationale without repository evidence.Source: E Generation and Verification Pipeline · p. 12
  • At inference, agents receive ordered prompts but not reference answers or validation artifacts; final QA correctness is model-judged, and released conditions are replayable dataset splits.Issue tests and creation-time proof execution provide deterministic evidence, while answer scoring compares submissions with hidden references using GPT-Sol.Source: E Generation and Verification Pipeline · p. 12

F TFAS Details and Dataset Tuning

TFAS compares a benchmark’s session lengths, task types, and directed transitions with a measured target, then guides selection among augmented trajectories. In the SWE-Bench Pro pilot, TFAS-optimized selection improved alignment over pure Split, while a separate SWE-chat ablation showed that the target affects which variants align.

F TFAS Details and Dataset Tuning

  • TFAS compares target and benchmark distributions for session-length bins, task types, and directed transitions, combining their alignment scores with a geometric mean.Each component score is one minus the Jensen–Shannon divergence between the corresponding distributions.Source: F TFAS Details and Dataset Tuning · p. 12
  • TFAS filters out non-instantiable labels, builds transitions with START/END edges, and uses independently normalized distributions; rare transitions can remain unstable.Bootstrap intervals should resample complete sessions rather than individual edges.Source: F TFAS Details and Dataset Tuning · pp. 12, 14
  • The figures show source-to-optimized Task Flow, intermediate flows after prompt splitting, and the complete generation, assembly, inference, and verification pipeline.Figure 7’s transformation keeps repositories, requested changes, and final test-based verifiers unchanged.Source: Figure 7 · p. 13, Figure 8 · p. 13, Figure 9 · p. 14
  • For each verified task, the candidate set varies split-turn merging, QA-turn count, and QA placement; long-slice tuning selects one trajectory per task to maximize TFAS.The optimized selection retained 454 of 467 validated questions and varied their counts and placements across sessions.Source: F TFAS Details and Dataset Tuning · p. 14
  • TFAS reached 77.8 versus 59.6 for Split alone against the long-session Production target.Adding QA lifted every component over pure Split; the original single-turn prompts scored SL=0 against the three-message target, making the geometric mean zero.Source: Table 2 · p. 15, F TFAS Details and Dataset Tuning · p. 15
  • Against SWE-chat, calibrating to a different target favored different variants, so that ablation is reported separately from the Production-target result.Table 3 reports values on the same 64 source tasks.Source: F TFAS Details and Dataset Tuning · p. 15, Table 3 · p. 15

G QA Answer Accuracy

The evaluation reports answer accuracy alongside resolve rate for the same TFAS-optimized Split+QA trajectories. Accuracy is measured against hidden reference answers on the 255 issues carrying QA questions, not all 700 tasks.

G QA Answer Accuracy

  • Answer accuracy is the fraction of repository-question answers judged correct against hidden references.The evaluation uses 454 Sol-generated questions across 255 of the 700 tasks.Source: G QA Answer Accuracy · p. 15
  • 69.8% (GPT-5.4 Mini) to 83.0% (GPT-5.6 Terra) is the answer-accuracy range across four models.Figure 10 reports both answer accuracy and resolve rate for the same trajectories on the 255 issues carrying questions.Source: G QA Answer Accuracy · p. 15, Figure 10 · p. 16
  • Both metrics come from the same trajectories, but the paper does not claim a relationship between answer accuracy and resolve rate.The measurements cover only the 255 issues carrying QA questions, so their resolve rate is not directly comparable to Figure 2.Source: G QA Answer Accuracy · p. 15, Figure 10 · p. 16

H Additional Experimental Results

The additional experiments test whether prompt composition, question placement, or intermediate verification changes measured task outcomes. The discussion also notes limits on access to production data and the role of LLMs in the methodology.

H Additional Experimental Results

  • The Concat control sends the same three rewritten segments as Split in one message, separating rewriting effects from sequential disclosure effects.The Single–Concat gap measures information change from rewriting, while the Concat–Split gap isolates sequential disclosure.Source: H Additional Experimental Results · p. 17
  • Across the tested QA-first, QA-last, and QA-random placements, resolve-rate differences showed no consistent ordering and were comparable to run-to-run variation.The comparison held the 64 tasks and 114 Terra-generated questions fixed, changing only phase order.Source: H Additional Experimental Results · p. 17, Figure 12 · p. 18
  • 78.1–86.8% (GPT-5.4 Mini), 86.6–93.8% (GPT-5.6 Luna), and 93.8–98.2% (GPT-5.6 Terra) were the answer-accuracy ranges across tested placements.Placement shifted accuracy by a few points with no consistent direction across models; Figure 13 uses the same 64 tasks and Terra-generated questions.Source: H Additional Experimental Results · p. 17, Figure 13 · p. 19
  • At every checkpoint, the three placement curves were nearly indistinguishable, while resolve rate and test-pass rate grew as issue turns accumulated.Checkpoints were measured on the clean repository and after each of the three issue turns; QA-last serves as a Split-only baseline because QA turns do not modify the repository.Source: H Additional Experimental Results · p. 17, Figure 14 · p. 20
  • Figure 11 compares Single, Split, and Concat; Single and Split summarize three runs on 700 tasks, while each Concat point is one run.Single and Split markers are arithmetic means with observed min–max whiskers; Concat comparisons are descriptive, not significance tests.Source: H Additional Experimental Results · p. 17, Figure 11 · p. 18

I Ethical Considerations and Broader Impacts

  • Production-session analysis is aggregate-only: raw sessions, session text, and user identifiers are not released.The paper identifies privacy risks from operational data and the possibility that matching frequent behavior could overweight noisy or low-value workflows.Source: I Ethical Considerations and Broader… · p. 21
  • LLMs classify intent and contribute to splitting, question generation, reference answers, candidate review, and answer judging.Executable tests and creation-time proof execution provide non-textual evidence but do not remove model dependence from classification and generation.Source: I Ethical Considerations and Broader… · p. 21
Loading 2610.09633v1…