PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking Task Takeover Across Agent Handoffs

Wenyuan Li, Guang Li, Dongyuan Li, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

arXiv:2610.36523v1cs.CL

TL;DR

When one agent takes over unfinished work, prior decisions and actions shape what the receiver should do next, a relationship that task-success or memory-retention scores alone do not capture. HSC-Bench evaluates evidence recovery and state-dependent continuation across 1,039 tasks. Structured state improves repository recovery for all six receivers and continuation for most, while the leading models differ between tracks.

  • Problem

    A task-success outcome or successful recall alone does not establish whether a receiver continued appropriately from the inherited progress of unfinished work.

    Source: §1 · p. 1

  • Method

    HSC-Bench evaluates 1,039 tasks across repository evidence recovery and continuation from conversational, tool, and reported file-work state.

    Source: §1 · p. 2

  • Results

    Structured state improves repository evidence recovery across all six receivers and continuation performance for most, but the leading models differ between tracks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Handoff representations are more usefully compared with the intended receiver and remaining task than evaluated in isolation.

    Source: §6 · p. 8

    The study examines single handoffs; repeated handoffs and how later receivers handle incorrect completion records remain for future study.

    Source: §6 · p. 8

Abstract

from arXiv · show

As agents collaborate on longer tasks, responsibility may pass between models before the work is complete. The receiving model must build on prior decisions, observations, and actions while addressing the remaining requirements. Evaluating this transition requires connecting the information preserved at handoff to how the receiver continues the work, a relationship that task-success and memory-retention scores alone do not capture. We introduce the Handoff State Continuity Benchmark (HSC-Bench), comprising 1,039 tasks that evaluate two complementary aspects of task takeover: recovering relevant repository evidence and continuing from established conversational, tool, and file-work state. Comparisons across six receiving models show that structured handoff state improves repository evidence recovery for every model and continuation performance for most models. The strongest receivers differ between the two tracks, showing that evidence-recovery rankings alone do not characterize performance when continuing from inherited state. By combining these comparisons with checks of subsequent responses and actions, HSC-Bench evaluates both access to prior evidence and adherence to established commitments and completed work. This provides a basis for selecting receiving models and assessing handoff representations according to the work that must be carried forward.

1 Introduction

Task takeover requires a receiver to continue unfinished work in light of decisions, evidence, and actions inherited from the prior agent. HSC-Bench evaluates both recovery of relevant evidence and whether subsequent behavior fits that inherited state; structured state improves results across the two tracks for most or all receivers.

1 Introduction

  • Task takeover occurs when a receiving model inherits a partially completed process whose prior decisions, accumulated evidence, and environmental changes shape the next appropriate steps.The introduction distinguishes this from evaluating only task completion or successful recall: continuation must also reflect what was settled and what remains unresolved.Source: §1 · p. 1
  • HSC-Bench contains 1,039 tasks spanning repository work and synthetic continuation scenarios for conversational state, tool use, and reported file-work state.The evaluation checks recovered evidence against annotated code locations and subsequent responses and actions against commitments, updated facts, and completed operations.Source: §1 · p. 2
  • The evaluation pairs recent transcripts with structured state at the same visible token count while holding the receiver and task request fixed.Shared source histories and fixed handoff points support receiver comparisons and within-model comparisons of context conditions.Source: §1 · p. 2
  • Structured state improves repository evidence recovery for all six receivers and continuation performance for most, while the leading models differ between tracks.These results make evidence recovery only part of the basis for choosing a receiver to continue unfinished work.Source: §1 · p. 2

2 Related Work

Prior work evaluates agent outcomes and memory access, but task takeover also requires judging whether a receiver continues appropriately from inherited progress. HSC-Bench connects evidence recovery and continuation under shared handoff conditions.

2.1 Agent Evaluation

  • Agent benchmarks assess completion, policy compliance, intermediate milestones, and specific requirements, while HSC-Bench evaluates whether subsequent behavior fits the requirements that apply at handoff.At takeover, some requirements are already satisfied and prior decisions and actions constrain what remains.Source: §2.1 · p. 2

2.2 Interactive Memory

  • Interactive-memory research studies access to accumulated or revised information, while HSC-Bench asks whether prior decisions and completed actions support continuation consistent with an unfinished task.Related work also examines how retrieved information contributes to subsequent responses and how compressed context supports later work.Source: §2.2 · p. 2

2.3 Evidence Recovery and Agent Handoffs

  • Prior handoff studies examine continuation performance, rediscovery costs, issue resolution, and progress from checkpoints; HSC-Bench compares receivers under shared handoff conditions.It evaluates both recovery of annotated repository evidence and satisfaction of explicit requirements when continuing from inherited state.Source: §2.3 · pp. 2–3

3 HSC-Bench

HSC-Bench prepares shared histories and fixed handoff points to compare receiving models on repository evidence recovery and state-dependent continuation. Its tasks combine recorded repository work with synthetic scenarios, and its continuation tasks are screened for dependence on prior history and attainable targets.

3 HSC-Bench

  • HSC-Bench prepares contexts from recorded source-agent histories at fixed handoff points, enabling receivers to be compared on the same inherited work.The benchmark includes 300 repository evidence-recovery tasks and 739 state-dependent continuation tasks.Source: §3 · p. 3

3.1 Repository Evidence Recovery Track

  • The repository track uses 300 coding tasks across 38 repositories and eight languages; receivers predict relevant files and line spans scored against withheld annotations using line-F1.Tasks are grouped by whether annotated paths appear in the history and recent transcript: 217, 42, and 41 tasks fall into the three visibility groups, respectively.Source: §3.1 · p. 3

3.2 State-Dependent Continuation Track

  • The 739 continuation tasks cover conversational, tool, and file-work state, with fixed binary criteria scoring whether required responses, actions, or outputs are satisfied.Tool tasks check both reported operation state and tool-call records; file-work tasks evaluate a progress report, not file edits or test execution.Source: §3.2 · p. 3

3.3 Task Construction and Screening

  • Of 1,130 candidate continuation tasks, 739 passed screening for dependence on prior history and attainable targets.Screening also checked internal consistency, clarity, plausibility, and difficulty under limited context.Source: §3.3 · p. 3

4 Evaluation Protocol

The evaluation fixes receivers and task requests while comparing handoff contexts, using separate measures for repository recovery and continuation and paired statistical analysis.

4.1 Models and Context Conditions

  • Six receivers take over the same prior work under recent-transcript and structured-state conditions; GPT-5.4 also receives supplementary comparisons with narrative summaries and reference contexts.For continuation, all six are additionally evaluated with the complete pre-handoff history; GPT-5.4 also continues from its own full history as a source-model reference.Source: §4.1 · p. 4

4.2 Evaluation Metrics

  • Repository recovery uses task-averaged line-F1 as its primary metric and file-F1 as a complementary measure of relevant-file identification.Continuation tasks receive scores based on the fraction of binary criteria satisfied; Mean captures partial satisfaction, while Pass requires every criterion in a task to be satisfied.Source: §4.2 · p. 4
  • Continuation metrics weight all 739 tasks equally, and continuation scores and pass rates use a 0–100 scale while repository F1 uses a 0–1 scale.Table 2 presents repository line-F1 on 300 tasks and continuation mean scores and all-criteria pass rates on 739 tasks.Source: §4.2 · pp. 4–5, Table 2 · p. 5

4.3 Statistical Analysis

  • Context effects are estimated from paired task-level score differences, with positive differences favoring structured state.The analysis reports 95% percentile bootstrap confidence intervals from 20,000 resamples; repository analyses resample repositories, while continuation analyses resample paired tasks and task families.Source: §4.3 · p. 5

5 Results

Structured state improves repository evidence recovery across all receivers and continuation for most, but the leading models differ between tracks. The results also show that gains vary with task type, retained evidence, context representation, and scoring criterion.

5.1 Model Performance Across the Two Tracks

  • Claude Sonnet 5 and Gemini lead repository recovery at line-F1 0.263 and 0.261, while GPT-5.4 leads continuation with mean 92.76 and pass rate 52.1%.The model rankings differ between evidence recovery and state-dependent continuation.Source: §5.1 · pp. 5–6

5.2 Effects of Handoff Context

  • Structured state raises repository line-F1 by 0.114–0.162 for all six receivers and improves continuation mean scores for five; gains do not fall on the same tasks for every receiver.At least one receiver improves on 231 repository tasks and 536 continuation tasks, but all six improve on only 91 and 48 tasks, respectively.Source: §5.2 · p. 6, Figure 6 · p. 8

5.5 Context Representation and Access to Task State

  • For GPT-5.4, structured state beats narrative summaries by 5.42 mean-score points and 9.3 percentage points in pass rate; its advantage remains 3.04 points on 429 conversational and tool tasks.The controlled comparison uses untruncated summaries with equivalent source access; full history reaches a 57.9% pass rate, versus 85.3% with privileged target state.Source: Table 3 · p. 6, §5.5 · p. 7

5.3 Evidence Recovery Beyond Recent History

  • On 217 tasks with target paths only in earlier history, structured state yields line-F1 gains of 0.132–0.189 and retains 93.0% of observed paths, versus 12.6% for recent transcripts.A fixed extraction rule reaches file-F1 0.578 but line-F1 0.080, below receivers’ 0.189–0.263 range.Source: §5.3 · p. 6, Figure 4 · p. 6

5.4 Continuing from Recorded Progress

  • Across 114 structured-state continuations with a visible completed update, two repeat it, compared with none under recent transcripts; both structured-state repetitions occur on the illustrated task.Completed-action reporting rises from 77.3% to 87.3%, while update-count criterion satisfaction changes from 95.5% to 95.3%.Source: §5.4 · p. 7, Figure 5 · p. 7

5.6 When Mean Scores and Pass Rates Diverge

  • Structured state raises all-criteria pass rates for all six receivers by 7.44–12.18 percentage points, although Kimi gains 67 passing tasks while its mean score falls 3.33 points.GPT-5.4 and Claude have mean scores of 92.76 and 91.05, but pass rates of only 52.1% and 50.2%.Source: Table 4 · p. 8, §5.6 · p. 8

6 Discussion

A handoff evaluation must check what the receiver does with inherited progress, not only whether that information survived. The intended receiver and remaining task also matter when choosing a handoff representation.

6 Discussion

  • A completion record can reach the receiver without preventing a repeated operation, so checking the next action distinguishes missing information from failure to use it.A repeated handoff adds a further risk: one receiver’s incorrect completion report can become the next receiver’s input.Source: §6 · p. 8
  • Match the handoff representation to the remaining work: code localization calls for a relevant path or function, while reporting an operation calls for using its completion record.Compare representations with the intended receiver and task, rather than evaluating the message in isolation.Source: §6 · p. 8
  • The study covers single handoffs; following multiple receivers could reveal where an incorrect completion record originates and whether later models correct it.The proposed extension would compare each completion record with actions actually executed.Source: §6 · p. 8

7 Conclusion

HSC-Bench evaluates takeover through repository evidence recovery and continuation from inherited task state. Results distinguish these abilities: structured state improves recovery across all six receivers, but recovery leaders do not necessarily lead continuation.

7 Conclusion

  • Structured state improves repository recovery across all six receivers, with the largest gains when relevant paths disappear from recent transcripts.Models leading evidence recovery do not necessarily lead task continuation.Source: §7 · p. 8
  • Receivers sometimes repeat completed operations despite explicit completion records, distinguishing recovery of prior information from appropriate continuation.The benchmark checks predictions, responses, and actions against inherited task state.Source: §7 · p. 8
  • Assessing task takeover requires checking both what receivers recover and whether their next actions respect work already completed.The findings motivate assessing handoff representations together with the models that use them.Source: §7 · p. 8

A Data and Evaluation Details

The benchmark draws repository trajectories and continuation scenarios from distinct selection pipelines, then prepares matched handoff contexts and evaluates six receiver models.

A Data and Evaluation Details

  • Repository selection aligns CodeTraceBench trajectories [13] with ContextBench annotations [14], retaining 300 trajectories across 327 eligible tasks after exclusions.The retained GPT-5 histories include 193 OpenHands, 52 SWE-agent, and 55 mini-SWE-agent trajectories.Source: A Data and Evaluation Details · p. 10
  • Repository predictions allow at most 12 files and 12 spans per file, with invalid predictions scored as empty.Overlapping spans are merged and unsupported predictions remain in the precision denominator.Source: A Data and Evaluation Details · p. 10
  • Continuation selection retains 205 conversational, 386 tool, and 148 file-work tasks after plausibility, ambiguity, and specification checks.The retained tasks span five conversational, seven tool, and five file-work scenario families.Source: A Data and Evaluation Details · p. 10
  • GPT-4.1 prepares structured state with task-specific character targets, while narrative summaries preserve observed facts under matched length caps.Recent transcripts are truncated to the visible token budget; summaries are whitespace-padded after character truncation.Source: A Data and Evaluation Details · p. 10
  • Six specified receiver models are evaluated at temperature 0 and top-p 1, with continuation sourced from gpt-5-2025-08-07.Tool continuations allow four model calls for order lookup, policy inspection, address updates, and credit calculation.Source: A Data and Evaluation Details · p. 10

B Continuation Criteria and Analysis

Continuation criteria assess reported conversational, tool, and file-work state, while the appendix also reports criterion-level satisfaction rates and describes repository recovery comparisons.

B Continuation Criteria and Analysis

  • Conversational reference values for C1–C8 come from the earliest matching candidate phrase across chronological source and receiver messages.Current facts come from the task specification, while C12 compares the accumulated total as an integer.Source: B Continuation Criteria and Analysis · p. 10
  • Tool criteria distinguish reported completion and status from action history, including whether address updates were completed, repeated, or prohibited.T9 counts calls across source and receiver histories, including failed calls; all evaluated tool tasks satisfy T10.Source: B Continuation Criteria and Analysis · p. 10
  • File-work criteria check reported paths, observations, rejected attempts, hypotheses, test success, rejected plans, and edit boundaries.These checks assess reported progress and scope, not executed edits.Source: B Continuation Criteria and Analysis · p. 10
  • File-work criterion satisfaction rates for recent versus structured state are F1 Relevant files 95.9% versus 98.0%, and F2 Test observations 12.6% versus 38.9%.Table 5 reports criterion satisfaction rates averaged equally across six receivers.Source: B Continuation Criteria and Analysis · p. 11, Table 5 · p. 11
  • Table 6 presents repository recovery overall and by annotated-path visibility, with structured-state File-F1 and repository-level bootstrap intervals for differences.The three visibility groups partition all 300 tasks.Source: Table 6 · p. 11

B.1 Which Requirements Improve?

Structured state improves some continuation requirements more than others: tool completion reporting and file-work test observations rise, while other criteria change little or decline.

B.1 Which Requirements Improve?

  • 10.0 percentage points for completed-action reporting and 7.8 points for update status are the tool-task gains with structured state.Update-count satisfaction changes from 95.5% to 95.3%, essentially unchanged.Source: §B.1 · p. 11
  • 12.6% to 38.9% is the rise in file-work test-observation satisfaction, while reported test-success satisfaction changes from 68.0% to 66.2%.Detailed test reports remain incomplete on many tasks even with structured state.Source: §B.1 · p. 11
  • 19.3 points is the conversational style-field improvement, whereas language-field matching falls by 10.7 points.These requirement-level changes differ within the conversational track.Source: §B.1 · p. 11

C Supporting Comparisons

The supporting comparisons quantify task-level localization changes and examine continuation under intact summaries, matched source access, and a source-model reference.

C Supporting Comparisons

  • 0.5782 file-F1 and 0.0799 line-F1 result from directly extracting file paths and positive line spans from structured state.The comparison yields file predictions for 299 of 300 messages and valid line predictions for 90.Source: C Supporting Comparisons · p. 11

C.1 Supplementary Continuation Comparisons

  • 284 of 739 summaries are truncated by the character cap, leaving 455 intact; the advantage remains positive when padding is at most 20%.Table 8 isolates intact conversational and tool summaries with the same source access as structured state.Source: §C.1 · p. 11, Table 8 · p. 12
  • 93.63 mean score and 53.0% pass rate are achieved by GPT-5 continuing from its own full history.Table 7 presents three recorded continuation examples under structured state, with observed outcomes and satisfied-criteria counts.Source: §C.1 · p. 11, Table 7 · p. 12

C Supporting Comparisons

  • 146–181 of 300 tasks improve in paired localization scores across receivers, while 21–54 decline; 90–131 improving tasks move from zero to positive line overlap.Figure 7 reports gains, ties, and losses relative to recent transcripts.Source: C Supporting Comparisons · p. 11, Figure 7 · p. 12

D Task Examples and Action Evidence

The examples show repository evidence recovered under structured state and a completed update repeated before the continuation limit without a final report.

D Task Examples and Action Evidence

  • Table 7 presents three recorded continuation examples focused on retaining earlier choices, recognizing completed operations, and reporting unfinished file work.Its last column summarizes observed continuation and the number of satisfied criteria.Source: D Task Examples and Action Evidence · p. 12
  • GPT-5.4 predicts 44 lines, including 41 of 55 reference lines, for line-F1 of 82/99 in the illustrated repository-location example.The location was missed under recent transcripts but recovered under structured state.Source: D Task Examples and Action Evidence · p. 12

D.1 Actions After a Completed Update

  • Qwen repeats a successful address update on its third call, before the four-call limit, and submits no final report.Table 9 covers update calls, including failed attempts and successful repeats for the same order and address; the two repeats occur under DeepSeek and Qwen on the illustrated task.Source: §D.1 · p. 12, Table 9 · p. 12

Related papers

Loading 2610.36523v1…