PaperlayerView exact revision on arXiv

Source-linked AI summary

The Unreliable Progress Bar: Can LLM Agents Reliably Report Task Progress Throughout Execution?

Boyang Wang, Yunhan Wang, Yalun Wu

arXiv:2609.08589v1cs.SEcs.AIcs.CL

TL;DR

The paper asks whether language models can reliably report task progress at every stage and where their reports fail. It evaluates this question on τ^2-bench and the controlled StageIF testbed, finding that reliability varies by stage and failure form across deployed models. These results support checking reports against independent task state rather than letting them alone control task flow.

  • Problem

    Whether language models can reliably report task progress throughout execution, and where and how those reports fail, has not been studied systematically despite agent frameworks using them for continue-or-stop decisions.

  • Method

    The paper evaluates progress reporting on τ^2-bench and StageIF, which places reporting checkpoints across scripted task lifecycles and separates acting, omission, malformed reports, and wrong values.

  • Results

    Reporting reliability depends on task stage and setting: most tested models lose accuracy mid-task and recover after completion, while the newest generation closes that drop but becomes conservative at the finish line.

  • Takeaways & Limitations

    Progress reports should be scored at every stage and checked against independent task state rather than treated as sole control authority for task flow.

  • Takeaways & Limitations

    Measurements characterize deployments at a point in time and are unsuitable for vendor comparison or procurement decisions; deployment identifiers are model-and-serving bundles, not rankings.

Abstract

from arXiv · show

Recent large language models can emit task-progress signals that agent frameworks use to decide whether a task should continue or stop, yet whether a model can reliably report its task progress at every stage of a task, and where and how its reports fail, has not been studied systematically. We evaluate this ability on the public benchmark $τ^2$-bench and on StageIF, a controlled testbed in which reporting checkpoints are placed across the task's lifecycle. Both settings require reports at multiple task stages. We find that reporting reliability depends on the stage a task has reached, and that almost every deployed model we test is reliable at some stages and unreliable at others. Where reporting breaks down is not the same everywhere. Most deployed models lose accuracy once work is under way and recover once the task is done. The newest generation closes that mid-task drop and instead grows conservative at the finish line. Our study exposes a capability gap in task-progress reporting and provides an evaluation protocol that spans the whole course of task execution for this ability on which agent operation depends. The findings indicate that agent frameworks should not control task flow on the strength of the model's state reports alone.

1 Introduction

The paper asks whether models can report lifecycle progress reliably throughout tasks, then evaluates where and how those reports fail. It finds stage-dependent weaknesses that vary across models and settings, motivating measurement across the full task lifecycle.

  • Motivation: Agent frameworks increasingly use model-generated signals to decide whether a task should continue or stop, making reliable progress reporting operationally important.The paper defines reliability as issuing the report when required and naming the stage implied by the environment state.
  • Motivation: 27.5 Mbps versus the required 200 Mbps left one telecom task unfinished, yet the model declared completion and the runtime stopped.The reference decision was to continue working, while the observed run called done().
  • Approach: The evaluation adds lifecycle-stage tags to customer-facing replies and scores them against the benchmark environment state, separating execution success from reporting accuracy.Some deployments omit reports, while others report stages the task has already passed.
  • Approach: StageIF places reporting checkpoints throughout scripted tasks, disables reporting where it would be erroneous, freezes the correct value before answering, and separates four failure layers.The layers are acting instead of reporting, omission, malformed reports, and wrong stage values.
  • Findings: The two settings are complementary: natural trajectories expose failures during execution, while scripted checkpoints isolate report delivery, stage correctness, and correct withholding.Bounded changes to reporting demands generally did not eliminate the deficits, and termination configurations differed in task outcome.
  • Contributions: The paper contributes a stage-dependent account of reporting weaknesses, including mid-task declines and completion-stage errors, plus a taxonomy and evaluation instrument for measuring them.Its contribution list also reports setting-dependent failure forms and bounded intervention tests.

2 The Progress Bar on τ 2-bench

On τ^2-bench, deployments differ in whether they report and in where their reports fail: some collapse mid-task, while others remain accurate mid-task but become conservative at completion. The benchmark reveals these patterns, while its natural dialogues require controlled measurement to locate checkpoints independently.

  • Setup: τ^2-bench evaluates tool-using agents in simulated telecom and retail customer tasks, with success judged independently by environment assertions, transfer actions, or database comparison.The benchmark adds a requirement that each customer-facing message end with a single lifecycle-stage tag.
  • Reporting coverage: Seven of eleven full-split deployments followed the reporting duty on 93.2–100% of checkpoints, while four did so on only 0.6–10.1%.Wrong-value patterns are analyzed only among deployments that usually report.
  • Stage-dependent accuracy: gpt-4.1 and gpt-5.5 were correct on 90.6–99.4% of pre-action checkpoints and 87.3–88.9% after completion, but only 5.8–11.5% mid-task.Among three deployments, accuracy fell by at least a factor of two and a half across mid-task bins despite the gold stage remaining POST_OBS.
  • Error form: Among five deployments worst at mid-task, 82–90% of well-formed mid-task errors named an already-passed pre-action state, while 10–18% prematurely named COMPLETED.These percentages concern wrong reports rather than all checkpoints.
  • Parsing: Envelope stripping changed interpretation: gpt-5.6-sol retained a mid-task trough, whereas gpt-6-astra reached 100.0% mid-task but only 48.1% at completion.Embedded values do not establish strict protocol compliance because the diagnostic is not a replacement for the main parser.
  • Interpretation: Final task success does not remove reporting errors, and report accuracy does not substitute for task success.One successful gpt-5.5 trajectory still misreported both mid-task stages, while premature completion reports also occurred on unfinished tasks.
  • Completion-stage errors: gemini-3.8-flash, gpt-6-astra after envelope stripping, and claude-opus-5 held 74–100% mid-task but only 48–64% at completion.Every completion-stage wrong value for these deployments named an earlier state.
  • Scope: Natural dialogues reveal stage-dependent weaknesses but cannot independently place reporting duties or distinguish the two pre-action stages; StageIF supplies that control.Controlled checkpoints make the duty’s position an experimental variable rather than an accident of dialogue.

3 StageIF: Measuring Lifecycle Reporting

StageIF turns lifecycle reporting into a controlled measurement problem by fixing histories and gold stage values before generation, then separating delivery, correctness, and withholding failures. This makes the reporting duty’s position in the task an experimental variable rather than an accident of dialogue.

  • Controlled testbed: StageIF fixes scripted checkpoint histories and derives the correct stage before the model answers, allowing report delivery and stage correctness to be scored independently.It also includes checkpoints where reporting is switched off because the correct behavior is a tool call without a report.
  • Measurement: The testbed measures whether a report is due, whether it was delivered, and whether it names the correct stage.An oracle reads runtime state before the model output, while a deterministic parser evaluates the response.
  • Failure taxonomy: StageIF distinguishes omission, tool use instead of reporting, malformed reports, and well-formed reports with wrong values.These failures can remain invisible when evaluation reads only the final environment state.
  • Interpretation: The controlled setup keeps runtime truth, model behavior, and trajectory outcome separate, so task truth is never inferred from the model’s report.This separates reporting failure from task failure: a useful reply can violate the reporting contract, while a valid report does not prove task success.
  • Study design: The study instantiates twelve scripted scheduling and customer-support scenarios with five frozen-history checkpoints covering all four stages and one no-report tool-call checkpoint.Seven of nine planned deployments pass identity checks and contribute 50,400 checkpoint positions.

4 Where Lifecycle Reports Fail

Lifecycle-reporting accuracy is stage-dependent: many deployments lose adherence during action-bearing intermediate checkpoints and recover at completion, but the failure composition differs across settings. Telecom declines often involve wrong stages, whereas StageIF declines are driven more by omissions and tool calls replacing reports.

  • Stage pattern: StageIF adherence falls at action-bearing checkpoints and recovers at completion, with substantial variation across deployments.The later cohort reproduces the shape with a smaller margin, and one deployment reaches its minimum one checkpoint later.
  • Stage pattern: 29.4–89.3 points separate nonterminal from terminal adherence for six of seven originally admitted deployments.The most extreme deployment reaches 97.4% terminal adherence but only 8.2% across intermediate checkpoints.
  • Failure modes: In StageIF, omitted reports and tool calls in place of reports explain much of the mid-task decline, while wrong values contribute less.Wrong-value errors reach 13.6 points at the post-action checkpoint and remain below 2 points elsewhere.
  • Failure modes: At post-done checkpoints, false-alarm rates reach 90–100%, showing that terminal accuracy does not guarantee correct withholding.Reporting the right value when a report is due and emitting no report when none is due are separate requirements.
  • Replication: Across an IT-helpdesk task and an English translation, seven of nine deployments reproduce a significant gap and eight or nine reproduce its direction.The exceptions indicate heterogeneity rather than a deployment ranking.

5 Bounded Tests of Reporting Interventions

Bounded changes to the reporting demand do not generally remove the mid-task deficit. The interventions instead show that reporting behavior shifts with the required value and continuation context, while different evaluation settings expose different failure profiles.

  • Interpretation: The two evaluation settings reveal different failures of the same reporting contract rather than identical error profiles.Natural dialogues expose wrong-stage values prominently, whereas controlled checkpoints expose omissions and tool calls replacing reports.
  • Changing the required report: Providing the true stage or replacing it with a constant does not generally close the reporting gap.For gold injection, upper confidence limits cover at most one sixth of the observed gap on six deployments with significant gaps.
  • Changing the required report: A trailing reminder reduces omission but leaves tool-call divergence largely unchanged, with a largest pooled gain of +9.3 points against 29–89-point gaps.The intervention changes some failure composition without eliminating the broader deficit.
  • Comparing duties: A final-reply duty reaches 99.0%, compared with 43.7% for a machine report and 57.4% for a status sentence at duty-active checkpoints.All three duties approach ceiling at the terminal checkpoint, so the contrast is not simply inability to emit a structured answer.
  • Changing the continuation context: Withdrawing a needed tool and delegating the unfinished step raises false completion from 6.2% to 64.4% across nine deployments despite unchanged task truth.This is a bundled continuation-context intervention and does not establish an internal explanation for the stage pattern.

6 Operational Consequences

Reporting and termination configurations are associated with different task outcomes, but the rollout does not isolate a causal gate-rule effect. The observed session-ending frequencies distinguish premature completion from missing-marker stops without decomposing their causal contribution to task success.

  • Task outcomes: The marker-reporting configuration completes 13.0 points fewer tasks than the no-tool-call configuration in a validating rollout.The rollout includes 12 scenarios, six repetitions, and eight deployments, with scenario-clustered CI [+7.1, +19.4].
  • Scope: The rollout contrasts reporting-and-termination configurations rather than isolating a gate-rule effect.The two arms match user messages but differ in full model input because only the marker arm receives reporting instructions.
  • Termination behavior: Premature completion stops occur on 1.6% of marker trajectories, while missing-marker stops occur on 22.7%.These frequencies describe how sessions ended and do not causally decompose the task-success difference.

7 Related Work

Prior work evaluates instruction following, tool use, milestones, and workflow control, but not the model’s own stage report as a runtime-facing value across active and inactive checkpoints. This paper positions its evaluation as filling that measurement gap.

  • Instruction-following and structured-output studies assess user-facing constraints, whereas this work evaluates a stage-dependent value consumed by the runtime.
  • Tool benchmarks, progress boards, state-based environments, and process evaluators score actions or milestones rather than the model’s own report of task state.
  • Existing approaches do not anchor reports to environment-derived state while distinguishing active from forbidden reporting checkpoints and separating failure types.
  • Prospective-memory studies motivate testing whether reminders repair reporting omissions, while runtime-control work motivates comparing model-emitted control with explicit workflow state and verifier-paired exit.

8 Conclusion

The paper concludes that task-progress reports are stage-dependent and should be evaluated against trusted runtime state across the full task lifecycle. Because final success or a terminal report can coexist with failures during execution, reports should not serve as sole control authority.

  • Findings: Reporting reliability varies by task stage, with models showing mid-task declines or completion-stage errors.The newest generation of one family closes the mid-task collapse but becomes conservative at the finish line.
  • Measurement: StageIF separates report omission, tool calls instead of reports, malformed reports, and wrong stage values.Its oracle freezes the correct lifecycle value before the model answers, while the parser evaluates report realization and value equality.
  • Interpretation: Final task success and an accurate terminal report do not establish reliable reporting throughout execution.Task outcome and reporting protocol performance are parallel tracks, so a successful task can still violate the lifecycle-reporting contract.
  • Implication: Where independent task state exists, agent frameworks should check progress reports against it rather than treating them as sole control authority.The paper defines task truth through trusted runtime state and keeps model behavior separate from model-reported state.
  • Interpretation: The tested reporting-demand interventions did not generally remove the observed deficits, and their effects identify bundled behavioral changes rather than internal mechanisms.The intervention bundle estimates a total effect, while deployment identifiers denote model-and-serving configurations rather than a ranking.
  • Limitations: The study’s measurements characterize deployments at a point in time and are unsuitable for vendor comparison or procurement decisions.The authors deliberately present no ranking, and reported token usage is a recomputable lower bound rather than total project cost.

F Framework termination-signal survey

The survey finds that model-generated signals are the majority pattern for continue-or-stop decisions in the sampled frameworks, while source audits reveal structured tool signals and naming choices that can terminate runs. Its prevalence claim is bounded by purposive sampling, changing defaults, and incomplete evidence.

  • Survey classification: The survey distinguishes runtime-owned control from model-emitted signals, including textual markers, tool-based signals, and other model outputs.Table 5 groups heterogeneous interfaces by the signal consumed by the runtime rather than ranking frameworks.
  • Survey interpretation: A model-generated signal can be implicit: under a no-tool-call gate, every natural-language turn participates in the continue-or-stop decision.The paper treats StageIF as a stress test of whether such signals are fit to serve as control authority, not as evidence that one textual protocol is universally deployed.
  • Survey interpretation: 45.5% of trajectories stall under a no-tool-call gate because the agent speaks at a point requiring action.This rollout measures the concern that acting where the stage called for speaking can become a wrong lifecycle signal.
  • Source-level audit: Source audits show runtimes commonly read structural tool-call signals, while intermediate progress reports remain optional or flow through separate structured channels.The five audited runtimes are an added ecological niche and do not close the survey’s evidence gaps.
  • Source-level audit: OpenHands records a text-only, tool-free model reply as FINISHED even though a separate STUCK status exists for loops or inability to proceed.This is a naming and control-path fact: a quiet mid-task run can be recorded under the completion status.

G τ 2-bench stage and position denominators

The τ^2-bench analysis separates reporting behavior from task outcome and examines stage- and position-specific accuracy. Mid-task reporting can deteriorate sharply, while terminal behavior and position-normalized views require careful denominator and state interpretation.

  • Denominators and scoring: Table 6 separates strict-parser counts, envelope-stripped readings, and completion categories across planned checkpoint lanes.The gold rule marks completion from environment assertions and, for 20 transfer tasks, the required transfer action.
  • Parsing and deployment detail: The analysis reports per-deployment rates after strict and envelope-stripped parsing, including a 28.6% to 42.3% mid-task shift for gpt-5.6-sol after stripping.Envelope coverage differs substantially across deployments, so parsing can materially change the observed stage pattern.
  • Task outcome: The benchmark’s native done() signal scores zero on 63 of 114 gpt-4.1 tasks and 30 of 114 gpt-5.5 tasks in no-user mode.This native stop signal is distinct from the added lifecycle-stage tag, so zero reward alone does not prove a false completion report.
  • Position effects: Mid-task accuracy falls from the first to last quintile for claude-sonnet-5, gpt-5.5, and gpt-4.1: 30.5% → 5.1%, 21.7% → 7.3%, and 12.4% → 4.7%.The two envelope-stripped generations instead show 49.4% → 47.1% and 100% throughout.
  • Position effects: 83.7% of pooled compliant lanes are correct before acting, 11.1% at the lowest mid-task quintile, and 75.1% at completion.The pooled mean covers four compliant deployments without requested thinking.
  • Position-normalized view: The position-normalized Figure 11 pools each trajectory into ten bins but mixes lifecycle states within each bin.Its last bin can combine completion checkpoints with mid-task checkpoints from trajectories ended early, so the main figure is organized by state.

H Per-experiment admission and exclusion

The experiments use explicit admission checks, repeated-measures accounting, and separate audit boundaries. These controls limit denominator errors and clarify which deployment, manipulation, and intervention results are interpretable.

  • Measurement discipline: 50,400 logical positions are repeated measurements across 12 scenario clusters and 20 repetitions, not 50,400 independent tasks.The analysis also keeps behavioral results, framework documentation, and source audits separate rather than pooling them.
  • Admission and exclusions: Each study freezes its own identity precheck, so admitted deployment sets differ and exclusions consume no denominator.Table 7 records planned deployments, admitted counts, and exclusion reasons from archived precheck and summary files.
  • Vendor-positioning check: Eight of nine planned deployments have explicit vendor claims for agentic or tool-use work, but the check evaluates documentation rather than the claimed capabilities.The two deepseek-v4 identifiers are rolling aliases, and two vendor blogs were available only through web archives.
  • Scoring audit: Equal-strictness rescoring identifies 147 genuine wrong-value cases after retaining stricter original verdicts as a forensic record.The result includes hallucinated out-of-vocabulary stage values.
  • Failure decomposition: 172 protocol failures are complete omissions, while 15 intermediate checkpoints issue tool calls instead of reports.This motivates separating interaction-mode divergence from report omission.
  • Robustness: Every deployment has a negative and significant intermediate coefficient, ranging from −0.135 to −0.879 in the mixed-model analysis.The pooled logistic GEE coefficient is −5.435 with odds ratio 0.004 and 95% CI [0.001, 0.022].
  • Interventions: The intervention bridge compares completion-only, human-readable stage, and machine-tag obligations with and without reminders across frozen histories.At terminal checkpoints, the three obligations reach 99.0%, 100.0%, and 97.8%; reminders move nine-deployment means by at most +6.5 points.

R Primary and diagnostic metrics and the protocol’s task tax

The paper distinguishes stage-conditioned control adherence from conditional report validity and evaluates whether protocol demands impose a task-quality cost. The results separate divergence from reporting validity and find no consistent task-quality tax, though divergence can increase for some deployments.

  • Metric definitions: Stage-conditioned control adherence bθz is the primary metric, while conditional report validity bϕz excludes assistant tool-call divergence from its denominator.The distinction aligns the primary measure with agent-policy adherence and treats protocol validity as a diagnostic.
  • Metric interpretation: The primary bθ gap persists more broadly than the no-tool-call bϕ gap, confirming that divergence and report validity are distinct.The paper therefore does not reduce all failures to malformed or incorrect textual reports.
  • Task-quality tax: Protocol demand has no consistent task-quality effect and is not required for tool selection, although it amplifies divergence on two deployments.Task and protocol outcomes are evaluated as separate axes rather than as one joint outcome.
  • Divergence intervention: Without protocol demand, two deployments speak on only 55.3% and 50.8% of intermediate checkpoints; requiring the protocol lowers those rates by another 15.8 and 10.3 points.Six of nine deployments are unaffected, bounding how divergence rates should be interpreted.

S Continuation-context interventions: design and per-deployment analysis

The continuation-context interventions test whether changing tools, responsibility, and feedback cues alters progress-report emission or false completion while lifecycle truth remains fixed. Across complementary analyses, interventions change reporting behavior but do not generally eliminate deficits, and repair prompts can recover omissions without correcting wrong state judgments.

  • Intervention design: The intervention bundle withdraws a second-step tool and delegates the pending step, jointly changing executability and responsibility while holding the unfinished lifecycle state fixed.Its estimand is the bundle’s total effect on report emission and unconditional false completion.
  • Pooled effects: +45.1 points report emission and +58.2 points false completion result from the pooled bundle intervention across nine deployments.Emission rises from 30.7% to 75.8%, while false completion rises from 6.2% to 64.4%; false completion increases on all nine deployments.
  • Decomposition: The fully crossed 2 × 2 × 2 follow-up varies responsibility, tool availability, and feedback cues while preserving the same unfinished lifecycle truth.Main effects are averaged over the other two factors across 12 scenarios, three wording families, and eight deployments.
  • Decomposition: Tool withdrawal is the only factor with a non-negative end-to-end effect on all eight deployments, ranging from +0.3 to +18.7 points.Its effects are positive in five of eight scenario-cluster intervals, and withdrawal increases reporting through additional reports that sometimes contain the correct POST_OBS state.
  • Repair diagnostics: Replay or direct state questions can recover omitted reports but do not reliably repair already incorrect state reports.Continuation recovered the correct tag on 45.1% of 1,677 omission-population responses, while an isolated question produced a signal on 100% of 2,654 requests and selected the correct value on 98.12%; separate wrong-state follow-ups remained incorrect.

W Runtime gate comparison: design and results

The validating rollout compares reporting-and-termination configurations in an environment where task success is derived from state. Configuration choices materially change outcomes, but the comparisons combine reporting instructions with gate behavior and therefore do not isolate a single gate effect.

  • Design: The rollout compares marker-gate B1, no-tool-call gate B2, and runtime-owned control A across 12 scenarios, six repetitions, and eight deployments.The environment rejects structurally invalid actions, enforces referential integrity, and derives task success from state.
  • Design: 2,304 trajectories are covered in Table 19, with turn-cap and transport terminations omitted from the reported outcome columns.The omitted termination rates range from 0.2–1.6%.
  • Results: B2 removes every dead end but stalls 45.5% of trajectories.Dead ends and premature stops require a marker to read, so removing marker dependence changes which outcomes are structurally possible.
  • Results: Task success rises from 40.8% under marker gate B1 to 53.8% under no-tool-call gate B2, a +13.0-point contrast.The scenario-clustered confidence interval is [+7.1, +19.4], but the contrast includes B1’s strict response to missing reports and does not isolate reporting instructions from gate behavior.
  • Results: The configuration contrast is heterogeneous: B2−B1 task success is positive on seven deployments and negative on one, while 110 of 131 dead ends occur on two deployments.Omission-heavy deployments incur dead ends, whereas divergence-heavy deployments often provide no natural-language turn for the marker gate to reject; this association is exploratory.
  • Failure outcomes: The marker gate ends 22.7% of trajectories for missing reports, while premature completion claims terminate 1.6%.These frequencies do not identify how much of the success difference each failure causes; the marker gate typically loses unfinished work before completion.
  • Interpretation: Surviving terminal reports are selected by earlier marker-gate survival, so their 4.2–38.0% intermediate adherence and CLEAN terminal values cannot establish a rollout-stage gap.All 235 surviving marker-gate terminal checkpoints were CLEAN, but those trajectories first survived earlier marker-gated turns.
  • Interpretation: The bounded conclusion is that model-generated lifecycle reports can provide diagnostic evidence but are unsafe as the sole authority when runtime events can derive the state.This conclusion follows from a configuration comparison rather than evidence that one architecture is universally superior.
Loading 2609.08589v1…