Source-linked AI summary
Push Your Agent: Measuring and Enforcing Quantitative Goal Persistence in Long-Horizon LLM Agents
Yuandao Cai, Yuzhang Zhu, Liyou Gao, Wensheng Tang, Shengchao Qin
TL;DR
Long-horizon agents may make useful local moves yet stop before an explicit count goal is verified. This paper defines Quantitative Goal Persistence and evaluates it with PushBench and progress-state controllers. The controllers improve success in controlled tasks, while high-count persistence remains challenging for frontier-agent systems.
Problem
Final success rates can obscure premature stopping, repeated work, duplicate submissions, and completion claims that exceed verified progress.
Source: §1 · p. 1Method
Results
69–78% success for state-tracking retrieval and 25–50% for backlog tracking, while standard and completion-gated controllers complete no work-unit task instances.
Source: §1 · p. 2Takeaways & Limitations
High-count persistence remains nontrivial even for stronger black-box frontier-agent harnesses, and QGP-style metrics complement end-to-end task success with a direct measure of verified progress.
Source: §10 · p. 8The controllers require online verifier or checker feedback and do not directly apply when progress can only be judged at the end by a human.
Source: §11 · p. 9
Abstract
from arXiv · showhide
Long-horizon language agents can make many plausible local tool calls yet fail to persist until a requested count is actually complete. We study this gap as Quantitative Goal Persistence (QGP): whether an agent keeps working until an external verifier confirms enough distinct valid items. PushBench turns this into a benchmark for repository-artifact collection and verifier-backed work units, so repeated work, duplicate submissions, false completion, and progress drift are measured directly rather than hidden behind a final success flag. In matched controller comparisons, a state-tracking retrieval controller reaches 69-78% success while eliminating duplicate submissions, and a backlog-tracking work-unit controller reaches 25-50% success in settings where standard and completion-gated controllers complete no task instances. Black-box frontier-agent evaluations with Claude Code (Sonnet 4.6) and Codex CLI (gpt-5.4) solve many 50-artifact tasks but drop to 3 out of 9 successes per condition at 100 artifacts. The results show that quantitative goals stress a different reliability requirement from local task competence: agents must maintain verified progress and stop only when the requested work is complete.
1 Introduction
The paper frames quantitative goal persistence as maintaining externally verified progress until a requested count is met, and introduces PushBench and controller interventions to study it. State-tracking approaches improve success in controlled tasks, while larger-count frontier-agent tasks remain challenging.
1 Introduction
- Quantitative Goal Persistence (QGP) measures whether an agent continues until an external verifier confirms at least N distinct valid work units.It treats duplicate submissions, premature stopping, false completion, and reported-count error as task outcomes rather than incidental trajectory artifacts.Source: §1 · p. 1
- PushBench evaluates verifier-audited progress through repository-artifact collection and verifier-backed work-unit task families.The framework also includes coding-style and black-box frontier-agent evaluations to test whether progress checks remain useful in more realistic settings.Source: §1 · pp. 1–2
- STATEQGP tracks submitted identifiers and search state, while UNITQGP tracks pending, attempted, and passed units to prevent repeated or no-progress work.The interventions change how verified progress is exposed, tracked, and enforced, not the verifier or the valid-item set.Source: §1 · p. 2
- Under matched model and backend settings, state-tracking control achieves 69–78% success in repository-artifact collection and eliminates duplicate submissions; backlog-tracking reaches 25–50% in verifier-backed work-unit tasks, where standard and completion-gated controllers complete none.Frontier-agent evaluations solve many 50-artifact tasks but fall sharply at 100 artifacts.Source: §1 · p. 2
2 Related Work
Prior agent benchmarks assess interactive task success, tool use, trajectory progress, or memory, while QGP focuses on externally checkable quantitative stopping. The paper formalizes a task around a verifier, target count, and interaction budget, and defines related completion and repeated-work failures.
2 Related Work
- Existing interactive and software benchmarks establish task realism and executable success, while PushBench isolates quantitative persistence and externally auditable stopping conditions.The cited benchmark landscape includes web, tool-using, and repository tasks.Source: §2 · p. 2
- Tool-use, middleware, planning, and trajectory-repair methods can improve local action choice and recovery, but typically leave the quantitative stopping invariant implicit.PushBench instead studies the controller contract between policy actions and verifier-visible progress state.Source: §2 · pp. 2–3
- QGP evaluates the stopping condition itself, making false completion, premature stopping, duplicate work, and reported-count error task failures rather than post-hoc trajectory summaries.This contrasts with progress evaluations that assign partial credit to trajectories.Source: §2 · p. 3
3 Quantitative Goal Persistence
- A QGP task is defined by an environment, objective, external verifier, target count, and interaction budget.The verifier checks submitted work units, whose identifiers are tracked distinctly across attempts.Source: §3 · p. 3
- Repeated work is tracked separately from invalid work, including duplicate submissions and repeated attempts that do not change verified progress.Verifier-gated completion blocks final or ask-user termination while the valid count remains below the target, without making the underlying policy more capable.Source: §3 · p. 3
4 Benchmark Design
PushBench uses verifier-checkable tasks with explicit objectives, target counts, budgets, and success conditions. Its two complementary families range from repository-identifier retrieval to short checker-backed work loops, with low-to-medium unit difficulty intended to isolate persistence failures.
4 Benchmark Design
- PushBench computes outcomes from verifier-visible state, separating target completion from false completion, duplicate submission, reported-count error, and premature stopping.Its workflow separates task definition, action execution, verification, and metric computation.Source: §4 · pp. 3–4
- The controlled task families keep unit difficulty low-to-medium so failures are less likely to reflect unsolved open-ended coding or domain-expertise demands.The design aims to attribute repeated work, stale progress state, and unsupported completion claims more directly to QGP failures.Source: §4 · p. 4
- QGP-RepoScan tests whether agents maintain verifier-visible progress while searching repository-derived candidates and submitting distinct accepted identifiers.Task instances use local snapshots of requests, pytest, and flask; success requires at least N distinct identifiers accepted by the hidden verifier.Source: §4 · p. 4
- QGP-DataOps-lite requires distinct work units to pass a checker, extending retrieval with short inspect, answer-or-edit, and submission loops.Its units include CSV-field checks, file-metadata repairs, and consistency answers.Source: §4 · p. 4
- LLMPolicy and LangGraph-backed policies are controlled execution backends, sharing task manifests, budgets, verifiers, hidden valid sets, and summary fields for controller comparisons.These backends are not presented as representative samples of all agent architectures.Source: §4 · p. 4
5 Persistence Controllers
The persistence controllers add externally visible progress tracking and repair behaviors while leaving the verifier and valid items unchanged. STATEQGP addresses retrieval loops and duplicates; UNITQGP tracks backlog work and verifier recovery, with assisted behavior reported separately.
5 Persistence Controllers
- STATEQGP tracks submitted identifiers and search state, filters duplicate submissions, advances repeated searches, and blocks termination before the target count is met.When filtering leaves no identifiers to submit, it can redirect the action to the next unseen search page.Source: §5 · p. 4
- Standard execution lacks duplicate-aware repair, while verifier gating blocks unsupported termination without adding state over already-submitted artifacts.The paper reports passive and controller-assisted execution separately because external state enforces persistence invariants that may be difficult to leave to model memory.Source: §5 · pp. 4–5
- UNITQGP tracks unit status, redirects stale or no-submit loops toward pending or attempted units, and steers post-edit behavior toward verifier execution.It records verifier recoveries when a later repair turns an initially failed or incomplete trajectory into a passing unit.Source: §5 · pp. 4–5
6 Experimental Setup
The experiments use fixed, verifier-checked task families and matched controller comparisons to test quantitative persistence. STATEQGP improves repository-artifact success while eliminating duplicate submissions, although larger targets remain difficult.
6 Experimental Setup
- QGP-RepoScan tests unique repository artifact retrieval at N = 10, 25, 50, and 100; QGP-DataOps-lite tests verifier-passing work units at N = 3, 5, 10, and 20.RepoScan uses 36 task instances and DataOps-lite uses six backlogs per target.Source: §6 · p. 5
- Success requires valid_count ≥ target_count, based on unique artifact identifiers in RepoScan and checker-accepted work units in DataOps-lite, rather than the agent’s final message.Reported metrics include target success rate, average verified valid count, duplicate submit rate, and valid artifacts per step.Source: §6 · p. 5
- Controller comparisons hold task, budget, verifier, and valid set fixed, varying progress-state exposure, duplicate filtering, completion gating, and repair routing.The fixed manifests and target-dependent budgets support verifier-derived success labels.Source: §6 · p. 5
7 Experiment 1: Repository-Scanning Results
- 69.4–77.8% success and 0.000 duplicate submit rate across all STATEQGP rows show stronger repository-artifact persistence than standard and verifier-gated controllers.The success rates range from 69.4% for gpt-5.4 Native to 77.8% for gpt-5.4 LangGraph; matched tasks, budgets, verifiers, and valid sets isolate controller-visible progress state.Source: §7 · p. 6
- Many standard and verifier-gated rows that succeed at N = 10 fall to zero by N = 50 or N = 100; STATEQGP preserves zero duplicates but can still exhaust budget.Across target counts, memory reduces duplicates but high-target success remains model-dependent, while STATEQGP has the most consistent scaling.Source: §7 · p. 6
- Memory baselines range from 0 of 36 and 2 of 36 completions for gpt-4.1-mini to 72.2% for both baselines with gpt-5.4, below or near STATEQGP depending on model.For gpt-4.1, LETTA/MEMGPT reaches 44.4% and LangGraph+Memory 47.2%, versus 72.2% for STATEQGP.Source: §7 · p. 6
8 Experiment 2: Work-Unit Results
In QGP-DataOps-lite, standard and verifier-gated controllers complete no task instances, while UNITQGP and memory baselines recover some completion. The results concern low-to-medium-difficulty work units accepted by deterministic verifiers.
8 Experiment 2: Work-Unit Results
- DataOps-lite uses public-data snippets and repository-derived fixtures, with work units requiring inspection, checking, editing or answering, verification, and submission.The units are kept low-to-medium difficulty so the evaluation primarily measures persistence, state fidelity, and recovery rather than open-ended software engineering.Source: §8 · p. 6
- Standard and verifier-gated controllers complete no QGP-DataOps-lite task instances across all three models and both controlled backends.Verifier gating prevents unsupported completion but lacks the progress state needed to finish the backlog.Source: §8 · p. 6
- 50.0%, 29.2%, and 25.0% are UNITQGP success rates for gpt-4.1-mini, gpt-4.1, and gpt-5.4, respectively, under both Native and LangGraph.Matched standard and verifier-gated rows are all zero.Source: §8 · p. 7
- LETTA/MEMGPT success rises from 20.8% to 45.8% across gpt-4.1-mini to gpt-5.4, while LangGraph+Memory rises from 4.2% to 41.7%.Memory performance is model-dependent and can sometimes recover behavior comparable to verifier-aligned progress state.Source: §8 · p. 7
- UNITQGP steers agents back to pending units and checks to repair stale inspection, missing verifier calls, and no-submit loops; conservative no-progress limits still count as budget exhaustion.Standard runs may overstate accepted units, stop early, or exhaust budget without enough accepted submissions.Source: §8 · p. 7
9 Discussion and Additional Evaluations
The discussion frames QGP failures as losses of accumulated verified progress, not merely an inability to perform useful local work. Additional evaluations show that stronger infrastructure helps, but larger-count goals remain difficult, motivating QGP-style measures alongside end-to-end success.
9 Discussion and Additional Evaluations
- Passive and completion-gated controllers complete no QGP-DataOps-lite instances, while backlog-tracking and memory-based variants recover nonzero completion, as shown in Figure 2.Figure 2 compares Native and LangGraph standard, verifier-gated, and UNITQGP controllers with standard-controller memory baselines.Source: Figure 2 · p. 8
- Agents may find valid units yet lose runs through repeated accepted work, early stopping, or failure to use verifier feedback; QGP requires accumulated accepted progress rather than a plausible final claim.The same pattern appears in QGP-RepoScan and QGP-DataOps-lite.Source: §9 · p. 8
- Visible progress state lets controllers expose submitted and accepted units, filter duplicates, block unsupported final actions, and route work-unit agents back to pending checks.With verifier, tasks, and budgets fixed, the gains point to a runtime failure mode in which agents repeat work or stop before meeting the count invariant.Source: §9 · p. 8
- In QGP-SWE-lite, passive and verifier-gated agents complete no task instances, while UNITQGP reaches 37.5–41.7% success; Aider succeeds on most instances.The coding-agent harness changes the problem being measured.Source: §9 · p. 8
- Claude Code with Sonnet 4.6 and Codex CLI with gpt-5.4 solve many 50-artifact cases but each drops to 3/9 successes at 100 artifacts; checklist prompts do not improve paired success.In these black-box settings, persistence is intertwined with coding skill, repository navigation, hidden prompting, tool routing, and product-specific loops.Source: §9 · p. 8
10 Conclusion
- The conclusion proposes QGP-style metrics as a complement to end-to-end success, directly measuring whether agents preserve verified progress until the requested work is complete.The experiments suggest that high-count persistence remains nontrivial even for stronger black-box frontier-agent harnesses.Source: §10 · p. 8
11 Limitations
PushBench’s controlled benchmarks isolate quantitative persistence rather than measuring general agent ability, and their task and evaluation settings constrain broader conclusions. The paper also identifies online verification as a boundary for the controllers and cautions against overinterpreting small pilots.
11 Limitations
- The controlled QGP benchmarks isolate quantitative persistence; they are not general software-agent leaderboards or measures of overall software-engineering ability.The included models test whether persistence failures remain visible under stronger policies.Source: §11 · p. 9
- The benchmarks cover three repositories and 36 RepoScan tasks, plus public data snippets and repository-derived fixtures, limiting claims about broader workloads.The paper calls for broader repositories, data sources, and task types before generalizing to software or data-agent workloads.Source: §11 · p. 9
- Memory baselines are practical comparisons, not controlled ablations, because their runtimes and memory semantics differ.Their results indicate whether generic memory helps, not the effect of one isolated controller component.Source: §11 · p. 9
- Coding-style and frontier-agent evaluations trade control for realism, so they do not establish general weaknesses of commercial or open-source agents.Coding tasks mix persistence with coding and navigation skills; frontier products also include hidden prompting, memory, routing, retries, and updates.Source: §11 · p. 9
- STATEQGP and UNITQGP require online verifier or checker feedback, so they do not directly fit tasks judged only by a human at the end.When explicit count goals are only part of success, the paper says QGP should be complemented with qualitative evaluation.Source: §11 · p. 9
12 Ethical Considerations
- The main ethical risk is overinterpreting small controlled pilots as broad claims about model safety or competence.The authors make controller assistance explicit and reserve numeric claims for verifier-derived metrics.Source: §12 · p. 9
13 Open Data and Artifacts
- The anonymized artifact includes code, manifests, repository snapshots, aggregate summaries, table data, and reproduction scripts, but excludes raw model traces and private endpoint configuration.It also excludes non-anonymized local paths.Source: §13 · p. 9
A PushBench Benchmark Details
PushBench includes repository-artifact retrieval and verifier-backed work-unit tasks, with additional task variants described. Memory results depend on model and policy, while target scaling shows that small-target success does not guarantee persistence at larger counts.
A PushBench Benchmark Details
- QGP-RepoScan tests artifact retrieval across repositories and predicate families, while QGP-DataOps-lite uses deterministic checker-backed units over data snippets and repository fixtures.The work units include checks, metadata repairs, consistency answers, verifier execution, and accepted-unit submission; variants also include synthetic retrieval and code-location tasks.Source: A PushBench Benchmark Details · p. 12
B.1 Memory Baseline Breakdown
- With gpt-4.1, LangGraph+Memory raises standard LangGraph success from 30.6% to 47.2% but trails STATEQGP at 72.2%; with gpt-5.4, it reaches 72.2%.At gpt-5.4, it matches LETTA/MEMGPT and approaches LangGraph STATEQGP at 77.8%; memory performance is model-dependent.Source: §B.1 · p. 12
B.2 Target Scaling and Paired Uncertainty
- Many controllers succeed at N = 10 but collapse by N = 50 or N = 100, while STATEQGP remains nonzero at the largest target for all three models.Each target bucket aggregates the same nine fixed task instances; STATEQGP maintains zero duplicate submissions across targets.Source: §B.2 · p. 12, Figure 3 · p. 13
- For gpt-4.1 LangGraph, STATEQGP raises success over standard by 41.7 percentage points (95% interval 27.8 to 58.3), while verifier gating raises it by 8.3 points (interval -2.8 to 22.2).The intervals estimate paired differences in success rates, not uncertainty in the success rates themselves.Source: §B.2 · p. 13
B.3 Stateful Component Ablation
The ablation and broader evaluations distinguish passive memory or verifier gating from controllers that track and act on verified progress. Results cover repository retrieval, work-unit backlogs, and coding-style tasks, with the latter evaluations trading experimental control for realism.
B.3 Stateful Component Ablation
- The focused ablation compares duplicate filtering, page memory, their combination without buffered submission, and full STATEQGP on the same 36 QGP-RepoScan tasks.It tests whether passive memory features account for the controller’s results.Source: §B.3 · p. 13
- Duplicate filtering eliminates duplicate submissions but does not improve success beyond verifier gating; page memory reaches 44.4%, while full STATEQGP reaches 72.2%.The full controller’s paired success delta over the no-buffer dedupe-plus-page variant is 0.278, with a 95% bootstrap interval of [0.139, 0.417].Source: §B.3 · pp. 13–14, Figure 4 · p. 14
C DataOps-lite Detailed Results
- UNITQGP improves success over standard and verifier-gated controllers for every model-backend pair, while verifier gating alone has zero success delta over standard.UNITQGP is strongest on artifact-validation units and weaker on consistency-check and metadata-update units; memory-baseline successes concentrate in mixed backlogs.Source: C DataOps-lite Detailed Results · p. 14
D.1 SWE-lite Realism Evaluation
- QGP-SWE-lite uses issue-style repairs or checks that must pass a local verifier, but its results mix persistence with coding, navigation, editing, and harness behavior.The evaluation uses isolated workspace copies of requests, flask, and pytest across 192 executions.Source: §D.1 · p. 14
- UNITQGP reaches 37.5% success for Native and 41.7% for LangGraph on QGP-SWE-lite, while standard and verifier-gated controllers complete no task instances.Aider reaches 91.7% under both standard and UNITQGP execution, showing these units are feasible under its model and verifier budget.Source: §D.1 · p. 14
D.2 Black-Box Frontier-Agent RepoScan Evaluation
The black-box RepoScan evaluation tests whether frontier agents persist to high artifact targets, finding substantially fewer successes at 100 than at 50 artifacts. An explicit checklist changes duplicate and tool-use behavior but does not improve success.
D.2 Black-Box Frontier-Agent RepoScan Evaluation
- The evaluation used 18 RepoScan tasks at N = 50 or 100, two black-box harnesses, and default or checklist prompts.Agents used a local search, submit, status, and final interface; hidden valid identifiers remained inside the verifier.Source: §D.2 · p. 15
- The checklist changed duplicate and tool-use behavior but produced zero paired success delta for both agents and target counts.Table 7 reports checklist-minus-default deltas matched by task and agent.Source: Table 7 · p. 15, §D.2 · p. 15
- Seven of nine tasks succeeded at N = 50, but only three of nine succeeded at N = 100 in every agent-prompt condition.Both black-box agents achieved the same counts under default and checklist prompts.Source: §D.2 · p. 15, Figure 5 · p. 17