Source-linked AI summary
Grounded Checklist Partial Credit for Agent Skill Trajectories
Suliu Qin, Lu Yin, Xilu Wang
TL;DR
Long-horizon agent evaluation often collapses trajectories into binary verifier outcomes, limiting measurement of partial progress and procedural skill effects. GCPC addresses this with human-governed, LLM-instantiated checklists scored from execution-log evidence, followed by scripted official-outcome handling. On SkillsBench, it better discriminates PASS and FAIL outcomes than holistic judging and aligns more closely with human progress assessments.
Problem
Binary task-level success rates reveal limited information about partial progress and skill-driven execution changes, while trustworthy fine-grained checklists are costly or unreliable to create at scale.
Method
GCPC uses reusable human rules to generate validated task-specific checklists from each task instruction and official verifier, scores items from execution-log evidence, and applies the official outcome afterward.
Results
GCPC achieves higher AUC than holistic judging on the shared SkillsBench subset, 0.689 versus 0.619, and aligns more closely with human assessments of progress.
Takeaways & Limitations
GCPC provides a finer-grained, auditable measure of agent trajectory progress beyond binary task outcomes.
Abstract
from arXiv · showhide
Language-model agents increasingly tackle long-horizon tasks in interactive environments, yet their evaluation commonly relies on task-level success rates by reducing an entire execution trajectory to whether the task passes an official verifier. This binary score hides partial progress and is particularly limited for procedural agent skill evaluations, since a skill can alter execution without changing the final outcome. While checklists provide finer-grained evaluation by scoring individual task requirements, costly manual authoring and unreliable automatic generation make trustworthy evaluation difficult to scale. To address these challenges, we introduce Grounded Checklist Partial Credit (GCPC), a human-governed and LLM-instantiated partial-credit evaluation of agent trajectories. Humans define reusable rules once, from which an LLM instantiates a task-specific checklist grounded in the task instruction and official verifier. To keep judgment tied to evidence, a judge scores each item from execution log evidence alone and abstains when evidence is missing. A separate scripted step then applies the official verifier outcome to the score. Across a 4,455-trajectory, deduplicated SkillsBench evaluation population, GCPC better discriminates official PASS and FAIL outcomes than holistic judging on the shared subset (AUC 0.689 vs. 0.619). Human evaluation on 96 trajectories from 12 tasks shows that GCPC aligns more closely with human assessments of progress. Applied to 1,946 matched with/without-skill pairs, GCPC exposes the effects hidden by pass@1: among 879 pairs whose binary outcome does not change, 20.9% improve by more than 0.10 while 18.7% regress by the same margin. The GCPC pipeline also transfers to Terminal-Bench and SWE-bench, demonstrating applicability beyond skill-conditioned evaluation.
1 Introduction
Binary task outcomes conceal partial progress and procedural skill effects, motivating GCPC, a human-governed and LLM-instantiated checklist method for auditable trajectory evaluation.
- Motivation: Binary evaluation can assign identical FAIL outcomes to runs satisfying three, five, or nine task requirements.Matched skill and no-skill runs can likewise remain failed while their satisfied requirements increase or decrease.
- Motivation: Human-authored rubrics require repeated task-specific construction, while LLM-generated checklists can omit, duplicate, or misprioritize requirements.These limitations make trustworthy fine-grained evaluation difficult to scale.
- GCPC: GCPC uses reusable human rules to guide an LLM in grounding task-specific checklist items in the task instruction and official verifier.A separate judge scores items from execution-log evidence, and fixed postprocessing incorporates the official outcome afterward.
- Illustration: 0.300, 0.500, and 0.900 partial-credit scores distinguish three official-FAIL runs on sales-pivot-analysis.The highest-scoring run supports all nine checklist items, but the rejected saved artifact contributes a violated endpoint item.
- Validation: 0.689 versus 0.619 AUC shows GCPC better discriminates official PASS and FAIL outcomes than holistic judging on the shared SkillsBench subset.The contribution summary also reports closer agreement with independent human assessments.
2 Related Work
Prior work supplements final-state success with intermediate structures and checklists, but scalability, grounding, and trustworthiness remain unresolved; GCPC addresses this through verifier-grounded partial credit.
- Agent Evaluation Beyond Task Success: AgentBoard, WebCanvas, Mind2Web 2, and Odysseys measure intermediate accomplishments beyond final task success.These approaches demonstrate the value of structured progress evaluation for long-horizon agents.
- Rubrics and Checklists: Human-authored requirements offer semantic control but require substantial task-specific labor, while automated approaches reduce authoring cost.The related work positions checklist decomposition as an alternative to holistic judgment.
- Rubrics and Checklists: Model-generated rubrics can underperform human-authored rubrics, and checklist guidance does not necessarily improve holistic judging.These findings motivate explicit provenance, checklist freezing, and separated judgment and aggregation.
- Agent Skills: Skill benchmarks primarily report verifier outcomes or whole-trajectory diagnostics, limiting insight into partial progress when skills alter requirements without changing PASS or FAIL.GCPC analyzes existing skill-benchmark trajectories rather than introducing another skill benchmark.
- Positioning: GCPC combines fine-grained requirements with verifier grounding to analyze execution changes that binary skill evaluation leaves unresolved.Its positioning emphasizes item-level task-text quotations and links to executable benchmark checks.
3 Method: Grounded Checklist Partial Credit
GCPC separates human-governed checklist synthesis from evidence-grounded trajectory scoring, then reports complementary metrics with deterministic official-outcome handling.
- Checklist Synthesis: Reusable human rules define item contracts, provenance requirements, coverage auditing, and mechanical validation gates for checklist synthesis.The LLM applies these rules to each task instruction and official verifier.
- Checklist Synthesis: Each task receives a frozen checklist synthesized without evaluated trajectories, outcomes, or oracle solutions.Freezing prevents trajectory-specific adaptation and enables reuse across trajectories.
- Checklist Synthesis: Checklist items use assertion, generated, or judge routes, and all 713 production items pass deterministic validation gates.Validation checks verbatim task grounding, verifier references, and requirement-level content.
- Trajectory Scoring: A judge labels each checklist item Yes, No, or Abstain using only displayed tool outputs and file contents from one execution log.Abstain indicates insufficient evidence rather than failure.
- Trajectory Scoring: The official outcome is incorporated only after item judgment through deterministic postprocessing.PASS can overwrite assertion-route items as Yes, whereas FAIL leaves item judgments unchanged and adds a virtual endpoint item.
- Evaluation Metrics: Checklist partial credit measures the fraction of satisfied criteria, excluding Abstain items from its denominator.Evidence coverage separately reports the fraction of criteria with sufficient execution-log evidence.
- Evaluation Metrics: Perfect equals one when no checklist item is judged No, while excluding the virtual official-outcome endpoint.With the endpoint included, Perfect agrees with official outcomes on 97.6% of 4,455 trajectories, versus 66.4% when excluded.
4 Experiments
Across SkillsBench and additional benchmarks, GCPC improves outcome discrimination, aligns more closely with human progress judgments, exposes skill effects hidden by binary evaluation, and transfers beyond skill-conditioned tasks. Its reliability depends on grounded, task-specific checklists, evidence-sensitive judging, and a separate official-outcome step.
- Evaluation setup: 4,455 deduplicated SkillsBench trajectories formed the evaluation population, with official PASS/FAIL outcomes and task-clustered uncertainty estimates.The primary comparison uses trajectories scored by both GCPC and the holistic judge.
- RQ1: Outcome discrimination: 0.689 versus 0.619 AUC: GCPC discriminated official outcomes better than holistic judging on identical trajectories.The paired ΔAUC was +0.056, with a 95% confidence interval of [−0.002, 0.118] and positive results in 97.1% of bootstrap draws.
- RQ2: Human agreement: 0.730 versus 0.613 pairwise agreement: GCPC matched human trajectory preferences more closely than holistic judging, and correlated more strongly with progress ratings.The correlations with human progress ratings were ρ=0.382 for GCPC and ρ=0.218 for the holistic judge.
- RQ3: Hidden skill effects: 0.227 median increase for FAIL→PASS pairs and 0.222 median decrease for PASS→FAIL pairs show that GCPC moved with official-outcome transitions.These matched comparisons use changes from without skill to with skill.
- RQ3: Hidden skill effects: 20.9% of 879 FAIL→FAIL pairs improved by more than 0.10, while 18.7% regressed by more than 0.10.GCPC therefore exposed skill-associated changes when binary evaluation reported no change and identified the checklist criteria that changed.
- RQ4: Trustworthy design: Task-specific checklists outperformed fixed questions, while the official endpoint supplied a larger PASS–FAIL separation increment than PASS propagation alone.GCPC achieved AUC 0.689 versus 0.613 for the fixed checklist; the official endpoint contributed +0.194 versus +0.034 for PASS propagation.
- RQ4: Trustworthy design: 73.5% of criterion judgments changed to No, and 78.1% changed to No or Abstain, after decisive evidence was replaced with a plausible incorrect value.This tests whether judgments respond to the execution-log evidence they are intended to assess.
- RQ5: Reliability and transfer: GCPC transferred to unseen tasks and benchmarks, including Terminal-Bench, where it scored passing trajectories higher in 40 of 69 decidable pairs and lower in only two.On the example pypi-server task, GCPC distinguished substantial partial progress with scores of 0.67 and 1.00.
5 Limitations
GCPC’s informativeness is bounded by what the task instruction, verifier, and execution log make observable, and it inherits dependencies on judge and verifier reliability.
- Provenance checks ensure checklist criteria are traceable to the instruction, verifier, and execution log but do not guarantee that every valid requirement is captured.
- GCPC may provide little additional information when execution logs lack sufficient evidence.In SWE-bench, 58 of 74 decidable pairs received the same GCPC score, and 42.1% of criteria received Abstain.
- GCPC depends on judge models and the official verifier, inheriting criterion-judgment variation and verifier errors.Its stronger agreement with PASS/FAIL is therefore not independent validity evidence.
6 Conclusion
GCPC complements binary task outcomes with an auditable criterion-level view of execution progress through reusable rules, checklist synthesis, validation, evidence-based judgment, and deterministic aggregation.
- GCPC provides an auditable criterion-level view of execution progress alongside binary task outcomes.
- The pipeline combines reusable human-defined rules, task-specific checklist synthesis, provenance validation, evidence-based criterion judgment, and a separate deterministic official-outcome step.
- Experiments show that GCPC reveals skill-associated improvements and regressions hidden when PASS/FAIL outcomes remain unchanged.
- Reliability and transfer experiments apply the same pipeline beyond skill-conditioned evaluation, while informativeness remains limited by trajectory evidence.
Ethical Considerations
GCPC’s partial-credit granularity creates a risk that agents optimize for checklist wording rather than underlying requirements, so the authors release it as an evaluation and audit instrument, not a validated reward function.
- GCPC’s individually verifiable criteria may be attractive as reinforcement-learning rewards that agents can optimize directly.
- An optimized agent could fabricate tool outputs or use superficially matching phrasing to satisfy criteria without completing the underlying requirement.
- The authors release GCPC as an evaluation and audit instrument rather than a validated reward function under active optimization.
- The paper identifies no substantial concern specific to this contribution about fairness, physical safety, or security.