What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignment

1 Introduction

• Prior step-level evaluations can miss end-to-end performance [32], human-annotated ground truth is costly [8] [14] [22] [30], and outcome-focused evaluations offer little root-cause insight [9] [32].Source: §1 · p. 1
• Agent GPA assesses failures within and between goals, plans, and actions through seven metrics, operationalized as automated, reference-free LLM judges.Its metrics are Goal Fulfillment, Plan Quality, Tool Selection, Plan Adherence, Tool Calling, Logical Consistency, and Execution Efficiency; it was evaluated on TRAIL/GAIA, TRAIL/SWE-bench, and Snowflake Intelligence.Source: §1 · p. 1
• On TRAIL/GAIA, GPA judges identified 95.0% (267/281) and localized 85.8% (241/281) of human-labeled errors, versus monolithic-baseline maxima of 54.8% (154/281) and 49.1% (138/281).The monolithic baseline received architectural context about the evaluated agent system.Source: §1 · pp. 1–2
• LLM judges averaged 73.1% LLM-human accuracy agreement on TRAIL/GAIA and 82.3% on Snowflake Intelligence.Source: §1 · p. 2

2. Generalizability:

• 52.4% to 76.2% Plan Quality recall and 83.6% to 88.8% Logical Consistency recall show GEPA-optimized GPA judges matched or exceeded manually crafted judges on TRAIL/GAIA.Source: §2 · p. 2
• 49.8% to 83.1% Logical Consistency and 55.5% to 84.7% Tool Calling show GEPA-optimized GPA judges improved over manually crafted judges on TRAIL/SWE-bench.Source: §2 · p. 2
• State-of-the-art prompt optimization automates custom prompt retuning to create domain-specific evaluation criteria for new agent architectures and domains.Across the public datasets, the techniques broadly yielded comparable or better evaluation performance than traditional manual prompting.Source: §2 · p. 2

3. Consistency

• Repeated TRAIL/GAIA judge runs yielded identical scores and average Krippendorff’s α of 0.77; OpenEvolve raised Plan Quality α from 0.628 to 0.814.OpenEvolve optimized prompts using score standard deviation as its fitness objective; evolved prompts often reduced variance through more granular rubrics and explicit scoring criteria.Source: §3 · p. 2

Goal Progression and Fulfillment

• Goal evaluations often compare final answers with references, limiting scale, while simulations found goal drift across all evaluated trading agents, especially amid competing or shifting objectives [4].This motivates reference-free assessment of goal fulfillment when labeled answers are unavailable.Source: §2 · p. 2

Planning via Reasoning Traces

• Separating planning from execution can improve performance: Plan-and-Act [12] maps high-level steps to actions, while AdaPlanner [28] refines plans from feedback; evaluations still rely on verifiers, humans, or references [30].Plancraft [10] compares an agent’s successful-plan action count with the optimal count; reference-free evaluation of plan quality and adherence remains needed [30].Source: §2 · p. 2

Execution via Action Traces

• Final outcomes can hide unsafe or invalid actions, but reference-trajectory checks may penalize valid deviations from predefined paths [20].Existing trace evaluations compare tool calls or steps with an expected sequence, while rules-based checks can be overly rigid.Source: §2 · p. 2

Debugging from Execution Traces

• TRAIL [11] and MAST [6] classify trace failures, but static symptom taxonomies can conflate planning with execution; replay-based refinement [13] offers a debugging direction.Enterprise evaluation also needs to consider operational cost and efficiency [15], underscoring the need for a granular, reference-free framework to identify breakdowns.Source: §2 · pp. 2–3

LLM Judges

• Single LLM judges can miss details in long traces: TRAIL reports 11% accuracy [11], while AgentRewardBench finds overestimation of success [20], motivating specialized judges and comparative validation.Existing trajectory evaluations often use one prompt across agents [18]; the benchmark alignment of some industrial component-level evaluations is unclear.Source: §2 · p. 3

3 Agent Goal-Plan-Action (GPA) Framework

• The framework models agent failures as emerging within or at the intersections of Goal, Plan, and Action.Figure 1 visualizes these intersections.Source: §3 · p. 3, Figure 1 · p. 3
• GF checks whether completed actions satisfy the user’s goal; PQ assesses plan optimality; TS evaluates whether each subtask is matched to an appropriate tool.An optimal plan uses minimal actionable subtasks, balances detail, and responds to replanning triggers; TS also checks tool capabilities and whether a tool is needed.Source: §3 · p. 3
• PA measures adherence to planned steps, TC evaluates individual tool calls, while LC and EE assess trajectory-wide consistency and efficiency.LC checks grounding in prior context, system-instruction adherence, error recovery, and completion of self-generated tasks; EE detects redundant actions, unnecessary calls, and resource use, including for agents without explicit plans.Source: §3 · p. 3
• A suite of specialized LLM judges targets distinct GPA components and intersections instead of relying on a single all-purpose judge.Tool-related evaluations focus on agent-controlled tool selection and calling, not enterprise-specific tool quality such as retrieval relevance or API throughput.Source: §3 · p. 3

4 Datasets

• TRAIL/GAIA contains 117 traces from a multi-agent research system, while TRAIL/SWE-bench contains 31 traces from a single coding agent; Snowflake Intelligence adds 17 enterprise data-agent traces.Together, the three datasets cover general reasoning, software engineering, and data-analysis workflows.Source: §4 · p. 4, §4.1 · p. 4, §4.2 · p. 4

4.1 TRAIL/GAIA

• 117 TRAIL/GAIA traces [11] are expert-annotated and were generated by Open Deep-Research Agent, whose Manager and Search agents plan and use tools to answer challenging questions [21].The traces are split 50/50 into development and test sets, containing 289 and 281 annotated errors, respectively.Source: §4.1 · p. 4

4.2 TRAIL/SWE-bench

• 31 TRAIL/SWE-bench traces [11] capture Code-Act [29] using a sandbox, Python interpreter, and gitingest to patch GitHub issues [14].The 50/50 development and test sets contain 113 and 126 TRAIL-annotated errors, respectively.Source: §4.2 · p. 4

4.3 Snowflake Intelligence

• Snowflake Intelligence was evaluated on 17 internal traces from a production-grade data agent using text-to-SQL and composite retrieval tools for complex, multi-step data-science queries.This dataset focuses on agent failures in data-analysis workflows, unlike the general-purpose and Python-coding settings.Source: §4.3 · p. 4

5.1 Data Pre-Processing

• TRAIL/GAIA traces were preprocessed to fit judge context windows by extracting system instructions and new Manager or Search agent messages and removing duplicates.The preprocessing addressed raw OpenTelemetry traces that exceeded the LLM judges’ input context windows.Source: §5.1 · p. 4

5.2 Mapping Errors to GPA Dimensions

• Two annotators assigned TRAIL/GAIA dev- and test-set errors to one or more GPA dimensions by root cause, and a third annotator verified the mappings.Source: §5.2 · p. 4

5.3 Designing LLM Judge Architecture

• The judge architecture combines fixed, domain-invariant GPA metric definitions with domain-specific instructions, agent descriptions, human-labeled dev examples, and structured output templates.The custom component can be automated using prompt optimization techniques on the target dataset.Source: §5.3 · p. 4

5.4 Selecting Dataset-Specific GPA Judges

• The study deployed all GPA judges on TRAIL/GAIA, LC, EE, and TC on TRAIL/SWE-bench, and LC and EE on Snowflake Intelligence according to observable traces and evaluation goals.TRAIL/GAIA exposed explicit planning, reflection, and tool use; the CodeAct and Snowflake traces lacked explicit plans, and Snowflake evaluation focused on SQL intent match and efficiency.Source: §5.4 · pp. 4–5

5.5 Validating the GPA Framework

• Human-reviewed TRAIL/GAIA and Snowflake evaluations tested error identification, localization, classification, and alignment between judge scores and human judgments.For TRAIL/GAIA, annotators verified error detection and cited span IDs against two TRAIL judge baselines; alignment used accuracy and correlation measures. Snowflake alignment used accuracy, correlation, and NMAE against human scores.Source: §5.5 · p. 5

5.6 Evaluating Generalizability

• GEPA optimized generic GPA judge prompts, with test-set recall evaluated on TRAIL/GAIA and unseen-domain TRAIL/SWE-bench using a meta-judge verifier.The study used claude-4-sonnet for prompt optimization and metric evaluation; the meta-judge graded GPA outputs against TRAIL errors.Source: §5.6 · p. 5

5.7 Improving Consistency

• OpenEvolve refined prompts for metrics with baseline score standard deviation above 0.1, while repeated judge runs measured agreement, score variability, and rationale consistency.TRAIL/GAIA (n=59) and TRAIL/SWE-bench (n=31) used five runs per trace; Snowflake Intelligence (n=17) used ten. Measures included Krippendorff’s α, per-trace standard deviation, and Semantic Consistency Index.Source: §5.7 · p. 5

6.1 GPA Framework Validation

• All 570 GAIA and 239 SWE-bench human-annotated errors mapped to at least one GPA judge; on GAIA test, LC, TC, and EE accounted for 140, 128, and 119 of 281 errors.PQ was least frequent, with 14 GAIA test errors.Source: §6.1 · pp. 5–6
• 95% (267/281) of TRAIL-annotated GAIA test errors were captured by GPA judges, versus around 54% (151-154/281) for monolithic baselines.Providing the monolithic judge with control-flow context improved identification by only 1.1%.Source: Figure 2 · p. 6, §6.1 · p. 6
• TC achieved test-set F1 >0.92 and TS recall >0.97; PA and TS had 86.4% and 86.8% bucketed accuracy against human judgment.On Snowflake Intelligence, GPA judges averaged 82% agreement with human judgment and identified systematic error patterns traced to architectural root causes.Source: §6.1 · pp. 6–7, Table 5 · p. 7
• 86% (241/281) of annotated GAIA test errors were localized to the exact span ID by GPA judges, versus 49% (138/281) for the context-equipped baseline.The uncustomized baseline localized 31% (87/281); architectural context improved its localization but remained below the factorized suite.Source: §6.1 · pp. 6–7, Figure 3 · p. 7

6.2 Generalizability

• On TRAIL/GAIA, GEPA optimization raised PQ coverage to 76.2% and LC coverage to 88.8%, matching or outperforming manual custom prompts across all six metrics.Generic prompts covered 57.9% of LC errors and 33.3% of PQ errors; manual custom prompts raised those figures to 83.6% and 52.4%.Source: §6.2 · p. 8
• On TRAIL/SWE-bench, GEPA raised LC coverage from 49.8% to 83.1% and TC coverage from 70.8% to 84.7%, outperforming manual custom prompts on both metrics.The transfer to coding traces required no domain-specific manual retuning; EE coverage remained generally high across prompt types.Source: §6.2 · p. 8

6.3 Consistency of LLM Judges

• Across runs, GAIA EE and TS had Krippendorff’s 𝛼 of 0.934 and 0.907; SWE-bench LC and TC had 𝛼 of 0.960 and 0.83, with std ≤0.08 across metrics.On Snowflake Intelligence, 10 runs yielded 𝛼=0.66 for LC and 0.81 for EE; GAIA PQ and LC rationales were less semantically consistent than EE rationales.Source: §6.3 · pp. 8–9, Table 6 · p. 9, Table 7 · p. 9
• OpenEvolve improved Krippendorff’s 𝛼 by 30% (0.628 →0.814) for GAIA manual PQ and by 38% (0.539 →0.743) for SWE-bench TC.It was applied to metrics with standard deviation above 0.1, generating more granular rubrics to reduce ambiguity in boundary cases.Source: §6.3 · p. 9, Table 8 · p. 9

• GPA evaluates LLM agents across goals, plans, and actions, capturing diverse and subtle failure modes overlooked by monolithic or outcome-based methods.Source: §7 · p. 9
• Specialized GPA judges outperform monolithic evaluators in error identification and localization while maintaining strong alignment with human evaluators.Source: §7 · p. 9
• Automated prompt optimization adapts GPA judges to unseen domains and agent architectures, frequently outperforming manual prompt engineering.Source: §7 · p. 9
• GPA judges show high reliability across runs and can be further optimized for consistency with evolutionary prompt optimization.Source: §7 · p. 9
• Future work should extend GPA to embodied agents and refine reference-free metrics for goal fulfillment and plan quality.Source: §7 · p. 9
• GPA aims to serve as a reliable diagnostic tool that contributes to building more capable, interpretable, and trustworthy agents.Source: §7 · p. 10

Reproducibility statement

• The authors aim to open-source the Agent GPA evaluation framework to support reproducibility.They state that the release is intended to enable independent replication and extension of their results.Source: Reproducibility statement · p. 10
• The framework release will include code for preprocessing traces and running the LLM judges.Source: Reproducibility statement · p. 10
• Evaluation prompts are available in Appendix D of the paper.Source: Reproducibility statement · p. 10
• The authors plan to release the re-annotated and augmented TRAIL/GAIA dataset used in their experiments.Together with the framework and prompts, these resources are intended to enable independent replication and extension.Source: Reproducibility statement · p. 10

A.1 GPA Framework Validation: TRAIL/GAIA Coverage

• Coverage measures a judge’s recall on the specific subset of errors it is designed to detect.The analysis uses dev/test confusion matrices to examine coverage across TRAIL errors.Source: §A.1 · p. 12
• The GPA judges collectively outperform the baseline, after which the analysis assesses whether each specialist judge fulfills its intended role.Coverage is defined as recall on the specific subset of errors that a judge is designed to detect.Source: §A.1 · p. 12
• TC, TS, and EE coverage exceeded 91%; LC coverage stayed below 60%, and PQ test coverage fell from 88% to 71%.The authors suggest PQ may favor overt logical-consistency errors over subtler mistakes.Source: §A.1 · p. 13, Table 9 · p. 13
• The 0% test coverage for PA and PQ on low-impact errors is not reliable evidence because each category had at most two samples.The authors caution that the sample size is statistically insignificant.Source: §A.1 · p. 13, Table 9 · p. 13
• The authors suggest PQ’s test-set decline may reflect a bias toward overt logical-consistency errors, causing it to overlook subtle mistakes.They also suggest the judge may have learned superficial dev-set patterns rather than robust principles of plan adherence and quality.Source: §A.1 · p. 13
• PA and EE localized over 83% of errors, versus 41% for TC, 64% for TS, and 57% for PQ.The paper attributes the contrast to atomic errors being easier to pinpoint; localizing procedural flaws may require a causal trace of the model’s reasoning chain [18].Source: §A.1 · p. 13, Table 10 · p. 13

A.2.1 Caught Errors

• A judge's utility varies with error severity, so no single judge is universally optimal and aggregate scores mislead high-stakes evaluation.Effective evaluation requires a portfolio of specialized judges deployed according to error context and the desired balance between sensitivity and precision.Source: §A.2.1 · pp. 13–14
• F1=1.0: TC is nearly flawless on low-impact errors, while PA and PQ fail on a statistically insignificant sample (n≤2).The low-impact results are reported for TRAIL/GAIA.Source: §A.2.1 · p. 13, Table 11 · p. 14
• 99%: TS catches critical failures; TC provides the most balanced F1 performance, while LC is most precise when flagging a critical error.These high-impact results distinguish maximum sensitivity (TS), reliability (TC), and confidence (LC).Source: §A.2.1 · p. 13, Table 13 · p. 14

• Judge utility shifts with error severity, producing performance inversions and role-reversals; no single judge is reliable across all conditions.Effective automated debugging therefore requires a dynamic ensemble selected for the specific failure context.Source: §A.2.2 · pp. 14–15, Table 14 · p. 15, Table 15 · p. 15
• PA fails completely on low-impact errors, while TC consistently delivers perfect precision but low recall, making its feedback sparse but highly trustworthy.Source: §A.2.2 · p. 14
• For high-impact errors, PA is the top-performing localizer (F1=0.85), while TS is the highest-precision localizer (P=0.85).TS is described as the most reliable choice for pinpointing the source of a critical failure.Source: §A.2.2 · pp. 14–15, Table 16 · p. 15

B GPA Framework Validation: Extended Judge Agreement Statistics (TRAIL/GAIA)

• Three human annotators graded 117 dev and test traces along six GPA judge dimensions for the agreement analysis.The paper reports accuracy and correlation for each GPA judge against human judgment.Source: B GPA Framework Validation: Extended… · p. 15
• The LLM judges generally exhibited strong agreement with human annotations across the board, including accuracy and correlation with human judgment.The agreement metrics were calculated by comparing each GPA judge with human judgments.Source: B GPA Framework Validation: Extended… · pp. 15–16
• Descriptive mean Krippendorff’s α across GPA judge types was 0.7346 on dev and 0.6718 on test; global Krippendorff’s α was 0.7690 and 0.7387, respectively.The global values are described as supportive of tentative conclusive.Source: B GPA Framework Validation: Extended… · p. 16
• Human-human consensus judge agreement was 0.7009 on dev and 0.6674 on test.Source: B GPA Framework Validation: Extended… · p. 16
• The paper reports per-metric Cohen’s κ for human-versus-LLM agreement on both TRAIL/GAIA dev and test sets.Source: Table 17 · p. 16, Table 18 · p. 16

C GPA Framework Validation: Cross-GPA Metrics Agreement and Orthogonality Analysis (TRAIL/GAIA)

• Low agreement across all four measures shows that the six metrics detect complementary, semantically different errors rather than capturing the full failure spectrum individually.The analysis binarizes each metric’s scores to focus on whether it identifies a trace failure, rather than its severity.Source: C GPA Framework Validation: Cross-GPA… · p. 16
• PQ is the most independent metric, while EE–TC is the strongest pair across agreement measures, suggesting execution failures co-occur with tool-calling issues.Source: C GPA Framework Validation: Cross-GPA… · p. 17
• TS has mild associations with LC, EE, and PA; LC differs from PQ and PA, while PA aligns moderately with TS but weakly with other metrics.Most Jaccard scores are 0.04–0.40, confirming sparse co-activation, and phi correlations largely mirror kappa’s weak associations.Source: C GPA Framework Validation: Cross-GPA… · p. 17

D.1 Custom Instruction: Control Flow of Open Deep-Research

• The research-agent trace is hierarchical: a manager plans and delegates subtasks to a search agent, which surveys, plans, performs tool steps, refines its plan, and returns results.Source: §D.1 · pp. 17–18

D.2 Logical Consistency Judge: Generic LC Criteria & Custom Instruction

• The LC judge scores consistency from 0 to 3 by assessing trace reasoning against prior context, system instructions, and each agent’s conversation history.Source: §D.2 · pp. 18–19

D.3 Execution Efficiency Judge: Generic EE Criteria & Custom Instruction

• The EE judge scores execution from 0 to 3, focusing on sequencing, resource use, and redundant or wasteful actions regardless of plan success or adherence.Source: §D.3 · p. 20

D.4 Plan Quality Judge: Generic PQ Criteria & Custom Instruction

• The PQ judge evaluates the initial written plan against the context and tools available when it was created, excluding execution and extracting the plan from labeled or inferred planning sections.Source: §D.4 · p. 21
• The Plan Quality judge scores plans and replans on intrinsic quality, including tool fit and definitions, use of existing context, and whether replans respond efficiently to observed triggers.It must ignore execution steps, tool outputs, and agent actions when judging the initial plan; replan quality is assessed as a response to the trigger.Source: §D.4 · pp. 21–22

D.5 Plan Adherence Judge: Generic PA Criteria & Custom Instruction

• The Plan Adherence judge evaluates step by step whether execution follows the plan and replans, counting omitted or incomplete mandated steps as failures regardless of final-output quality.It identifies plans and replans in the trace, then assesses adherence to each in turn.Source: §D.5 · pp. 23–24

D.6 Tool Selection Judge: Generic TS Criteria & Custom Instruction

• The Tool Selection judge scores tool-choice appropriateness relative to goals and available tools, including suitability, instruction compliance, and awareness of tool capabilities and limits.It excludes call syntax, output interpretation, execution efficiency, and plan adherence from its assessment.Source: §D.6 · p. 25
• The Tool Selection judge assesses tool suitability for each task, compliance with mandated tools, and awareness of tool constraints, excluding calling quality, efficiency, and plan adherence.It compares tools by preconditions, postconditions, direct support, and guarantees, and checks whether internal steps should avoid tool use.Source: §D.6 · pp. 25–26

D.7 Tool Calling Judge: Generic TC Criteria & Custom Instruction

• The Tool Calling judge evaluates agent-controlled input validity and appropriateness, output interpretation, and handling of tool errors, while excluding tool choice, workflow efficiency, and external reliability.Its rubric covers arguments, queries, required parameters, preconditions, and grounded use of outputs.Source: §D.7 · p. 27

E Automatic Prompt Evolution and Optimization with GEPA

• The section presents prompts for each GPA metric before and after GEPA optimization.Source: E Automatic Prompt Evolution and… · p. 28

E.1 LC Starting Prompt (Pre-GEPA): Generic LC Criteria

• The starting Logical Consistency prompt scores trace reasoning across planning and execution, rewarding context-grounded, coherent actions and claims and flagging unsupported assertions or instruction failures.It asks evaluators to identify problematic steps and implicit assumptions.Source: §E.1 · pp. 28–29

E.1.1 LC Final Prompt (Post-GEPA).

• The final Logical Consistency prompt assigns a score from 0 to 3, with the highest score requiring traceable claims, acknowledged corrections, instruction-following, and coherent reasoning.Middle scores allow occasional lapses or unsupported assumptions; the lowest score reflects severe grounding or logic breakdowns.Source: §E.1.1 · p. 29
• The LC prompt also prioritizes exact format and tool-availability checks, including whether tools are actually invoked rather than merely described.It recommends listing explicit format requirements and checking exact strings and other specified constraints.Source: §E.1.1 · pp. 29–30
• The LC rubric checks whether claims trace to observations, data are grounded in actual sources, planned steps are executed, and requested verification is performed.It also flags fabricated data after tool failures and use of substituted sources without checking equivalence.Source: §E.1.1 · pp. 29–30

11. Cascading Error Analysis

• The cascading-error analysis traces how early failures affect downstream reasoning and final output, and requires identifying issue locations, missing evidence or actions, and consequences.It prioritizes format and instruction compliance, tool invocation failures, and unacknowledged fabrication, recommending a format scan before logical analysis.Source: §11 · p. 31

E.2 EE Starting Prompt (Pre-GEPA): Generic EE Criteria

• The starting EE rubric scores execution from 0 to 3, judging sequencing, resource use, redundant work, and whether verification adds distinct value.Its highest score describes relevant actions done once in a streamlined sequence; its lowest describes duplicated, poorly ordered, or wasteful work.Source: §E.2 · pp. 31–32

E.3 EE Final Prompt (Post-GEPA)

• The final EE rubric retains the 0–3 efficiency score and diagnoses preventable errors, redundant calls, sequencing, resource limits, verification quality, and infrastructure versus logic failures.It also distinguishes appropriate retries after external failures from repeated malformed calls, and assesses delegation and algorithm choice.Source: §E.3 · pp. 32–33

E.4 PA Starting Prompt (PRE-GEPA): Generic PA Criteria

• The starting PA rubric extracts initial plans and replans, then scores step-by-step adherence, treating omitted mandated tools or subtasks as failures regardless of output quality.If no plan is found, it instructs the evaluator to say so rather than infer one from execution.Source: §E.4 · pp. 33–34

E.5 PA Final Prompt (POST-GEPA)

• The final PA prompt identifies plans from labeled sections or planning text without filling gaps from execution, and distinguishes plans created by parent agents from sub-agents.It covers traces with execution steps, observations, replans, and multi-agent delegation.Source: §E.5 · p. 34
• PA evaluates adherence by tracing planned steps through actual tool calls, captured and used outputs, and evidence supporting verification claims.It distinguishes tool substitution from omission and checks multi-agent commitments and failure propagation; correct outcomes do not compensate for ignoring the plan.Source: §E.5 · pp. 35–37

E.7 PQ Final Prompt (POST-GEPA)

• The final PQ rubric assesses tool selection and capabilities, prior-context use, step necessity and completeness, and logical sequencing while separating strategy from outcome.Its stated evidence includes the written plan, theoretical executability with available tools, and soundness under the given constraints and resources.Source: §E.7 · pp. 38–39
• The final plan-quality prompt scores intrinsic plan quality, including explicit tool specification and logical sequencing, and evaluates whether replans respond appropriately to their triggers.It instructs evaluators to assess the plan rather than execution or final-answer correctness.Source: §E.7 · pp. 39–40

E.8 TS Starting Prompt (PRE-GEPA): Generic TS Criteria

• The generic tool-selection prompt scores whether chosen tools suit stated goals and available tools, considering fit, alternatives, instruction compliance, and tool constraints.It excludes call syntax, output interpretation, efficiency, and plan deviations from tool-selection scoring.Source: §E.8 · pp. 40–41

E.9 TS Final Prompt (POST-GEPA)

• The final tool-selection prompt evaluates tool-goal fit, comparative suitability, instruction compliance, and awareness of constraints, while separating selection errors from call execution and plan adherence.It also distinguishes tool preparation from actual invocation and directs evaluators to examine planning first, compare plans with execution, and categorize mismatches.Source: §E.9 · pp. 41–42

E.10 TC Starting Prompt (PRE-GEPA): Generic TC Criteria

• The generic tool-calling prompt introduces a single 0–3 score for evaluating tool inputs and output interpretation.The supplied excerpt provides the scoring range and evaluation scope but no further criteria.Source: §E.10 · p. 43
• The starting tool-calling criteria score agent-controlled input formation and output interpretation, covering valid arguments, semantic fit, faithful output use, and error handling.They exclude tool choice, workflow efficiency, and external tool reliability.Source: §E.10 · pp. 43–44

E.11 TC Final Prompt (POST-GEPA)

• The post-GEPA tool-calling rubric flags fabricated verification and ungrounded certainty, distinguishing skipped verification from falsely claiming it occurred.It also checks syntax and schema compliance, error recovery, and whether outputs support the agent’s claims.Source: §E.11 · pp. 44–45

F Automatic Consistency Optimization with Openevolve

• The appendix introduces Openevolve as a way to present each GPA metric’s prompts before and after optimization.Source: F Automatic Consistency Optimization… · p. 45

F.1 PQ Starting Prompt (Pre-OpenEvolve): Generic PQ Criteria

• The starting plan-quality prompt judges the initial plan using available tools, tool definitions, and prior knowledge, while evaluating replans against their triggers and excluding execution quality.It requires intrinsic plan quality judgments rather than inferring flaws from the agent’s actions or outcome.Source: §F.1 · pp. 45–46

F.2 PQ Final Prompt (Post-Openevolve)

• The post-OpenEvolve plan-quality prompt retains plan extraction rules and evaluates the initial plan solely on its intrinsic quality, assessing replans against preceding evidence and triggers.Source: §F.2 · p. 47
• The final PQ prompt scores plans solely on intrinsic plan and replan quality, assessing query coverage, feasibility, logical ordering, and replanning rationale.It uses score bands from 0.0–3.0, with protocol guidance to consider tool feasibility and default to the lower range when uncertain.Source: §F.2 · pp. 47–48

F.3 TC Starting Prompt (Pre-OpenEvolve): GEPA optimized TC prompt

• The GEPA-optimized starting TC prompt evaluates agent-controlled tool-input correctness and output interpretation, excluding tool choice, workflow efficiency, and external reliability.It directs evaluators to inspect unsupported verification claims, syntax or schema problems, and handling of tool errors.Source: §F.3 · p. 49

F.4 TC Final Prompt (Post-OpenEvolve)

• The post-OpenEvolve TC rubric assigns integer scores from 0 to 3 using issue thresholds, severity, error acknowledgment, interpretation accuracy, and self-correction.It distinguishes critical, major, and minor issues and counts self-corrected issues at half weight.Source: §F.4 · pp. 50–51

G Snowflake Intelligence Results

• Table 23 reports Snowflake Intelligence results across models using 2-point and 3-point accuracy, correlation, and normalized mean absolute error.Accuracy is reported as both binary error-versus-correct scoring and a 3-point scale.Source: G Snowflake Intelligence Results · p. 52, Table 23 · p. 53
• Claude-4-Sonnet achieves LC reliably, while Claude-3-7-Sonnet, gpt-4o, and gpt-4.1 reach similarly high execution-efficiency performance.The passage describes LC as requiring complex reasoning and execution-efficiency errors as potentially requiring less abstract thinking.Source: Table 23 · p. 53, G Snowflake Intelligence Results · p. 53

H.1 GEPA Configuration

• GEPA setup used DSPy ([16]) defaults and compared human-graded generic-plus-custom criteria with meta-judged generic, custom, and auto-light/medium prompts.The auto-light and auto-medium prompts used generic metric criteria as their seed.Source: §H.1 · p. 53

H.2 OpenEvolve Configuration

• OpenEvolve (), inspired by AlphaEvolve ([23]), evolves Python scoring-rubric and guidance code through mutation and fitness evaluation.It uses a MAP-Elites archive with island-based populations; the population has 30 programs across 2 islands, with an elite archive of 15.Source: §H.2 · p. 53
• OpenEvolve fitness rewards low variance and fidelity to the initial prompt’s score distribution across sampled TRAIL/SWE-bench traces.Each candidate is assessed on 15 traces with 5 evaluations per trace; the score combines a variance-based base score with a penalty for deviation from ground-truth mean scores.Source: §H.2 · p. 53
• A two-stage cascade screens candidates with 2 runs on one trace, advancing those above a combined_score threshold of 0.01 to full evaluation.Full evaluation uses 15 traces and 5 runs per trace.Source: §H.2 · p. 54
• OpenEvolve ran separately from generic and GEPA-optimized seeds, with 50 iterations for the generic-seed variant.The GEPA-seed variant tested further optimization of an already optimized prompt; its max_code_length increased from 15,000 to 25,000, while other settings stayed constant across six runs.Source: §H.2 · p. 54

I Extended Future Work Directions

• The authors propose using GPA rubric feedback to improve agent code through evolutionary systems such as AlphaEvolve ([23]) and OpenEvolve ().They also note that rubric feedback can support reflective prompt optimization methods GEPA ([1]) and MIPROv2 ([25]).Source: I Extended Future Work Directions · p. 54

J Errata

• The errata section is intended to report data-validation or formatting issues if applicable.The passage does not identify any specific issue.Source: J Errata · p. 54