PaperlayerView exact revision on arXiv

Source-linked AI summary

Bridging Foundational and Agentic Capabilities of LLMs

Wei Guo, Yuanjian Xu, Jianing Hao, Yongyao Duan, Ran Chen, Zhong Li, Guang Zhang

arXiv:2610.29751v1cs.CLcs.SE

TL;DR

End-to-end agent scores leave unclear how foundational capabilities translate into specific execution behaviors. The paper links benchmark-derived foundational profiles with trajectory-derived operation profiles and measures capability-realization gaps as departures from predicted operation performance. In Banking, foundational profiles only partly predict operations, while operation profiles add task-score information and gaps persist across task subsets.

  • Problem

    End-to-end scores do not reveal how foundational benchmark dimensions translate into planning, action selection, and recovery, leaving capability-to-execution links unresolved.

    Source: §1 · p. 1

  • Method

    The framework pairs benchmark-derived foundational profiles with trajectory-derived operation profiles and treats departures from predicted operation scores as capability-realization gaps.

    Source: §1 · pp. 1–2, §3.3 · p. 4

  • Results

    In Banking, foundational profiles explain only part of operation performance, with persistent model-level gaps across task subsets, and operation profiles add task-score information.

    Source: §1 · p. 1, §4.2 · p. 5, §5 · p. 8

  • Takeaways & Limitations

    Agent evaluation should measure both capabilities and their reliable deployment during interaction.

    Source: §5 · p. 8

    The study evaluates only Banking, so transfer of its operation definitions, realization gaps, and predictive gains to other domains and task structures remains to be validated.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Large language models (LLMs) are increasingly deployed as agents, yet it remains unclear how their foundational capabilities translate into agentic capabilities. End-to-end scores leave unresolved whether low benchmark capability or weak deployment drives a failure, limiting actionable diagnosis. We bridge foundational and agentic capabilities through an operation-level measurement framework. We represent each model with a foundational profile derived from capability benchmarks and an operation profile derived from interaction trajectories, and define capability realization gaps as deviations of observed operation performance from predictions based on foundational capabilities. Experiments on fifteen model services in $\tau^2$-Bench show that foundational profiles only partially predict agentic operations. Models with comparable profiles can differ substantially in execution, and these gaps remain consistent across task subsets. Operation profiles add information and substantially improve task-score prediction. The framework locates execution departures at the operation level to inform model development and agent evaluation. We will release our code and analysis artifacts.

1 Introduction

End-to-end agent scores do not show how foundational capabilities translate into specific execution behaviors. This study links benchmark-derived capability profiles to trajectory-derived operation profiles and tests whether operations add information about task outcomes.

1 Introduction

  • End-to-end scores leave unclear whether weak task performance reflects low foundational capability or execution below profile-based expectations.The paper argues that linking capabilities to operations can locate execution-stage departures.Source: §1 · p. 1
  • The framework pairs each model’s foundational and operation profiles to test how well the former predict specific behaviors and whether the latter add task-outcome information.Provider-held-out prediction and residual analysis address the two research questions.Source: §1 · p. 1
  • In Banking tasks from τ 2-Bench, foundational profiles explain only part of operation-score variation, while operation profiles improve held-out task-performance prediction.The study contributes a provider-held-out mapping and residual analysis of execution departures, alongside evidence of improved held-out Banking reward prediction.Source: §1 · pp. 1–2

2 Related Work

Prior work evaluates aggregate capability-to-outcome prediction and execution processes separately. This study pairs foundational profiles with trajectory-level operations for the same models and task conditions.

2 Related Work

  • End-to-end agent benchmarks primarily report task-level success, while capability-oriented approaches map abilities to aggregate outcomes.Examples include MMAU, ADeLe, and PACE; the passage contrasts these with process-level evaluations.Source: §2 · p. 2
  • Process-level evaluations score execution components or competencies directly from trajectories, but the passage describes them separately from independently measured foundational profiles.T-Eval uses reference trajectories; AgentDiagnose scores competencies such as task decomposition and self-verification.Source: §2 · p. 2
  • This study connects capability and process evaluation by pairing foundational profiles, trajectory-level operations, and task outcomes under shared task conditions.It tests capability-to-operation prediction and the incremental outcome information in operation profiles.Source: §2 · p. 2

3 Framework

The framework connects resources measured outside agent environments to their deployment during interaction and resulting task outcomes. It measures how foundational profiles relate to operation profiles and whether operations add information about outcomes.

3 Framework

  • The framework links pre-interaction resources, trajectory-level deployment, and task outcomes under a fixed inference protocol, common interface, and task conditions.The task fixes the horizon, observability, action space, and feedback.Source: §3 · p. 2

3.1 Foundational Capability Profile

  • The foundational profile has seven separately reported dimensions: task understanding, information selection, knowledge use, context integration, reasoning, constrained expression, and self-monitoring.Self-monitoring provides cross-stage feedback and may trigger revision of integration, reasoning, or expression.Source: §3.1 · pp. 2–3

3.2 Agentic Operation Profile

  • The operation profile tracks seven trajectory behaviors—goal interpretation, state tracking, planning, action selection, progress monitoring, recovery, and termination—in a recurring control loop.State tracking and progress monitoring recur after informative transitions; recovery returns control to planning or action selection.Source: §3.2 · p. 3
  • Applicable operation requirements receive 0, 0.5, or 1; unobservable opportunities are excluded, and scores are averaged by operation across eligible trajectories.Recovery is scored only when an error and an opportunity to respond are present.Source: §3.2 · p. 3

3.3 Analysis Methodology

  • The analysis uses descriptive 7 × 7 Pearson associations with Spearman checks, then family-held-out prediction and residuals to assess departures from expected execution.Positive residuals indicate execution above the foundational-profile prediction; negative residuals indicate execution below it.Source: §3.3 · pp. 3–4
  • Outcome prediction compares a training-mean baseline with foundational-only, operation-only, and joint profiles; improvement from the joint model measures added operation-profile information.The analysis predicts mean Banking reward within the corpus using shared family splits and fixed estimator and preprocessing.Source: §3.3 · p. 4
  • Shared-initial-state continuations compare natural behavior with a checking prompt, while counterfactual runs alter operation-relevant content with task state, tools, goal, and budget fixed.Damage and repair interventions are paired with semantics-preserving rewrites as controls for surface-form changes.Source: §3.3 · p. 4

4 Experiments

The experiments connect benchmark-derived foundational profiles to operation-level behavior in Banking trajectories, then test whether those profiles predict task outcomes and whether execution gaps persist across tasks and continuations.

4.1 Setup

  • The study evaluates model services spanning eight families on Banking trajectories and predicts outcomes with complete-family-held-out ridge models.Operation scores use complete trajectories and include only those with a scoring opportunity; profiles combine public evaluation results with local measurements.Source: §4.1 · pp. 4–5

4.2 Main Results

  • Foundational and operation profiles rank models differently, separating benchmark-measured capabilities from performance during task execution.The operation scores distinguish strengths in Goal Interpretation and State Tracking as well as differences in Action Selection, Recovery, and Termination.Source: §4.2 · p. 5, Table 4 · p. 6
  • Foundational profiles have positive family-held-out Q2 for all seven operation categories, while capability realization gaps reveal model-level and operation-specific departures.Claude services tend to exceed predicted operation scores, Gemini 3.1 Pro falls below them, and other services show mixed profiles.Source: §4.2 · p. 5, Figure 2 · p. 6

4.3 Further Analysis

  • Capability realization gap rankings remain stable across independent task-group splits, although Recovery varies more with task composition.Each subset independently re-estimates expected operation performance; Recovery depends on an observable failure and an opportunity to respond.Source: §4.3 · p. 7
  • Most local content interventions do not change mean scores; a planning-text repair leaves the submitted call and final score unchanged.A tool-identifier repair succeeds in one repetition but does not reproduce, and later tool results and decisions mediate effects.Source: §4.3 · pp. 7–8

4.2 Main Results

  • The joint foundational-and-operation profile improves held-out Banking reward prediction over foundational features alone, with the lowest error and strongest rank agreement.The comparison uses model-family-held-out splits; the joint gain comes from adding observed deployment signals.Source: §4.2 · p. 6, Table 6 · p. 7, Figure 3 · p. 8

4.3 Further Analysis

  • Constraint-execution differences recur when models continue from shared initial states, and later steps can reverse initially compliant actions.The ordering across Gemini 3.1 Pro, GLM-5.2, and Claude Fable 5 appears under both natural continuation and a generic checking prompt.Source: §4.3 · p. 8

5 Conclusion

The conclusion presents capability realization gaps as a link between foundational capabilities, agentic operations, and task outcomes. In Banking, operation profiles add task-score information, while deployment gaps persist across task subsets and execution differences recur from shared initial states.

5 Conclusion

  • The framework links foundational capabilities, agentic operations, and task outcomes through capability realization gaps.In Banking, foundational profiles explain part of operation performance; operation profiles add task-score information, and gaps persist across task subsets.Source: §5 · p. 8
  • Shared-initial-state experiments reveal repeatable execution differences, while local repairs produce narrower, case-dependent effects.The conclusion recommends measuring both capabilities and their reliable deployment during interaction.Source: §5 · p. 8

Limitations

The study evaluates its framework only in Banking and calls for validation in additional domains and agent environments.

Limitations

  • The evaluation covers only the Banking domain.The authors identify validation in additional domains and agent environments as future work.Source: Limitations · p. 8
  • Whether the operation definitions, capability realization gaps, and predictive gains transfer across task structures remains to be tested.Source: Limitations · p. 8

A Benchmark Details

The appendix describes the Banking trajectory corpus, annotation and evaluation procedures, and protocols for prediction, cross-task checks, and local content interventions.

A.1 Agentic Benchmark Snapshot

  • The corpus contains 1,451 Banking trajectories from fifteen model services, with a common-task subset of 1,395 trajectories.Each record includes messages, tool calls and results, and native task reward.Source: §A.1 · p. 11

A.2 Trajectory Annotation Protocol

  • A two-pass annotation process builds task contracts and evaluates complete trajectories with model identity and task reward hidden.Missing required actions receive zero, while simulator deviations are recorded separately; twelve trajectories received AI-assisted adjudication outside independent human double annotation.Source: §A.2 · p. 11

B.1 Model and Inference Configurations

  • Foundational scores combine public results and local measurements, while public scores retain source-specific settings and evaluation cohorts.Source: §B.1 · p. 11

B.2 Foundational Evaluation Configuration

  • Self-Monitoring elicits an answer followed by a confidence report without correctness feedback.Some local Self-Monitoring results use 148 GPQA items, and valid-response estimates retain their smaller denominators.Source: §B.2 · p. 11

B.3 Agent Harness Configuration

  • Historical Banking runs use saved task files, tool interfaces, policies, and native reward evaluation, with restored checkpoints for experiments.Local content interventions preserve the source configuration and remaining budget; shared-initial-state experiments retain native reasoning settings and add no user messages or confirmations.Source: §B.3 · p. 11

B.4 Prediction and Uncertainty Protocol

  • Ridge predictors use family-held-out folds, training-fold standardization, equal model weights, and a common α = 10 reference.Uncertainty intervals resample the eight model families while keeping fitted predictions fixed.Source: §B.4 · p. 11

B.5 Cross-Task Stability Checks

  • Cross-task checks split ninety-three common tasks into disjoint scenario-group sets and fit separate model-family-held-out predictors.Twenty additional fixed group splits assess partition sensitivity.Source: §B.5 · pp. 11–12

B.6 Local Content Intervention Protocol

  • Local content interventions compare original, edited, and control branches from restored checkpoints across frozen cases and repeated trials.The main cohort has 38 cases from thirteen tasks and three models; stratum estimates are descriptive because each has fewer than five contributing tasks.Source: §B.6 · p. 12

B.7 Shared-Initial-State Experiment Protocol

This material specifies shared-state experiments, prompt and annotation procedures, and additional prediction and association analyses.

B.7 Shared-Initial-State Experiment Protocol

  • Gemini 3.1 Pro, GLM-5.2, and Claude Fable 5 continued from historical action-trigger checkpoints, spanning eighteen distinct task IDs and 195 continuations, of which 193 were scoreable.Source: §B.7 · p. 12

INSTRUCTION

  • The checking prompt asks the model to identify its authorized goal and requirements, then take the smallest lawful next step without claiming unsupported completion.It adds no task-specific answer or authorization, and local edits preserve the rest of the restored prompt.Source: INSTRUCTION · p. 12, PROTOCOL NOTES · p. 12, C Prompts · p. 12

C.1 Self-Monitoring Prompts

  • Self-Monitoring uses two stages: answer first, then estimate the probability that the original answer is fully correct without solving again.Stage 2 receives no reference answer or correctness feedback.Source: §C.1 · pp. 12–13

C.2 Automatic Annotation Prompts

  • Automatic annotation first builds a task contract, then judges a blinded trajectory against fixed contract items, stage checks, and outcome anchors.Local code derives dimension scores from the returned labels, and turn indices identify evidence and gaps.Source: §C.2 · p. 13

D.1 Common-Penalty Reference and Sensitivity

  • Across reported ridge settings, the joint profile has lower MAE than foundational features; Action Selection and Planning are most predictable, while Recovery and Termination are harder.The joint fit remains close to the operation-only fit across nearby penalties.Source: Table 8 · p. 13, §D.1 · p. 13

D.2 Nested-Selection Protocol

  • Nested selection chooses α from six candidate penalties using inner family holdouts, with standardization and selection restricted to each outer training fold.The main results use the common α = 10 reference.Source: §D.2 · p. 13

D.3 CE–Recovery Association and Sensitivity

  • Constrained Expression is the only foundational dimension negatively associated with all seven operations, with its strongest association to Recovery (Pearson r = −0.751; Spearman ρ = −0.736).The association remains negative after removing Anthropic models (r = −0.42) and after centering scores within each family (r = −0.44).Source: §D.3 · pp. 13–14

D.4 Cross-Task Split Sensitivity

The appendix assesses whether capability realization gaps persist across task splits and reports model-dependent effects from local content interventions.

D.4 Cross-Task Split Sensitivity

  • Gap stability is assessed across a primary pair of disjoint task subsets and twenty additional task-group splits.The primary comparison uses Spearman correlations with intervals from scenario-group resampling; the additional splits report median and range correlations.Source: Table 11 · p. 14, Table 12 · p. 14

D.5 Local Content Intervention Results

  • Local content intervention effects remain small and model dependent: MiniMax M3 has positive contrasts, DeepSeek V4 Flash has small negative contrasts, and only one tool-identifier repetition changes reward.Source: §D.5 · p. 15

D.6 Shared-Initial-State Experiment Results

From shared initial states, Gemini 3.1 Pro repeatedly violates constraints at both checkpoints, while GLM-5.2 and Claude Fable 5 avoid the reported violations. The checkpoint results show that a compliant first action does not ensure later actions remain compliant.

D.6 Shared-Initial-State Experiment Results

  • Gemini 3.1 Pro blocks the stolen card without required verification in all three repetitions under both continuation conditions.GLM-5.2 and Claude Fable 5 avoid the violation and leave the protective action incomplete in the first step.Source: §D.6 · p. 15
  • Gemini 3.1 Pro increases an ineligible account’s ATM limit after increasing the eligible account’s limit, and this recurs in added repetitions.GLM-5.2 and Claude Fable 5 avoid the corresponding invalid increase; the later action undermines a correct first step.Source: §D.6 · p. 15
  • Table 15 counts each task checkpoint once per model; “repeated” requires violations in at least two repetitions within a round and prompt condition.Source: Table 15 · p. 15

E Data Statement and License

The data statement describes the benchmark-derived profiles, Banking interaction records, and licensing of the software and third-party data. The notation section summarizes the framework’s mathematical objects and clarifies that vectors are column vectors unless stated otherwise.

E Data Statement and License

  • The supplementary archives provide model profiles, held-out predictions, sensitivity results, and prompt templates.Source: E Data Statement and License · p. 15
  • The τ 2-Bench software uses the MIT license, while third-party benchmark data retain their original licenses and access terms.Source: E Data Statement and License · p. 15
  • Figure 5 contrasts observed operation scores with foundational-profile predictions and with a separate overall-operation-score predictor; exact values are in Table 10.Blue marks execution above prediction and red below prediction.Source: Figure 5 · p. 16
  • Figure 6 presents Constrained Expression and Recovery scores for 15 models and correlations under six sensitivity specifications.Parentheses give the number of included models.Source: Figure 6 · p. 16

F Notation

  • Table 16 summarizes mathematical objects used in the framework and experiments, and vectors are column vectors unless stated otherwise.Its type categories include indices, sets, mappings, vectors, scalars, parameters, metrics, experimental branches, and conditions.Source: F Notation · p. 17, Table 16 · p. 17

Related papers

Loading 2610.29751v1…