Source-linked AI summary
Beyond Leaderboards: Tokenomics of Agentic Small Language Model Ensembles
Alexei N. Skurikhin, Emily M. Taylor, Nathan A. DeBardeleben
TL;DR
As LLMs move into agentic workflows, leaderboard accuracy alone does not capture reliability, cost, latency, and token efficiency. This paper evaluates an SLM ensemble with programmatic checking and judge-mediated feedback on IFEval; its best configuration reaches 97.34% strict prompt accuracy, 5.81 percentage points above gpt-5.4, while the analysis also examines operational tradeoffs.
Problem
Agentic workflows require reliable satisfaction of explicit constraints, but leaderboard accuracy alone does not capture deployment-relevant factors such as feedback, orchestration, latency, and token cost.
Source: §1 · p. 1Method
The paper evaluates an agentic SLM ensemble using programmatic IFEval checking and an SLM judge-mediated feedback loop, alongside tokenomic and operational measures.
Source: §1 · p. 1Results
Takeaways & Limitations
The results support evaluating agentic systems jointly across accuracy, cost, token goodput, latency, and robustness.
Source: §4 · p. 2
Abstract
from arXiv · showhide
As large language models (LLMs) move from standalone assistants into agentic workflows, evaluation must extend beyond scalar leaderboard accuracy to account for operational reliability, cost, latency, and token efficiency. We use an agentic ensemble of small language models (SLMs) with an SLM-judge-mediated feedback loop as a case study for such beyond-leaderboard evaluation. On the 541-prompt IFEval benchmark, the best ensemble achieves 97.34% strict prompt accuracy, exceeding the strongest standalone LLM baseline, gpt-5.4, by 5.81 percentage points while operating in a lower-cost regime. We then analyze the tokenomics and operational behavior behind this gain, including cost per sample, token composition, useful-output goodput, feedback-loop recovery, latency decomposition, and performance across instruction categories and constraint counts. Our results show that agentic SLM ensembles can trade additional test-time tokens and orchestration overhead for improved instruction-following fidelity, motivating multi-dimensional evaluation protocols for future agentic AI systems.
1 Introduction
As LLMs enter agentic workflows, evaluation must capture reliable constraint satisfaction alongside deployment costs and operational behavior. This study uses an agentic SLM ensemble on IFEval to examine instruction-following accuracy and the token and orchestration tradeoffs behind it.
1 Introduction
- Agentic workflows require systems to satisfy explicit constraints, schemas, and protocols reliably, making aggregate leaderboard accuracy an incomplete measure of deployability.Iterative feedback, orchestration, latency, and token cost can all be part of inference.Source: §1 · p. 1
- The study evaluates an MCP-based SLM ensemble with programmatic IFEval checking and an SLM-judge-mediated feedback loop.It examines cost, token composition, goodput, recovery, latency, category performance, and constraint-count robustness.Source: §1 · p. 1
- In these experiments, SLM ensembles can be competitive with and outperform standalone LLM counterparts on IFEval instruction-following accuracy, while the study evaluates operational behavior beyond leaderboard accuracy.Source: §1 · p. 1
2 Ensemble System Design and Evaluation
The system coordinates SLM generators, a judge, and a programmatic checker to generate, verify, and iteratively correct instruction-following outputs. Its evaluation combines standard IFEval accuracy with operational and token-efficiency measures.
2 Ensemble System Design and Evaluation
- MCP structures communication among SLM generators, an SLM judge, and the programmatic IFEval checker for reproducible multi-model execution and iterative correction.The judge breaks ties among compliant outputs and turns checker-identified violations into corrective feedback when none pass; the ensemble allows up to two corrective attempts.Source: §2 · p. 1
- The design separates generation, constraint verification, and feedback-guided correction, and treats maximal constraint satisfaction as a Best-of-N selection criterion for IFEval.Five ensemble configurations are benchmarked against standalone large and small models.Source: §2 · p. 1
2.1 Beyond-Leaderboard Evaluation
- IFEval contains 541 prompts with rule-based instruction checks.The benchmark checks formatting, length, inclusion or exclusion rules, and multi-instruction composition.Source: §2.1 · pp. 1–2
- IFEval evaluation includes strict prompt-, strict instruction-, and loose prompt-level accuracy, alongside operational and tokenomic measures.The additional measures cover cost per sample, token allocation and goodput, iteration progress, latency components, and accuracy by category and constraint count.Source: §2.1 · p. 2
3 Experimental Results
Across five runs per configuration, the ensembles outperform standalone baselines on strict prompt accuracy, with the best configuration reaching 97.34%. Beyond accuracy, the results examine cost, token allocation, feedback-loop recovery, and useful-output goodput.
3 Experimental Results
- Each ensemble configuration is run five times, with mean accuracy and standard deviation reported.The study then analyzes cost, token allocation, goodput, iteration dynamics, latency, and robustness.Source: §3 · p. 2
- 97.34% strict prompt accuracy is achieved by [Ens-1], 5.81 percentage points above the strongest standalone baseline, gpt-5.4, at 91.53%.All five ensemble configurations outperform the standalone baselines on strict prompt accuracy; [Ens-5] reaches 95.64%.Source: §3 · p. 2
- [Ens-1] reaches 98.3% instruction-level accuracy and 98.4% loose prompt-level accuracy, extending its gains beyond strict prompt accuracy.These complementary IFEval metrics are reported in Figure 2.Source: §3 · p. 2
- The ensembles exceed the strongest standalone LLM's accuracy while costing less per prompt than several large-model baselines.The reported token allocation shows orchestration at approximately 45–47% versus exploration at approximately 25–31%; the paper attributes the lower cost despite orchestration overhead to compute-efficient mini-model inference.Source: §3 · p. 2
- Cumulative strict accuracy rises over successive attempts, while the token-goodput comparison places ensembles above 95% accuracy and standalone models below 92%.The paper reports ensembles in the top-left accuracy/token-ratio cluster and standalone models mostly at 0.70–0.90 token ratio; the feedback loop contributes to recovery of initially invalid outputs.Source: §3 · p. 2
4 Conclusions and Future Work
The paper presents an IFEval case study in which an agentic SLM ensemble improves instruction-following accuracy while incurring token and orchestration tradeoffs. It concludes that accuracy, cost, token goodput, latency, and robustness should be evaluated jointly.
4 Conclusions and Future Work
- The best ensemble reaches 97.34% strict prompt accuracy, 5.81 percentage points above the strongest standalone LLM baseline.The authors describe the improvement as coupled to orchestration and judging overhead, iterative recovery, and latency costs.Source: §4 · p. 2
- The results support evaluating agentic systems jointly across accuracy, cost, token goodput, latency, and robustness.The conclusion is scoped to the paper's IFEval evaluation of agentic SLM ensembles.Source: §4 · p. 2
- Future work will examine richer consensus mechanisms, larger model pools, and science-oriented agentic workflows with domain-specific validity checks.Source: §4 · p. 2
A.1 Model and Ensemble Configurations
Table 1 presents the evaluated model and ensemble configurations.
A.1 Model and Ensemble Configurations
- Table 1 shows the evaluated model and ensemble configurations.Source: §A.1 · p. 5
A.2 Category-Level and Constraint-Complexity Analysis
Ensemble gains vary across instruction categories, with the largest benefits in categories where feedback can correct identifiable violations. Ensembles also retain higher strict accuracy as prompts accumulate constraints.
A.2 Category-Level and Constraint-Complexity Analysis
- The largest ensemble benefits occur in formatting, content-detection, and multi-condition categories, where failures are often recoverable through feedback.The authors interpret this pattern as correction of identifiable constraint violations rather than a rise in generic model capability.Source: §A.2 · p. 5
- As simultaneous constraints increase, standalone accuracy falls more sharply, while agentic ensembles retain higher strict accuracy.The authors suggest feedback is especially useful when responses partially satisfy a prompt but miss atomic constraints.Source: §A.2 · p. 5
A.3 Latency Decomposition
Under sequential execution, leading SLM ensembles improve strict instruction-following accuracy while requiring less active processing time than gpt-5.4, despite judge and feedback overhead.
A.3 Latency Decomposition
- Even with sequential execution, leading SLM ensembles achieve lower active processing time than gpt-5.4 while improving strict instruction-following accuracy.The estimate is conservative because ensemble members and feedback cycles are not run concurrently; the implementation supports higher concurrency.Source: §A.3 · p. 5
A.4 The Non-ASCII Roulette Anomaly
IFEval preprocessing can randomly alter certain characters in instruction constraints, making evaluation stochastic for affected character-based prompts. The authors retained the original parameters for comparability, but caution that small metric differences may be affected.
A.4 The Non-ASCII Roulette Anomaly
- IFEval preprocessing may randomly replace selected non-ASCII or other out-of-range characters in instruction constraints, not model outputs.The checker remains rule-based, but preprocessing changes what is checked for affected character-based instructions.Source: §A.4 · p. 5
- Repeated evaluations of the same model output may produce slightly different scores when the instruction contains affected characters.Source: §A.4 · p. 5
- Original evaluation parameters were retained for leaderboard comparability; interpret small instruction- and prompt-level metric differences cautiously, particularly for affected characters.Source: §A.4 · pp. 5–6