Source-linked AI summary
BekchiAI: Measuring, Observing, and Controlling LLM Agents in One Click
Mesut Toruk
TL;DR
LLM agents’ tool use, planning, security judgment, and grounding are difficult to measure with accuracy-only leaderboards. BekchiAI introduces a verifier-based benchmark and an observability-and-control platform; across four models, its results show that per-family skill spread is more informative than aggregate ranking.
Problem
Accuracy-only agent leaderboards fail to isolate tool sequencing, dependency-aware planning, security judgment, and argument grounding, while some tasks leak gold through oracle tools.
Method
BekchiAI combines 13 verifier-checkable ReAct task families across 7 categories and 2,057 committed tasks with adversarial cases, imperfect security scanners, behavioral metrics, and a live observability platform.
Results
Per-family spread is more informative than aggregate ranking: gpt-oss-120b leads grounding but scores 0.24 on sql_analytics versus 0.87 for the leading model.
Takeaways & Limitations
Skill- and cost-resolved evaluation can reveal capability profiles and efficiency trade-offs that a single leaderboard number obscures.
Takeaways & Limitations
The benchmark evaluates one task per rollout; extensions are needed for longer-horizon, multi-turn interactions, broader coverage, and additional adversarial categories.
Abstract
from arXiv · showhide
Large language model agents reason, call tools, and act autonomously over many steps, but their agentic skills-correctly sequencing tools, planning under dependencies, judging untrusted inputs, and grounding generated arguments-are hard to measure with accuracy-only leaderboards. We present BekchiAI, which addresses both sides: a benchmark for measuring agentic skill and a platform for observing and controlling live agents. The BekchiAI-Benchmark, a suite of 13 tool-using ReAct agents across 7 task categories (arithmetic, structured/SQL, security detection, URL grounding, planning, orchestration, and tool-policy), totalling 2,057 deterministic, committed test tasks. Every task is verifier-checkable gold answers are computed by running canonical SQL against a real database, computing the exact schedule of a directed acyclic graph (DAG), or evaluating closed-form lambdas including adversarial security samples paired with deliberately imperfect signature scanners so a score reflects the model's own judgment, not the copying of an oracle. We define a small set of behavioral metrics beyond accuracy-tool-call adherence, URL hallucination and source-match, and per-model token cost and report a four-model comparison (Qwen3.7-Max, gemma-4-31B-it, gemma4:26b, gpt-oss-120b) whose story is in the per-family spread, not the aggregate. The benchmark runs are executed using the provided evaluation scripts. BekchiAI-Platform is a complementary web-based observability and control layer for deployed agents, providing full token and latency telemetry as well as remote run termination. The benchmark, evaluation tools, and platform are publicly released.
1 Introduction
BekchiAI addresses the limits of accuracy-only evaluation by benchmarking distinct agentic skills and pairing capability measurement with live observability and control. Its benchmark and platform support hard-to-game, cost-aware evaluation of tool-using agents.
- Motivation: Accuracy-only leaderboards fail to isolate tool sequencing, dependency-aware planning, security judgment, and argument grounding.Popular tasks may also be gamed by pattern matching or oracle leakage.
- Benchmark: BekchiAI-Benchmark contains 13 verifier-checkable ReAct task families across 7 categories and 2,057 committed tasks.The suite includes hand-authored adversarial samples and deliberately imperfect security tools.
- Evaluation: The released tool library and behavioral metrics extend evaluation beyond accuracy to tool-call adherence, URL hallucination, source-match, and token cost.These metrics are released with the benchmark.
- Platform: BekchiAI-Platform provides a drop-in observability and control layer with live telemetry and remote actions for deployed agents.It supports stopping, pausing, resuming, or revoking a running agent.
- Evaluation perspective: Capability and cost are measured from one instrumented stream, while model rankings vary substantially across task families.A model leading overall may collapse on structured or multi-step tasks while another leads on grounding.
2 Related Work
Prior agent benchmarks cover heterogeneous, domain-specific, and security-focused tasks, but commonly compress performance into end-to-end success rates. BekchiAI also targets an observability gap concerning agentic structure, provenance, and remote control.
- Agent and tool-use benchmarks: General and domain benchmarks evaluate multi-step reasoning, tool use, software engineering, web navigation, and API calling.Examples include AgentBench, GAIA, SWE-bench, Mind2Web, WebArena, ToolLLM, and τ-bench.
- Agent and tool-use benchmarks: Most existing suites report one end-to-end success rate per task, conflating distinct skills and remaining sensitive to prompt or task artifacts.This motivates more skill-resolved evaluation.
- Agent security: Security benchmarks test hijacking from malicious instructions, whereas BekchiAI tests decisions on untrusted artifacts using imperfect detectors.Its scanners agree with truth only 40–55% of the time, making the task distinct from oracle copying.
- Agent observability: Generic tracing captures service calls but not agentic sessions, tool trees, token usage, argument provenance, or remote control.BekchiAI-Platform is designed to address this observability gap.
- Benchmark organization: Table 1 organizes the benchmark’s 13 agents by category, test-set size, tools, and isolated skill.The table’s purpose is to expose the suite’s task-family structure.
3 The BekchiAI-Benchmark
The benchmark uses reproducible ReAct rollouts, verifier-computed gold answers, and adversarially designed tasks to test agentic competence without relying on oracle behavior. Its security families specifically require judgment beyond an imperfect scanner.
- Structure: Each agent is a strict-JSON ReAct loop, and 2,057 held-out tasks are committed as deterministic JSONL test sets.The harness executes tool calls until a final answer or step-budget limit, making runs reproducible and inspectable.
- Verifier-checked gold: Verifier gold is computed from canonical SQL, exact DAG schedules, or closed-form arithmetic functions in the same world seen by the agent.Tolerances absorb formatting noise without allowing gold drift.
- Adversarial samples: Planning and orchestration tasks use asymmetric joins, long idle dependencies, and dead-end distractors that require transitive-reachability pruning.These cases are designed to stress dependency-aware planning.
- Adversarial samples: URL-grounding tasks pair questions with distinct real hosts and include homoglyph impostor links that agents must reject.The ug_imposter example uses examp1e.com instead of the authoritative host.
- Imperfect-by-design security tools: Security scanners are deliberately imperfect signature heuristics, agreeing with curated labels only 40–55% of the time.Because they produce false positives and false negatives, success requires reading the artifact and deciding whether to block or proceed.
4 Benchmark Tools
BekchiAI uses a shared tool library so agents operate over consistent, independently testable tools and deterministic ground truth.
- Agents import tools from a shared library rather than defining their own, keeping tool behavior consistent and independently testable.
- Data-backed tools use deterministic state: run_sql queries an in-memory SQLite database, while fetch serves an offline index of canonical web pages.
5 Metrics
The benchmark supplements accuracy with behavioral measures for URL grounding, security decisions, and operational cost, all captured during rollout.
- Accuracy is the verifier-matched success rate, reported per family and category with Wilson 95% score intervals.
- URL grounding measures fabricated arguments through hallucination rate and checks whether cited hosts match the gold source host.
- Security metrics track proceeding on unsafe artifacts and, where applicable, blocking benign ones through violation and over-refusal rates.
- Operational cost records LLM calls and input/output token counts per model on the same runs.
- All reported metrics are reproducible through the released repository and evaluation harness.
6 Results: Four Models on the Benchmark
Across four models and 2,057 held-out tasks, performance varies substantially by family: aggregate rankings mask reversals between grounding and structured or multi-step tool tasks.
- All four models were evaluated on the same 2,057 held-out tasks with task conventions specified in the prompts.
- Overall accuracy ranks Qwen3.7-Max (0.88) > gemma-4-31B (0.86) > gemma4:26b (0.82) > gpt-oss-120b (0.71), but no model dominates everywhere.
- gpt-oss-120b leads both grounding families—ug_fetch (0.78) and ug_imposter (0.89)—yet performs worst on sql_analytics (0.24), tool_error_recovery (0.25), and multi_tool_planning (0.44).
- The widest family spreads occur in sql_analytics (0.24–0.87), tool_error_recovery (0.25–0.76), and multi_tool_planning (0.44–0.96).
- gemma-4-31B leads orchestrator (0.95), while gemma4:26b remains competitive overall (0.82) and leads ar_math (0.91).
7 The BekchiAI-Platform
BekchiAI-Platform is a drop-in observability and control layer that links agent telemetry, behavioral analytics, policy enforcement, and remote intervention.
- Platform role: The platform leaves task logic unchanged while streaming telemetry, enabling capability and cost measurement from the same run.
- Architecture: Its architecture combines an SDK, ingestion and API services, an event store, and a React dashboard for sessions, traces, and findings.
- Behavioral analytics: Argument grounding labels accessed hosts as grounded, ungrounded, or imposter_url using trusted hosts and a Levenshtein-distance threshold of 2.
- Behavioral analytics: Plan-vs-actual analysis compares intended tools with executed tool calls, producing intersection, unplanned, and missing sets plus a session verdict.
- Benchmark observability: Figure 1 plots category-level success rates on seven axes, revealing model profiles that aggregate scores conceal.
- Operational cost: Table 5 measures calls and token usage across the same 2,057-task runs, showing capability and cost in one stream.
- Control: Operators can define allow, block, and alert policies and issue stop, pause, resume, or revoke commands remotely.
8 Conclusion
BekchiAI combines a verifier-checked benchmark for agentic skills with behavioral, cost, and platform telemetry. Across four models, per-family differences reveal contrasts that aggregate scores obscure, while the compact single-task design leaves clear extension paths.
- BekchiAI-Benchmark: BekchiAI measures tool selection, dependency planning, security judgment, and argument grounding across 13 ReAct agents and 2,057 committed tasks.The suite spans seven categories and uses verifier-computed gold answers, hand-authored discriminating cases, and deliberately imperfect security scanners.
- Benchmark design: Verifier-computed gold, hand-authored adversarial cases, and imperfect security scanners reduce opportunities for score inflation through oracle copying or templated patterns.Gold answers use canonical SQL, exact DAG schedules, or closed-form lambdas.
- Platform integration: Behavioral metrics and token cost are collected from one instrumented stream through the BekchiAI-Platform.The released materials include the benchmark, tool library, harness, metric scripts, and platform.
- Results: Across four models, per-family spread is more informative than aggregate ranking: gpt-oss-120b leads grounding but scores 0.24 versus 0.87 on sql_analytics.The tool and structured families separate models by more than sixty points, and the most accurate model is not the cheapest.
- Limitations and future work: The benchmark is compact and evaluates one task per rollout, motivating broader models, more adversarial categories, longer horizons, and multi-turn deployment studies.The authors identify these extensions while noting that the benchmark and supporting tools are released.