Source-linked AI summary
AuditKIT: A Unified Evaluation Spine for Benchmarking, Judging, and Comparing Large Language Models and Agentic Systems
Shreeyans Arora, Utsav Avaiya, Zera Lyngkhoi, Hem Gosalia, Ram Mohan Rao Kadiyala, Vinay Kumar Sankarapu, Pratinav Seth
TL;DR
Language-model and agent evaluation combines metrics and evidence that existing tools represent differently, complicating comparison. AuditKIT records benchmark, judge, runtime, tool-use, and retrieval measurements in a shared evaluation framework with direction-aware metrics and configuration tracking. Seven case studies illustrate how evaluation choices change reported results.
Problem
Language-model and agent evaluation spans different metrics and evidence types, while tools use differing interfaces, metric definitions, and result formats.
Source: §1 · pp. 1–2Method
Results
Across seven case studies, answer extraction changed apparent model rankings, serving-runtime differences changed exact-match outcomes, and retrieval metrics separated retrievers that hit rate treated as tied.
Source: §10 · p. 16Takeaways & Limitations
Evaluation context should be retained, and each metric read together with the evidence used to compute it.
Source: §10 · p. 16The case studies are small and use one judge configuration; judged verdicts were not validated against human annotation and may favor the judge’s model family.
Source: §9 · p. 16
Abstract
from arXiv · showhide
Evaluating language models and agents requires combining benchmark accuracy, judge-based assessment, runtime performance, tool-use behavior, and retrieval quality. Differences in interfaces, metric semantics, and trace formats complicate these comparisons and can hide failures in agentic workflows. We present AuditKIT, a source-available evaluation library that records these measurements in one result representation, with direction-aware metric contracts, configuration fingerprints, and extensible metrics and backends. For agents, it provides turn-aware tool-call matching, separates parallel from sequential execution, and distinguishes verified task outcomes from trajectory-level diagnostics. Missing or unobservable evidence is marked unscored rather than counted as failure. For retrieval-augmented generation, it combines ranked retrieval metrics, judged grounding, and deterministic evidence checks. Seven case studies spanning model comparison, serving runtimes, retrieval, tool calling, and interoperability show how answer extraction, runtime behavior, and metric selection change evaluation outcomes.
1 Introduction
Language-model and agent evaluation spans multiple metrics and evidence types, while existing tools use differing interfaces, metric definitions, and result formats. AuditKIT brings these evaluations into one library and preserves distinctions in evidence, configuration, and agent outcomes.
1 Introduction
- Agent evaluation must account for tool choice, arguments, execution order, retrieved context, and external state, because plausible answers can accompany failed tasks.An agent may also reach the correct outcome through a path unlike a predefined reference.Source: §1 · p. 1
- Different interfaces, metric definitions, and result formats complicate combining evaluation outputs and judging whether configurations are comparable.The cited frameworks cover benchmark evaluation, application-level workflows, and red teaming.Source: §1 · pp. 1–2
- Metrics can have opposite optimization directions, flattened tool calls hide sequential-versus-parallel execution, and missing trace fields can be mistaken for zeros.A judge’s apparent completion can also conflict with the final task state.Source: §1 · p. 2
- AuditKIT combines benchmark, judge, performance, tool-use, and retrieval evaluation, marking unavailable evidence unscored rather than as failure.Its contributions include direction-aware comparison, configuration fingerprints, extensible components, and agent and RAG assessment.Source: §1 · p. 2
- Seven case studies illustrate how answer extraction, serving behavior, retrieval metrics, and trace structure change result interpretation, not model rankings.The paper covers architecture, metric interfaces, agent evaluation, and interoperability.Source: §1 · p. 2
2 Background and Related Work
Existing tools cover benchmark, judge-based, red-team, agent, and retrieval evaluation, but differ in how they represent metrics, traces, configurations, and comparisons. AuditKIT departs from several agent and retrieval scoring conventions while positioning itself as a measurement library rather than a broad attack-generation suite.
2.1 Evaluation frameworks
- Existing frameworks span standardized benchmarks, judge-based application evaluation, red teaming, and separate performance instrumentation, but represent evaluation artifacts differently.AuditKIT addresses red teaming only at a basic level.Source: §2.1 · p. 2
2.2 Agent and retrieval evaluation
- AuditKIT uses maximum bipartite matching for tool calls, preserves turn and parallel-versus-serialized structure, and avoids a retrieval-precision denominator that can hide misses.Its structural scores diagnose adherence to a reference execution structure but do not independently establish task success.Source: §2.2 · p. 3
2.3 AuditKIT in relation to existing frameworks
- Table 1 positions AuditKIT across ten tools and evaluation dimensions; its feature marks mean capabilities were not found in checked documentation, not that they cannot be built.AuditKIT’s red-team probes are basic, and it targets measurement and comparison rather than attack generation.Source: §2.3 · p. 3
3 Design Principles
AuditKIT uses minimal, extensible interfaces and separates core dependencies from optional components. Its design makes metric direction explicit, tracks configuration provenance, and treats unavailable evidence distinctly from measured failure.
3 Design Principles
- Custom and built-in components share minimal interfaces, while the core has no required dependencies and optional extras add heavier backends and metrics.The agent layer, parser, retrieval and deterministic RAG metrics, and Lexsi importers are stdlib-only.Source: §3 · p. 4
- Every metric declares MAXIMIZE or MINIMIZE, so grading, diffing, and leaderboards rank values using the same direction contract.A SHA-256 fingerprint covers model, data, adapter, annotator, metrics, and run settings; it identifies configuration, not reproducibility.Source: §3 · p. 4
- Unavailable measurements return no score, unknown and not_tested results are excluded from means, and missing evidence is recorded as unscored rather than failure.Unsupported tool schemas and images raise errors instead of being silently dropped.Source: §3 · p. 4
4 System Architecture
AuditKIT organizes evaluation around a shared result object, with comparison and agent-episode layers built on its pipeline. The pipeline carries samples through request adaptation, model execution, optional annotation, scoring, and aggregation; comparison and backend options extend that flow.
4 System Architecture
- The system combines an evaluation pipeline, a comparison layer, and an agent episode layer, carrying structured traces alongside predictions.Figure 1 depicts shared-result paths for replay, harness, agent, and fingerprints, with distinct arrow styles for execution modes.Source: §4 · p. 4, Figure 1 · p. 5
4.1 Evaluation pipeline
- The pipeline converts datasets into Samples, adapters into requests, model outputs into scoring context, and metric scores into an aggregated RunResult.The result retains predictions, configuration, errors, unscored counts, performance measures, and a fingerprint.Source: §4.1 · p. 4
4.2 Adapters and annotators
- Adapters change how samples become requests, while annotators extract scoreable values without altering raw text, which remains available for scoring.Regex, reasoning-span, and model-based extraction are supported.Source: §4.2 · p. 5
4.3 Model comparison and profiling
- Model comparison consumes stored RunResults and selects per-metric winners by direction; profiling records latency, throughput, model size, token usage, and cost when prices are supplied.A concurrency-one local run that answers the dataset in one generate() call records run wall-clock time as its latency sample, not per-request latency.Source: §4.3 · p. 5
4.4 Model backends
- Native tool calling is supported by api:, agent:, hf:, and vllm: backends; other backends use a prompt-based ToolCallAdapter.Chat-template settings enter the fingerprint, and image inputs reach hf: and api: backends.Source: §4.4 · p. 6
5 Using AuditKIT
AuditKIT uses a shared evaluation path across techniques, maps lm-evaluation-harness results into its RunResult, and supports multi-dataset experiment management with provenance tracking.
5 Using AuditKIT
- The same evaluate() path supports different techniques by changing components, including the backend, judge, or custom metrics.Custom metrics use the same path as built-in metrics.Source: §5 · p. 6
5.1 Academic benchmarks through lm-evaluation-harness
- ak.run_lmeval() maps real lm-evaluation-harness results into RunResult, retaining its task definitions while adding AuditKIT provenance, caching, and comparison.Source: §5.1 · p. 6
5.2 Experiment management
- evaluate_many() runs one model across several datasets, while YAML and the CLI expose the same operations.ak.generate() stores outputs for later rescoring with model="precomputed".Source: §5.2 · p. 6
- Each scoring configuration receives a fingerprint, and ExperimentDB keeps a queryable history that can be mirrored to MLflow.When available, lexsi_provenance.json links results to model checkpoints and data through RunResult metadata and saved or uploaded results.Source: §5.2 · p. 7
6 Metric Library, Custom Metrics, and Direction-Aware Comparison
AuditKIT requires metrics to declare their optimization direction and lets custom scorers share the built-in metric contract. Its library includes generation, judged RAG, lexical RAG, agent, retrieval, and deterministic RAG metric families.
6.1 The metric contract and custom metrics
- Every metric must implement score(sample, output, context) and declare a direction; custom scorers are aggregated and compared like built-in metrics.Omitting the direction raises TypeError.Source: §6.1 · p. 7
6.2 Supported metrics
- The generation family includes BLEU, ROUGE, and BERTScore [50], while GEval implements G-Eval [25].Source: §6.2 · p. 7
- Lexical RAG metrics name computed signals, while judged RAG metrics perform claim-level judging and remain separate to avoid averaging the two methodologies under one name.Source: §6.2 · p. 7
- Performance measurements attach automatically to each RunResult, and registry size counts available metrics rather than result fields.Source: §6.2 · p. 8
- Table 4 lists agent, retrieval, and deterministic RAG metrics alongside their required inputs.Source: Table 4 · p. 9
7 Evaluating Agentic and Retrieval-Augmented Systems
AuditKIT evaluates agent and retrieval-augmented systems through shared metric interfaces while preserving trace structure and distinguishing verified outcomes from trajectory diagnostics. The section describes turn-aware tool scoring, oracle-based task evaluation, retrieval checks, and interoperability with external artifacts.
7.1 Trace representation
- A tool-call reference groups calls by response turn, and unavailable trace coverage makes reference metrics ineligible rather than a “no call” score.Within a turn, calls are treated as independent; calls in later turns may depend on them.Source: §7.1 · p. 9
7.2 Tool-call and trajectory metrics
- Maximum bipartite matching pairs tool calls, while parallel-call metrics distinguish independent calls split across turns from dependent calls batched prematurely.These scores measure adherence to reference turn structure, not task success; scores without a denominator are omitted.Source: §7.2 · pp. 9–10
7.3 Task outcomes and reliability
- A verified outcome is the agent-evaluation headline, while tool-call scores remain trajectory diagnostics and depend on observed evidence.Oracles return success, failure, unknown, or error; verdict reliability depends on the oracle’s correctness and coverage.Source: §7.3 · pp. 10–11
7.4 Retrieval-augmented generation
- Retrieval scoring uses the contexts actually retrieved when available, and model-free checks assess evidence completeness, freshness, access rights, citations, numerical correctness, and abstention.Average precision divides by all relevant items, so an unretrieved gold chunk lowers the score.Source: §7.4 · p. 11
7.5 Integration with external artifacts
- AuditKIT maps several AgentTune metrics to stricter counterparts, but healing-audit records are not imported and some metrics have no counterpart yet.Mapped metrics include argument correctness, redundancy ratio, loop frequency, tool-chain order, and message-id recall.Source: §7.5 · p. 12
8 Empirical Studies
Seven small or controlled case studies illustrate how evaluation choices affect reported results across model comparison, serving, retrieval, interoperability, and tool calling. They demonstrate what AuditKIT records rather than establish model rankings.
8.1 Study 1: Evaluating pruned models
- Retention varied non-monotonically across pruning severities, and every variant failed the predefined threshold; ratios describe relative retention, not a pruning effect.Baseline absolute scores were not retained and may be near chance; the non-degeneracy heuristic also misses empty outputs.Source: §8.1 · p. 13, Table 6 · p. 13
8 Empirical Studies
- Regex extraction changed the apparent model ranking despite unchanged outputs, while serving-route behavior changed exact-match outcomes and retrieval metrics distinguished retrievers tied on hit rate.The studies are illustrative, not model rankings; the retrieval study used six questions and the runtime probe tested compatibility.Source: §8 · p. 12, §8.2 · p. 13, §8.4 · p. 14
8.2 Study 2: Comparing language models
- Regex extraction made the initially lowest-scoring model competitive without changing its generated outputs, but post-extraction differences were not statistically significant on aggregate data.Whole-run wall time was not used to rank speed, and paired significance testing was unavailable because per-sample scores were not retained.Source: §8.2 · p. 13, Table 8 · p. 13
8.3 Study 3: Financial question answering
- Grounding favored one model, while the quantized larger model led on numeric matching and judge score; overlapping uncertainty intervals prevented ranking those latter gaps.The comparison used one company’s filing questions, and quantization confounded the 8B-versus-13B comparison.Source: §8.3 · pp. 13–14, Table 9 · p. 14
8.4 Study 4: Evaluation across serving runtimes
- A model loaded across serving routes yet produced different evaluation outcomes when one route retained a special token that caused an exact-match failure.The probe checked runtime compatibility, not comparative model quality.Source: §8.4 · p. 14, Table 10 · p. 14
8.5 Study 5: Retrieval and answer quality
- BM25 and TF-IDF tied on hit rate, but ranking-sensitive measures distinguished their retrieved evidence; answer scores changed with prompt order while retrieval scores stayed fixed.Six questions illustrated separate retrieval and generation measurements rather than ranking the retrievers.Source: §8.5 · p. 14, Table 11 · p. 14
8.6 Study 6: Evaluating artifacts from the Lexsi Labs stack
- Placeholder-artifact tests checked parsing and evidence linking, with final-state assertions verifying recorded episodes and missing tool arguments or rewards left unscored.Links were accepted only when identifiers supported them; descriptive labels and aggregate manifests could not identify an episode.Source: §8.6 · p. 15, Table 12 · p. 15
8.7 Study 7: Choosing a tool-calling model
- The models performed similarly on tool-call categories, while the larger model more often declined irrelevant requests; overlapping intervals made that difference suggestive.For parallel calls, call-level F1 retained partial credit that exact-set scoring hid; the results describe only this benchmark slice.Source: §8.7 · p. 15, Table 13 · p. 15
9 Limitations and Ethics Statement
AuditKIT’s scope is constrained in latency measurement, agent evaluation, and the evidence base of its case studies. Its red-teaming support is basic, and its source-available license conditions commercial use.
Scope and ethics
- Agent evaluation scores single-response models one turn at a time, while red-teaming support is basic; the source-available license is not OSI-approved and conditions commercial use.Image-specific metrics and several evaluation extensions are outside the current implementation.Source: §9 · pp. 15–16
Latency and performance
- Latency is sampled per generate() call, so a single-call local run yields whole-run time rather than per-request latency; time-to-first-token and inter-token latency are outside scope.The local hf: backend also does not record token usage.Source: §9 · p. 16
Case-study scope
- Case studies are small and use one judge configuration, so judged verdicts lack human-annotation validation and may favor a candidate from the judge’s model family.Study 2 significance tests use aggregate accuracies because per-sample paired scores were not retained.Source: §9 · p. 16
10 Conclusion
AuditKIT provides a shared evaluation and comparison interface for language models and agents. Its case studies show that evaluation choices shape reported results, supporting the need to retain evaluation context and interpret metrics alongside their evidence.
Conclusion
- AuditKIT combines direction-aware metrics, configuration tracking, and extensible components with structured assessment of tool use, retrieval, and task outcomes.For agents, it preserves turn-level execution, separates trajectory diagnostics from oracle-based outcomes, and marks insufficient evidence unscored.Source: §10 · p. 16
- Answer extraction changed apparent model rankings, serving-runtime differences changed exact-match outcomes, and retrieval metrics separated systems that hit rate treated as tied.The paper concludes that evaluation context should be retained and each metric read alongside the evidence used to compute it.Source: §10 · p. 16
Availability
- AuditKIT is available with documentation, a CLI, and examples under the Lexsi Labs Source Available License v1.2.The license is not an OSI-approved open-source license and conditions commercial use.Source: §10 · p. 16
A Appendix: Code Listings for Library API and Usage (Section 5)
The appendix collects examples referenced by the paper, covering the core API, agents and RAG, command-line and YAML usage, and the judge prompt.
A Appendix: Code Listings for Library API and Usage (Section 5)
- Appendix examples are grouped into core API, agents and RAG, command-line and YAML configuration, and the judge prompt used in Studies 1 and 3.The main text keeps six short listings and refers readers to the appendix for the remaining examples.Source: A Appendix: Code Listings for Library… · p. 19
A.1 Core API
The appendix demonstrates AuditKIT’s evaluation interfaces across models, metrics, agent traces, retrieval, and command-line or YAML workflows. It also illustrates how answer extraction, comparisons, and evidence checks are represented in results.
A.1 Core API
- AuditKIT examples evaluate local and callable backends, extract parsed answers for scoring, and compare baseline and pruned models, with exact_match moving from 0.8200 to 0.7900 (delta=-0.0300; n=500->500).The comparison exposes quality-versus-measured-size/latency; examples report latency mean 65174.78 ms and local-model size 15316.51 MB.Source: §A.1 · pp. 19–21
A.2 The full Metric-subclass form
- A registered Metric subclass supports configurable scoring logic and is used like a built-in metric by its registered name.The example requires a class-level metric direction.Source: §A.2 · p. 22
A.3 Agents and RAG
- For retrieval, AuditKIT combines ranked retrieval metrics, judged grounding, and deterministic checks; the example reports evidence_set_recall 1.0, freshness 1.0, and acl_compliance 1.0.The checks also report stale_hit_rate 0.0 and unauthorized_hit_rate 0.0. Agent examples distinguish tool-call matching and trajectory checks; constructed AgentTune reports retain only the trace-supported metrics and can leave others ineligible.Source: §A.3 · pp. 23–24
A.4 Command-line interface
- The CLI examples cover evaluation, model comparison, red-teaming, resource listing, and agent inspection, execution, and rescoring.Agent inspection and dry-run are described as local and read-only.Source: §A.4 · pp. 24–25
A.5 Configuration via YAML
- The YAML example configures a model, generation settings, prompts, adapter, output format, and experiment metadata.The example includes temperature, token limit, seed, and concurrency settings.Source: §A.5 · p. 25
A.6 Judge prompt
- The factual-correctness judge prompt keeps the same structure across Studies 1 and 3, changing the retrieved-text field name to match each study’s context.It asks whether the model answer matches the real answer and requests a yes-or-no choice.Source: §A.6 · p. 25