PaperlayerView exact revision on arXiv

Source-linked AI summary

SEAR: Schema-Based Evaluation and Routing for LLM Gateways

Zecheng Zhang, Han Zheng, Yue Xu

arXiv:2603.26728v1cs.DBcs.AIcs.CL

TL;DR

Production LLM gateways need fine-grained evaluation and operationally grounded routing because workloads, failures, providers, and costs vary substantially. SEAR addresses this with an extensible relational schema and schema-driven judge using in-schema reasoning and multi-stage generation. Across 3,000 production sessions, it achieves high signal accuracy and identifies a substitute model with 90% lower input cost and 92% lower output cost while maintaining comparable quality.

  • Problem

    Production LLM traffic requires quality-aware, interpretable evaluation and routing, but existing benchmarks, evaluators, and routers provide limited real-world coverage or signal-level transparency.

  • Method

    SEAR uses an extensible relational schema and LLM judge to generate around one hundred typed evaluation signals alongside gateway metrics through self-contained instructions, in-schema reasoning, and multi-stage generation.

  • Results

    Across 3,000 production sessions, SEAR achieves high accuracy on human-labeled signals and identifies a substitute model with 90% lower input cost and 92% lower output cost while maintaining comparable quality.

  • Takeaways & Limitations

    SEAR unifies evaluation and routing in a SQL-queryable layer that supports human-interpretable explanations and practical cost-effective routing decisions.

  • Takeaways & Limitations

    The routing case study covers one organization, one workload slice, and an offline replay of 100 sessions, so it is not a general routing benchmark.

Abstract

from arXiv · show

Evaluating production LLM responses and routing requests across providers in LLM gateways requires fine-grained quality signals and operationally grounded decisions. To address this gap, we present SEAR, a schema-based evaluation and routing system for multi-model, multi-provider LLM gateways. SEAR defines an extensible relational schema covering both LLM evaluation signals (context, intent, response characteristics, issue attribution, and quality scores) and gateway operational metrics (latency, cost, throughput), with cross-table consistency links across around one hundred typed, SQL-queryable columns. To populate the evaluation signals reliably, SEAR proposes self-contained signal instructions, in-schema reasoning, and multi-stage generation that produces database-ready structured outputs. Because signals are derived through LLM reasoning rather than shallow classifiers, SEAR captures complex request semantics, enables human-interpretable routing explanations, and unifies evaluation and routing in a single query layer. Across thousands of production sessions, SEAR achieves strong signal accuracy on human-labeled data and supports practical routing decisions, including large cost reductions with comparable quality.

1 Introduction

Production LLM traffic is diverse, costly, and difficult to evaluate with existing coarse or opaque methods. SEAR addresses these gaps by combining fine-grained schema-based evaluation with interpretable, operationally grounded routing.

  • Motivation: Production workloads span domains, modalities, conversation structures, and complexity levels, while failures concentrate in particular traffic subsets.No single model or provider is optimal for all cases, and costs can differ by orders of magnitude.
  • Motivation: Public benchmarks often fail to reflect real-world usage, leaving teams dependent on manual spot checks, small internal benchmarks, and proxy metrics.The challenge grows with agentic inference, reasoning models, tool calls, and longer contexts.
  • Evaluation Gap: Existing production evaluators commonly produce free text, collapse quality into one score, or apply fixed rubrics across limited dimensions.These approaches provide limited structured coverage for complex production evaluation.
  • Routing Gap: Existing routers select models through optimization objectives but remain black boxes without interpretable, signal-level explanations for production decisions.Gateway teams must balance performance with provider choice, cost, latency, and throughput.
  • SEAR: SEAR generates around one hundred typed, interlinked signals spanning request context, intent, response characteristics, issue attribution, and quality scores.Signals are derived through LLM reasoning and linked to per-signal evidence for human-interpretable routing explanations.
  • SEAR: SEAR combines SQL-queryable evaluation records with gateway operational metrics and validates the approach on production traffic with high signal accuracy and practical routing decisions.The generation process uses self-contained instructions, in-schema reasoning, and multi-stage structured generation.

2 Related Work

Prior work provides scalable LLM evaluation, schema-aware extraction, gateway observability, and cost-quality routing, but these capabilities remain fragmented. SEAR targets the missing integration of shared, interpretable signals with continuous evaluation and routing.

  • LLM Evaluation: LLM evaluators range from single-score systems to multi-dimensional rubrics, but typically use few predefined dimensions or manual rubric design.Single-score evaluators can suffer from scoring bias and self-inconsistency.
  • Structured Generation: Native structured-output constraints improve formatting but can degrade reasoning relative to free-form generation.This creates a design challenge for schema-constrained evaluation.
  • Schema-Aware Extraction: Schema-aware extraction systems coordinate multi-step extraction but do not provide a shared signal space for continuous quality analysis and routing.Their scope differs from SEAR’s unified evaluation-and-routing layer.
  • Gateways and Operations: Gateways, guardrails, and observability tools support serving, policy enforcement, tracing, and cost analytics, while evaluation results are rarely connected to routing without custom engineering.This leaves evaluation and operations operationally separated.
  • Routing: Cost-quality routers span preference-based, graph-based, cascade, and reinforcement-learning approaches, but most lack per-signal attributions and remain limited in transparent drift diagnosis.Signal-driven routers use heuristic or classifier-extracted features, which are shallow compared with SEAR’s reasoning-based signals.

3 The SEAR Framework

SEAR organizes evaluation and gateway operations into an extensible relational layer. A gateway logs all traffic, sampled sessions are judged into linked semantic tables, and standard SQL queries support analysis, diagnosis, and routing.

  • Overview: SEAR co-locates structured evaluation records and gateway operational metrics in one SQL-queryable data layer.Accumulated signals support routing, drift detection, and provider benchmarking.
  • Gateway Architecture: The gateway routes requests, logs latency, token counts, cost, and errors for every request, and samples a configurable fraction for LLM-as-judge evaluation.Scoring all traffic is described as prohibitively expensive.
  • Semantic Evaluation Tables: The judge populates four linked semantic tables covering context and intent, response characteristics, issue attribution, and quality scores, alongside the gateway metrics table.Together, the schema contains around one hundred typed columns spanning the request lifecycle.
  • Typed Signals: Discrete boolean, categorical, and ordinal types replace floating-point scores to give values explicit semantic boundaries and reduce ambiguity on wide numeric scales.The design responds to instability, score clustering, prompt sensitivity, and weak discrimination between nearby numeric values.
  • Cross-Table Design: Cross-table links align requested behavior, produced behavior, issue responsibility, and severity, enabling consistency checks and signal traceability through SQL joins.Flagged records can be re-judged or removed when linked columns disagree.
  • Extensibility: Optional foreign-key links and appended columns let the schema evolve incrementally without disrupting existing data or core evaluation workflows.Extensions can be attached selectively, leaving unrelated records without an extension foreign key.
  • Operational Metrics: Gateway metrics cover 100% of traffic and support aggregate observability plus slice-level quality–latency, cost, throughput, and provider analyses when joined with evaluated sessions.Recorded metrics include latency, throughput, failures, token usage, and cache usage.

4 Schema-Driven Judge

SEAR makes large structured evaluation generation reliable by combining self-contained signal instructions, in-schema reasoning, and staged calls aligned with schema dependencies. The design reduces output complexity while preserving reasoning and enabling database-ready records.

  • Challenges: Generating around one hundred typed columns creates challenges in output validity, reasoning quality, and orchestration across dependent tables.SEAR addresses these challenges with self-contained instructions, in-schema reasoning, and multi-stage generation.
  • Structured Generation: Typed JSON schemas map fields directly to table columns, producing database-ready records without post-processing parsers.Each table emits all of its signals in a single structured-output call.
  • Signal Instructions: Self-contained column-level instructions reduce confusion among the 15–30 related fields in each table.This design targets the multiplicative decline in instruction-following compliance as simultaneous constraints increase.
  • In-Schema Reasoning: In-schema reasoning places a temporary reasoning field first in the JSON schema, allowing reasoning and signal emission in one autoregressive pass.The reasoning field is dropped before database insertion and requires no additional LLM invocation.
  • In-Schema Reasoning: The reasoning field guides the judge to identify the task, derive signals step by step, and verify consistency with the task.The approach improves signal extraction and reasoning while preserving single-call-per-table generation.
  • Multi-Stage Generation: Multi-stage generation follows request flow from context and intent to response characterization, issue attribution, and quality scoring.Each stage receives the conversation context and upstream outputs required by its dependencies.
  • Multi-Stage Generation: Staging reduces each call to 19–31 columns and scopes context to immediate dependencies, improving generation stability and reducing instruction ambiguity.A single-stage attempt frequently produced malformed or incomplete JSON, while staged outputs can be committed with linked foreign keys in one transaction.

5 Data-Driven Evaluation and Routing

SEAR turns evaluation and routing into SQL-queryable workflows over typed signals and gateway metrics. It supports model, provider, user, and real-time routing scenarios with interpretable, signal-level evidence.

  • Data-Driven Evaluation: SEAR reduces evaluation and routing workflows to standard SQL queries over joint evaluation and gateway tables.Signals are typed and timestamped, while gateway records provide operational metrics for joint analysis.
  • Data-Driven Evaluation: Model evaluation compares candidate models using LLM-caused issue rates and average severity by domain over recent coding-task sessions.The comparison uses the most recent thirty days of data.
  • Data-Driven Evaluation: Provider evaluation ranks providers by task quality, using median latency as a tiebreaker alongside gateway operational metrics.This combines evaluation scores with operational measurements.
  • Data-Driven Evaluation: User evaluation profiles risk by aggregating safety-sensitive content, ambiguous instructions, and noisy context across sessions.The gateway can use these statistics to trigger guardrails or adjust sampling rates.
  • Data-Driven Routing: SEAR derives routing signals through LLM reasoning over full request context, producing human-interpretable recommendations grounded in per-signal evidence.This captures complex semantics such as issue attribution and cross-signal quality that shallow extractors miss.
  • Data-Driven Routing: Routing queries observed quality and cost across traffic slices to produce model and provider strategies, including cheapest-model and near-best-provider selection policies.Real-time routing uses a lightweight LLM to classify incoming context and look up matching SEAR-derived policies.

6 Experiments

SEAR is evaluated on production sessions using multiple judge configurations and routing experiments. Its default judge achieves strong per-signal accuracy, while a case study finds a much cheaper model with effectively tied quality under a limited offline evaluation.

  • 6.1 Setup: 3,000 production sessions from three organizations are sampled, with 300 human-annotated sessions reserved as a held-out test set.The workloads include multilingual, roleplay, and translation-heavy profiles, plus single-turn and multi-turn agentic sessions.
  • 6.1 Setup: The experiments vary GPT-5-mini and GPT-5.2 judges across low and high reasoning effort, using in-schema reasoning for the reported configurations.Metrics cover boolean, categorical, ordinal, and aggregate prediction performance.
  • 6.2 Evaluation Performance: Under the default GPT-5-mini high-effort configuration, boolean accuracy exceeds 91%, categorical accuracy exceeds 92%, and ordinal accuracy ranges from 80% to 86%.Higher reasoning effort consistently improves accuracy, and GPT-5.2 and GPT-5-mini at high effort perform similarly across most metrics.
  • 6.3 Routing Performance: gemini-2.5-flash-lite reaches a composite quality score of 17.57 while costing 90% less on input tokens and 92% less on output tokens than the deployed model.The candidate is selected on Organization C’s simple-complexity slice.
  • 6.3 Routing Performance: Across 100 replayed sessions, 72 are ties, 12 favor gemini-2.5-flash-lite, and 16 favor claude-haiku-4-5, yielding a 48% win rate.The result is reported as effectively tied quality at the stated lower input and output costs.
  • 6.3 Routing Performance: The routing case study covers one organization, one workload slice, and an offline replay of 100 sessions, so it is not a general routing benchmark.This scope boundary limits how broadly the cost-quality result should be interpreted.
  • 6.3 Routing Performance: Real-time routing requires a lightweight model to classify incoming context signals before applying precomputed SEAR policies at serving time.The routing case study itself operates offline, with post-hoc judging and warehouse-derived policies.

7 Ablation Studies

SEAR’s ablations show that consistency filtering, multi-stage generation, reasoning effort, and in-schema reasoning jointly determine structured-evaluation reliability. The study also identifies meta-task confusion and model-size trade-offs affecting downstream routing use.

  • Cross-Table Consistency Checks: 2 (0.7%) inconsistent records remained for GPT-5-mini high with in-schema reasoning, versus 34 (11.3%) for GPT-5-mini low without it.Filtering reduced error rate by up to +7.5% for weaker configurations but slightly over-pruned the strongest configuration by −0.3%.
  • Generation Strategy: Single-stage generation frequently produced malformed or incomplete JSON, whereas multi-stage generation improved coherence by splitting the schema across calls.Each later stage can reference upstream outputs, making schema-constrained evaluation practical at this scale.
  • Reasoning Effort and In-Schema Reasoning: Higher reasoning effort produced larger gains than adding in-schema reasoning at the same effort level, while in-schema reasoning consistently improved most metrics.GPT-5-mini high with in-schema reasoning outperformed GPT-5.2 low on error rate, indicating that increased inference-time compute can offset capability differences.
  • Meta-Task Confusion: Meta-task confusion caused false positives when the judge conflated schema instructions with the user’s original task.The confusion rate fell from 7.3% for GPT-5-mini low to 0.0% for GPT-5.2 high as reasoning strength increased.
  • Lightweight Routing Signals: GPT-5-nano was less accurate than the default SEAR judge but required only one of four tables and could support lower-latency routing-focused extraction.Restricting extraction to task type, domain, and complexity can reduce latency further, while coarse complexity buckets limit the effect of adjacent ordinal errors.

8 Future Work

Future work targets online validation, broader data coverage, and model diversity. These directions extend preliminary low-cost signal extraction toward organization-specific routing across more tasks, complexity levels, and providers.

  • Online Routing: End-to-end online experiments will test whether organization-specific policies preserve downstream quality while providing schema-grounded explanations for each routing decision.The proposed experiments combine the lightweight context classifier with per-request routing policies.
  • Data Coverage: The current dataset covers limited models and task types, so routing experiments focus on simple-complexity slices with sufficient cross-model overlap.Planned expansion targets broader task and complexity ranges and a wider model pool.
  • Model Diversity: SEAR will be evaluated with non-GPT closed-source and open-source models beyond the GPT family.This extends the judge evaluation across a broader model ecosystem.

9 Conclusion

SEAR combines schema-based LLM evaluation with gateway routing in one SQL-queryable layer. Across 3,000 production sessions, it achieved high signal accuracy and identified substantially cheaper substitute models with comparable quality.

  • Conclusion: SEAR combines around one hundred typed evaluation signals, gateway metrics, cross-table consistency checks, and multi-stage structured generation in one relational layer.The system covers context, intent, response characteristics, issue attribution, quality scores, latency, throughput, cost, and errors.
  • Conclusion: 90% lower input cost and 92% lower output cost were achieved by a substitute model while maintaining comparable quality in a routing case study.Across 3,000 production sessions, the judge also achieved high accuracy against human-labeled boolean, categorical, and ordinal signals.

A SEAR Full Database Schema

The full SEAR database links semantic evaluation tables and gateway metrics through typed columns and foreign keys. Its implementation emphasizes consistency validation and safe structured insertion rather than direct LLM-generated SQL.

  • Schema Organization: The full schema contains four semantic evaluation tables and a gateway metrics table linked by foreign-key relationships.Figure 4 presents their columns and types alongside the relationships connecting them.
  • Consistency Validation: Cross-table checks detect tool-call absence violations, issue–severity mismatches, and orphan severity entries.Code 1 implements these checks with SQL joins across context, response, issue-attribution, and evaluation tables.
  • Data Insertion: Structured output avoids SQL-generation overhead, code-generation errors, and security risks by emitting typed values for ORM insertion.Direct LLM-generated SQL would add syntax, type, escaping, and execution concerns unrelated to evaluation quality.

E.1 Evaluation Examples

The examples show how SEAR’s structured signals support SQL-based diagnosis, model comparison, provider ranking, and cost-aware routing. These queries connect semantic evaluation with gateway metrics to make routing decisions operationally actionable.

  • Model and provider analysis: Code 3 compares coding-task models by LLM-caused issue rate and average issue severity across domains over the most recent 30 days.The query groups results by context domain and orders candidates by issue rate and severity.
  • Model and provider analysis: Code 4 ranks providers by average task quality and median latency, using successful requests and requiring at least 30 observations.It joins gateway metrics, response records, and evaluation scores before ordering by quality and latency.
  • Risk profiling: Code 5 profiles users by aggregating safety-sensitive content, ambiguous instructions, and noisy context across sessions.The query requires at least 10 sessions per user and orders profiles by safety and ambiguity rates.
  • Cost-aware routing: Code 6 finds the cheapest model whose aggregate quality is within 10% of the best-performing model.The model_provider table supplies input and output costs for each model–provider pair.
  • Cost-aware routing: Additional routing examples substitute models by quality on simple tasks, identify cheaper creative-writing alternatives that outperform the deployed model, and select providers within 5% quality by TTFT.These examples operationalize different quality, cost, and responsiveness constraints.

F Dataset

The dataset section reports ground-truth signal distributions for a 300-session test set and illustrates the schema-conforming generation and routing queries used with those signals. The examples combine structured evaluation fields with quality, price, and time-to-first-token measures.

  • Ground-truth dataset: Tables 9–12 report ground-truth distributions for all signals in a 300-session test set, with 100 sessions per organization.Boolean signals use True percentages, while categorical and ordinal signals report the dominant value and its percentage.
  • Schema-conforming generation: The SEAR judge is instructed to produce one JSON object matching the target table schema and to use raw input and upstream structured outputs as evidence.Field descriptions specify definitions, value conditions, scope, and edge cases.
  • Schema-conforming generation: Reasoning fields guide the judge to identify the task, derive fields step by step, and self-check consistency, using conservative schema values when evidence is insufficient.This procedure is designed to keep generated records grounded in available evidence.
Loading 2603.26728v1…