Source-linked AI summary
TRACER: Trace-Based Adaptive Cost-Efficient Routing for LLM Classification
Adam Rida
TL;DR
LLM classification calls can generate free supervision for cheaper surrogate models, but deployment requires reliable routing and transparent boundaries. TRACER learns surrogates from teacher traces and activates them only through a parity gate, while producing artifacts about handled and deferred regions. It achieves 83.2–100% coverage on Banking77, fully replaces the teacher on CLINC150, and correctly refuses deployment on MNLI.
Problem
Production LLM classifiers incur per-query cost, motivating the question of when trace-trained surrogates are reliable enough to deploy and how their handled-versus-deferred boundary evolves.
Method
TRACER trains surrogate classifiers and an uncertainty acceptor on production teacher traces, promotes only pipelines passing a parity gate at threshold α, and generates routing-boundary artifacts.
Results
TRACER achieves 83.2–100% coverage on Banking77, fully replaces Sonnet 4.6 on CLINC150, and the parity gate refuses deployment on MNLI.
Takeaways & Limitations
The system matches teacher quality while offloading traffic to near-zero-cost surrogate inference when the representation supports reliable separation.
Takeaways & Limitations
On CLINC150, calibration TA of 0.952 at α = 0.95 corresponded to test TA of 0.930, so the gate’s guarantee depends on calibration-split representativeness.
Abstract
from arXiv · showhide
Every call to an LLM classification endpoint produces a labeled input-output pair already retained in production logs. These pairs constitute a free, growing training set: a lightweight surrogate trained on them can absorb a significant portion of future traffic at near-zero marginal inference cost. The open questions are when the surrogate is reliable enough to deploy, what it handles versus defers, and how that boundary evolves as data accumulates. We introduce TRACER (Trace-based Adaptive Cost-Efficient Routing), an open-source system that trains ML surrogates on an LLM's own production traces and governs deployment through a parity gate: the surrogate is activated only when its agreement with the LLM exceeds a user-specified threshold α. To make the routing boundary transparent, TRACER generates interpretability artifacts describing which input regions the surrogate handles, where it plateaus, and why it defers. On a 77-class intent benchmark with a Sonnet 4.6 teacher, TRACER achieves 83-100% surrogate coverage depending on the quality target α; on a 150-class benchmark, the surrogate fully replaces the teacher. On a natural language inference task, the parity gate correctly refuses deployment because the embedding representation cannot support reliable separation. The system is available as open-source software.
1. Introduction
TRACER turns production LLM traces into a continually improving surrogate-routing system. A parity gate controls deployment quality, while interpretability artifacts expose the surrogate’s routing boundary and evaluation tradeoffs.
- Motivation: Production LLM classification calls create logged labeled traces that can train surrogates without manual labeling.These traces are already retained for billing and compliance and can become training data.
- Contribution: TRACER removes the need for upfront labels by using deferred teacher responses as supervision for a self-reinforcing trace flywheel.Remaining deferrals concentrate new training signal in regions where the surrogate is uncertain.
- Deployment: The parity gate activates the surrogate only when held-out agreement with the teacher exceeds a user-specified threshold α.This addresses whether the surrogate is reliable enough to deploy.
- Interpretability: TRACER’s interpretability artifacts describe the routing partition, showing which inputs the surrogate handles and which it defers.The artifacts target the metalevel routing decision rather than individual predictions.
- Evaluation: The evaluation studies coverage–quality tradeoffs across two intent benchmarks and reports a negative NLI result supporting the gate’s safety behavior.The contribution includes an alpha sweep on 77- and 157-class intent tasks and a negative result on NLI.
2. Related Work
TRACER combines learning-to-defer, LLM routing, and explainability while changing the supervision and routing targets. It learns from production teacher traces, routes to classical ML, and explains the deferral boundary.
- Learning to defer: Existing learning-to-defer methods assume ground-truth labels collected before deployment, whereas TRACER acquires labels from teacher traces during production.This targets settings with limited labeling budgets or evolving label schemas.
- LLM routing and cascades: Prior LLM-routing systems route among LLMs, while TRACER routes from an LLM to classical ML as a cheaper target.The related routing methods include cascades, predicted-difficulty routing, preference-based routing, and self-verification.
- XAI for routing boundaries: Prior explainability methods target classifier predictions, whereas TRACER adapts them to explain what the surrogate handles versus defers.Its artifacts describe the deferral partition rather than individual predictions.
- Summary: TRACER integrates no-upfront-label supervision, classical-ML routing, and an inspectable routing boundary in one system.The paper states that no prior system integrates all three components.
3. Method
TRACER fits surrogates and acceptors from teacher traces, then promotes only pipelines that satisfy a parity-based quality constraint. Its artifacts make handled and deferred regions inspectable across refits.
- Problem setting: TRACER fits a surrogate f and acceptor a from teacher-labeled traces, producing a hybrid classifier that handles or defers each input.The acceptor determines whether an input is served by the surrogate or sent to the teacher.
- Problem setting: The routing objective maximizes surrogate coverage subject to teacher agreement TA≥α.Coverage is the probability that the acceptor score reaches the routing threshold.
- Pipeline selection: At each refit, TRACER compares classifier families and selects the highest-coverage pipeline meeting the agreement constraint.Global uses one surrogate for all traffic when overall agreement suffices; L2D gates only high-confidence predictions.
- Acceptor: The acceptor combines top-1 probability, top-2 probability, their margin, and normalized entropy to estimate teacher agreement.Its threshold is calibrated by sweeping unique acceptor scores, and it provides tighter agreement than raw probability thresholding at the same coverage in the cited Banking77 comparison.
- Deployment gate: Before production promotion, a candidate must pass a held-out parity gate with TA≥α and clear a 5% coverage floor.If no candidate passes, TRACER falls back to full teacher reliance.
- Continual learning: The active pipeline evolves over time: day-one traffic trains the first candidate, later traffic is split between surrogate handling and teacher deferral, and each update refits on accumulated traces.Deferred calls provide new examples concentrated near the surrogate’s decision boundary.
- Interpretability artifacts: TRACER generates artifacts that summarize slice-level handling and agreement, representative handled/deferred examples, contrastive boundary pairs, temporal coverage changes, and disagreements.Together these artifacts expose which categories remain teacher-dependent and how routing changes after refits.
- Interpretability artifacts: Temporal deltas compare per-label handled rates across refits, allowing operators to audit expansion or regression in coverage.These deltas are available from the second refit onward.
4. Experiments
TRACER evaluates trace-trained surrogates across intent and NLI benchmarks, sweeping the parity target α to measure coverage, quality, accumulation dynamics, cost, and routing interpretability. It fully offloads CLINC150, trades coverage for agreement on Banking77, and correctly refuses MNLI deployment.
- Setup: TRACER evaluates Sonnet 4.6 traces on Banking77, CLINC150, and MNLI using precomputed BGE-large-en-v1.5 embeddings and an α sweep.Training uses incremental daily batches, followed by held-out test evaluation.
- Main Results: 100% coverage at α = 0.95 on CLINC150 lets a logistic-regression surrogate fully replace Sonnet 4.6, although test TA is 93.0% versus 95.2% calibration agreement.The calibration–test gap makes split representativeness an important caveat.
- Main Results: The confidence-threshold baseline reaches 87.8% versus 83.2% coverage at α=0.95 and 99.8% versus 96.1% at α=0.90, benefiting from all 10,003 traces at once.TRACER reaches similar performance from 2,001 traces and achieves tighter TA at α=0.95: 0.959 versus 0.951.
- Main Results: MNLI coverage remains exactly 0% across nine tested configurations, so the parity gate refuses deployment because BGE embeddings cannot separate entailment, neutral, and contradiction.The evaluation uses ground-truth labels as a generous stand-in teacher.
- Flywheel Dynamics: At α=0.95, Banking77 coverage grows from 73.4% with 2,001 Day-1 traces to 83.4% by Day 4 while maintaining TA above 0.95.At α=0.80, Banking77 reaches 100% coverage on Day 1 from 2,001 traces; CLINC150 reaches 100% by Day 2 at α=0.95.
- Cost: At α=0.95, projected Banking77 teacher-call cost falls from $26/day to $4.40/day at 83.2% coverage, an 83% reduction; α=0.80 reaches a 100% reduction.The projection assumes Sonnet 4.6 pricing of $3/M input and $15/M output tokens.
- Artifact Inspection: Interpretability inspection is qualitative rather than a practitioner user study; Banking77 artifacts expose class- and length-specific handling rates and meaningful handled/deferred contrasts.Handled inputs score 0.94–0.96, while deferred inputs score 0.00.
5. Discussion
TRACER is effective for intent classification with well-separated embedding boundaries, but its safe deployment depends on parity calibration and remains limited on compositional tasks and teacher errors.
- Intent classification with well-separated embedding clusters is the current setting where frozen embeddings plus classical ML perform well.
- On CLINC150, calibration TA of 0.952 at α = 0.95 exceeds the gate threshold, but test TA is 0.930, exposing incomplete transfer to unseen data.Conservative thresholds or conformal calibration are suggested as mitigations.
- On MNLI, compositional reasoning produces embedding boundaries that remain inseparable regardless of data volume, so the parity gate correctly prevents deployment.Fine-tuning the encoder on accumulated traces is proposed as a future extension.
- On Banking77, the surrogate matches teacher behavior rather than ground truth, with teacher test accuracy of 81.0% and end-to-end GT accuracy of 81.6% at α=0.95.Deferring the hardest inputs raises GT accuracy toward the teacher-only baseline but does not improve on the teacher.
- Future work targets broader tasks, stronger routing baselines, encoder diversity, practitioner evaluation of artifacts, incremental fitting, and conformal guarantees.
6. Conclusion
TRACER treats LLM calls as sources of training signal for downstream surrogates and uses a parity gate to control cost-efficient offloading. Its results span substantial intent-task offload, full replacement on CLINC150, and refusal on MNLI.
- TRACER operationalizes the view that LLMs can serve as labeling engines whose calls produce training signals for downstream surrogates.
- On Banking77, the alpha knob controls coverage from 100% at α=0.80 to 83.2% at α=0.95.
- On CLINC150, the surrogate fully replaces the teacher at all quality targets.
- On MNLI, the parity gate correctly refuses deployment, demonstrating safety by design.
- The system is available as open-source software.
A. Teacher Prompts
The teacher prompts use Claude Sonnet 4.6 with structured JSON output for banking and general intent classification, while traces can produce spurious labels outside the intended class set.
- All teacher calls use Claude Sonnet 4.6 with structured output constrained by a JSON schema.
- The banking prompt asks for exactly one label from 77 listed intent categories and returns it as JSON.
- The general intent prompt asks for exactly one label from 150 in-scope categories and returns it as JSON.
- Seven spurious teacher labels expand the trace data to 157 effective classes in the 150-class benchmark.
B. Full Artifact Output
The artifact output reports Banking77 routing behavior through contrastive examples and per-label handled rates, revealing both easy and difficult intent regions at α = 0.95.
- At α = 0.95, the artifact includes five Banking77 contrastive boundary pairs for qualitative inspection.
- Examples contrast handled and deferred utterances for refund visibility, card activation, mobile payments, ATM support, and automatic top-up.
- Handled rates range from 78.2% for card payment not recognised to 80.4% for top up failed among selected slices.
- The selected slice table reports 83.2% for declined card payment, 93.4% for activate my card, 95.0% for cash withdrawal charge, and 96.4% for transaction charged twice.