PaperlayerView exact revision on arXiv

Source-linked AI summary

Signed Lexical Confidence for Risk-Calibrated Intent Routing

Yezhou Cheng, Zehua Yang, Bojun Lin

arXiv:2610.00262v1cs.CLcs.AI

TL;DR

Selective intent routing needs confidence evidence beyond the base classifier’s representation to decide which predictions to accept or defer. The paper combines semantic margin with prediction-aligned signed lexical support, then calibrates an acceptance threshold independently. The score improves error ranking across three datasets relative to a learned semantic gate, with coverage gains where the primary operating point leaves headroom.

  • Problem

    Common confidence scores summarize the base classifier’s preferences, while complementary lexical cues may provide evidence about whether its selected intent is supported.

    Source: §I · p. 1

  • Method

    A two-feature gate combines the sentence classifier’s semantic margin with signed lexical support for its predicted intent, then independently calibrates an acceptance threshold.

    Source: §I · p. 1, A. Prediction and witness evidence · p. 2

  • Results

    Error ranking improves relative to a learned semantic gate across BANKING77, CLINC150, and HWU64, with increased accepted coverage at the primary target on datasets with available headroom.

    Source: §VII · p. 7

  • Takeaways & Limitations

    Prediction-aligned lexical support is a useful complement to semantic confidence for selective routing while preserving the fixed classifier’s predictions.

    Source: §VI · p. 6, §VII · p. 7

    Orthographic perturbations and unsupported intents reduce calibrated performance, motivating representative calibration streams and explicit out-of-scope handling.

    Source: D. Calibration transfer and input shift · p. 6

Abstract

from arXiv · show

Selective intent routing allows an assistant to act on reliable predictions while deferring uncertain requests. Standard confidence scores primarily reflect the base model's representation, leaving an opportunity to incorporate complementary evidence without changing its decisions. We introduce a signed lexical gate that combines a sentence classifier's logit margin with a sparse lexical model's support for the classifier's predicted intent. By assigning positive evidence to lexical agreement and negative evidence to a lexically favored competing intent, the gate retains more information than either unsigned lexical confidence or a hard agreement rule. An independent binomial calibration stage selects an operating threshold for a specified risk target. Across ten runs on BANKING77, CLINC150, and HWU64, the proposed score reduces area under the risk-coverage curve by 15.8%, 15.1%, and 11.8% relative to a learned semantic-only gate. At a nominal 5% error target, it increases accepted coverage by 1.83 and 5.14 percentage points on BANKING77 and HWU64, while CLINC150 is already near full coverage. At a stricter 2% target, the simultaneous binomial procedure yields a nonempty policy in all 30 dataset-run combinations at the available calibration budgets. Matched controls show that the proposed feature improves average error ranking over the tested unsigned lexical-confidence feature, with dataset-dependent gains over binary agreement. The resulting two-feature gate provides a compact, interpretable confidence enhancement for risk-calibrated intent routing while preserving the base classifier's predictions.

I. INTRODUCTION

Selective intent routing accepts requests with sufficient evidence and defers the rest, but common confidence scores reflect mainly the base classifier. This work adds prediction-aligned lexical evidence to refine confidence while keeping the classifier’s intent fixed.

I. INTRODUCTION

  • Selective routing trades accepted-request coverage against errors among accepted predictions, with the remaining requests deferred.Intent labels select downstream handlers, making confidence relevant to whether an assistant acts on an interpretation.Source: §I · p. 1
  • Sparse word and character features can provide label-specific lexical cues alongside sentence representations, but the relevant evidence is support for the classifier-selected intent relative to competitors.Common confidence scores such as maximum class probability and logit margin summarize the base classifier’s preferences.Source: §I · p. 1
  • The gate combines semantic margin with signed lexical support to estimate correctness, then independently calibrates an acceptance threshold while preserving the base prediction.Lexical evidence is positive when it favors the selected intent and negative when it favors a competing intent; strong semantic evidence can outweigh modest disagreement.Source: §I · p. 1
  • The central contribution is a compact confidence gate using prediction-aligned lexical evidence to improve selective routing for a fixed intent classifier.The supporting contribution is a feature distinguishing support for the selected intent from evidence favoring a competing intent, without retraining or relabeling the base classifier.Source: §I · p. 1
  • The evaluation retains identical semantic predictions across the proposed method and its principal comparators, isolating confidence estimation.It also examines calibration-budget and input-distribution sensitivity.Source: §I · p. 1

II. RELATED WORK

Prior work motivates confidence signals beyond a model’s native output and methods for converting confidence into selective-risk policies. This study uses complementary sparse lexical evidence and an established risk-calibration procedure for fixed intent classification.

II. RELATED WORK

  • The evaluation studies selective reliability with a frozen MiniLM sentence encoder and trained linear head, holding the predictor fixed to isolate confidence estimation.The cited benchmark context includes CLINC150 out-of-scope requests and HWU conversational-agent domains.Source: §II · pp. 1–2
  • The study adds a sparse discriminative representation summarized by a score aligned with the base model’s selected label, and evaluates it for fixed intent classification.Unsigned-confidence and agreement controls do not separately identify the effects of sign, label alignment, and magnitude.Source: §II · p. 2
  • An established LTT-style procedure converts confidence ranking into an accept-or-defer policy for binary intent errors, separating signal quality from calibration procedure.The study compares it with a fixed-sequence alternative.Source: §II · p. 2

III. SIGNED LEXICAL GATING

The method holds the sentence classifier’s predicted intent fixed and evaluates lexical evidence relative to that label. A positive or negative signed lexical margin captures whether the witness favors the selected intent or a competitor.

A. Prediction and witness evidence

  • A frozen sentence embedding feeds a multinomial logistic head that produces logits and the predicted intent.The embedding is normalized and has 384 dimensions.Source: A. Prediction and witness evidence · p. 2
  • A separate linear support-vector classifier uses concatenated word and character TF-IDF features to produce lexical decision scores.Its evidence is evaluated against the sentence head’s predicted label.Source: A. Prediction and witness evidence · p. 2
  • The lexical index is the sentence head’s prediction, so lexical preference for a competing intent becomes negative evidence rather than an unsigned confidence value.The sentence and lexical models are fit on the same model-training partition and provide complementary views of each request.Source: A. Prediction and witness evidence · p. 2

B. Two-feature correctness gate

The correctness gate combines standardized features on a disjoint fitting partition, then uses an independently calibrated threshold to accept or defer predictions. Its simultaneous binomial bounds support selecting the highest-coverage candidate under a risk target, conditional on the stated IID assumption.

B. Two-feature correctness gate

  • A two-feature logistic gate is fitted on a disjoint partition using the semantic and lexical scores with correctness as its target.Feature-wise standardization uses that partition’s means and standard deviations.Source: B. Two-feature correctness gate · p. 2
  • The router accepts the base classifier’s label when its gate score clears the threshold; otherwise, it defers.The score retains the magnitude of lexical support rather than reducing it to a hard agreement decision.Source: B. Two-feature correctness gate · p. 2

C. Independent risk calibration

  • Threshold candidates come from 50 development-score quantiles plus an accept-all rule, with coincident thresholds deduplicated before calibration labels are accessed.The quantile grid runs from 0 to 0.98 in steps of 0.02.Source: C. Independent risk calibration · pp. 2–3
  • For each candidate, a one-sided Clopper–Pearson upper bound is compared with the target; the qualifying candidate with the most accepted requests is selected, or the router defers all.The procedure uses a confidence-failure budget across the candidate rules.Source: C. Independent risk calibration · p. 3
  • The guarantee is conditional on independently fitted models and an IID calibration/future-request assumption; it bounds population selective error, not finite-test-set error.Exchangeability alone does not justify the binomial model.Source: C. Independent risk calibration · p. 3

A. Data and experimental protocol

The evaluation uses the original BANKING77 and CLINC150 releases and a first-fold HWU64 subset, with ten shared-partition runs per dataset. The primary analysis calibrates on the training pool and evaluates on official test sets.

A. Data and experimental protocol

  • The study uses original BANKING77 and CLINC150 releases and the first HWU cross-validation fold, with duplicate training texts removed before splitting.Seven BANKING77 and two CLINC150 test requests match normalized training texts; a sensitivity analysis excludes overlaps and repeated test texts.Source: A. Data and experimental protocol · p. 3
  • Each of ten seeds divides the training pool into model fitting (60%), gate fitting (10%), threshold-grid development (10%), and calibration (20%) partitions shared across methods.Calibration sizes are 2,000 for BANKING77, 3,000 for CLINC150, and 1,992 for HWU64.Source: A. Data and experimental protocol · p. 3
  • The primary analysis uses training-pool calibration and official-test evaluation; target-matched and other sensitivity analyses were added after the design freeze.Those analyses reuse frozen predictors and gates where applicable and are not fresh unseen-benchmark confirmation.Source: A. Data and experimental protocol · p. 3

B. Comparators and measurements

The evaluation compares fixed-prediction confidence gates and matched controls, measuring ranking with AURC and coverage under calibrated risk targets. The signed gate lowers AURC on all three datasets and increases coverage at the 5% target on BANKING77 and HWU64.

B. Comparators and measurements

  • Fixed-prediction comparators include semantic confidence scores, Trust Score, hard agreement, and gates augmented with agreement or signed lexical support.The additional matched controls replace signed support with the lexical model’s unsigned top-two margin or an agreement bit.Source: B. Comparators and measurements · p. 3
  • The primary endpoint is coverage under source calibration at α = 0.05 and δ = 0.05 versus the semantic gate; AURC measures ranking without selecting a calibration threshold.Intervals are pointwise and conditional on the benchmark and fitted runs, with test examples shared across seeds.Source: B. Comparators and measurements · p. 4

A. Error ranking and calibrated coverage

  • AURC falls by 15.8% on BANKING77, 15.1% on CLINC150, and 11.8% on HWU64 versus the semantic gate.Lower AURC indicates improved average error ranking across coverage levels, but does not establish lower risk at every coverage level.Source: A. Error ranking and calibrated coverage · p. 4
  • At the 5% target, mean coverage rises from 87.80% to 89.63% on BANKING77 and from 79.16% to 84.30% on HWU64.The paired gains are 1.83 and 5.14 percentage points, respectively.Source: A. Error ranking and calibrated coverage · p. 4
  • CLINC150 is a complementary saturated regime: most methods already approach full acceptance at the 5% target.The signed gate still yields a 15.1% AURC reduction in this setting.Source: A. Error ranking and calibrated coverage · p. 4

B. Why signed evidence matters

Signed lexical support improves error ranking over the tested unsigned feature, while gains over binary agreement depend on dataset and metric. At a 2% target, simultaneous binomial calibration yields a nonempty policy in every dataset–seed run at the available budgets.

B. Why signed evidence matters

  • Signed support lowers AURC versus the unsigned gate on all three datasets, while gains over binary agreement vary by dataset and metric.Unsigned versus signed AURC is 12.11/5.03/15.13 versus 10.44/4.51/13.64; signed-minus-agreement coverage gains are 0.36 points on BANKING77 and 2.37 on HWU64, with the BANKING77 interval crossing zero.Source: B. Why signed evidence matters · p. 4, Table III · p. 5
  • Lexical support alone gives AURC 12.40/6.90/16.87, supporting combination with semantic margin; the seven-feature gate changes coverage by less than 0.4 points in either direction, showing compact construction captures most benefit.Standardized signed-support coefficients are positive in all 30 fitted gates, ranging from 0.89 to 1.71.Source: B. Why signed evidence matters · p. 4
  • At the primary target, the continuous gate accepts some lexical-disagreement requests rather than imposing a categorical veto.These requests are 2.49% of BANKING77 and 2.04% of HWU64 requests, with mean errors of 14.33% and 6.50%, respectively.Source: B. Why signed evidence matters · p. 4

C. Strict targets and calibration budgets

  • At a 2% target, signed-gate mean source-calibrated coverage is 72.77%, 92.93%, and 62.94%, with a nonempty policy in all 30 dataset–seed runs.The semantic gate defers every request in one BANKING77 run and three HWU64 runs; with M = 51, α = 0.02, and δ = 0.05, zero observed errors still require 343 accepted calibration requests.Source: C. Strict targets and calibration… · p. 4, Table IV · p. 5
  • At a 2% target on HWU64, fixed-sequence testing gives 69.15% signed-gate coverage versus 61.25% for the semantic gate, narrowing the Bonferroni coverage gap from 28.14 to 7.91 points.This comparison holds models, gates, and independent grids fixed while varying the calibration procedure.Source: C. Strict targets and calibration… · p. 5

D. Calibration transfer and input shift

The target-matched analysis retains the signed gate’s coverage advantage, but its setup changes calibration size and evaluation rows as well as calibration source. Stress tests show substantial errors under orthographic perturbations and unsupported intents; recovery through recalibration remains untested.

D. Calibration transfer and input shift

  • In the target-matched setting, signed-minus-semantic coverage gains are 3.92, 1.26, and 4.78 points on BANKING77, CLINC150, and HWU64.Calibration and evaluation each use half of the official test set, with seed offsets fixed at 7000; predictors, gates, and training-derived grids remain unchanged.Source: D. Calibration transfer and input shift · pp. 5–6
  • The target-matched comparison cannot isolate the source effect because calibration budget and evaluation rows also change.For CLINC150, calibration uses 2,250 rather than 3,000 requests and evaluation uses 2,250 rather than 4,500; an equal-budget comparison on identical rows would isolate source matching more directly.Source: D. Calibration transfer and input shift · p. 6
  • Without recalibration, orthographic perturbations raise signed-gate error to 6.04%, 19.35%, and 7.05%; adding weighted CLINC out-of-scope requests raises it to 14.11%.Recalibration’s ability to restore useful coverage under either shift was not tested, and the experiments do not establish out-of-scope detection or downstream agent safety.Source: D. Calibration transfer and input shift · p. 6

E. Reproducibility

The reproducibility artifact includes the locked environment and materials needed to reproduce the operating points and acceptance masks. The discussion presents prediction-aligned lexical support as a compact complement to semantic confidence while noting unresolved controls and calibration limits.

E. Reproducibility

  • The reproducibility artifact provides the locked environment, pinned data and model revisions, split indices, per-request outputs, calibration scores, and supplementary results.It includes an independent exact-binomial implementation reproducing all 1,800 operating points and acceptance masks.Source: E. Reproducibility · p. 6

VI. DISCUSSION

  • The discussion characterizes prediction-aligned lexical support as a useful complement that turns agreement and disagreement into graded evidence for the gate.Improvements over the tested unsigned feature occur across all three datasets, while gains beyond binary agreement depend on dataset and metric.Source: §VI · p. 6
  • The existing controls do not separately identify the effects of sign, alignment, and magnitude; the paper proposes additional controls to distinguish them.The proposed controls are an absolute-aligned-margin control and a joint agreement-plus-unsigned-confidence gate.Source: §VI · p. 6
  • The discussion separates score-ranking quality from calibration, while noting that recovery through recalibration under distribution shift remains untested.Changing calibration source, budget, and evaluation rows also prevents isolating the source effect.Source: §VI · p. 6

A. Scope and future opportunities

The signed lexical gate improves error ranking across the evaluated datasets and increases accepted coverage where the semantic baseline has room to improve. The evaluation and risk guarantee remain bounded to a specific benchmark and calibration setting.

A. Scope and future opportunities

  • The evaluation covers English single-intent benchmarks with one frozen sentence encoder, using shared predictions and matched comparisons across confidence methods.Source: A. Scope and future opportunities · pp. 6–7
  • The aggregate risk guarantee assumes conditional IID calibration and deployment requests and does not cover per-intent or lexical-disagreement budgets or downstream execution success.Source: A. Scope and future opportunities · p. 7
  • Further work can connect these confidence policies to complete assistant workflows, deployment-specific calibration streams, out-of-scope detection, and downstream recovery.Source: A. Scope and future opportunities · p. 7

VII. CONCLUSION

  • Across three datasets, the signed gate improves error ranking over a learned semantic gate and raises accepted coverage where the semantic baseline has headroom.Source: §VII · p. 7

Related papers

Loading 2610.00262v1…