Source-linked AI summary
When Do LLMs Replace Fine-Tuned NLU? A Decision Framework for Intent Detection in Production Conversational Systems
Carson Rodrigues, Oysturn Vas
TL;DR
The paper asks when zero-shot LLMs can replace fine-tuned intent classifiers in production and evaluates that question across datasets and deployment conditions. Head-to-head experiments show that the answer depends on the intent space: fine-tuning wins on stable, data-rich domains, while LLMs provide advantages for OOS detection, noisy ASR, and dynamic schemas. The resulting decision framework helps practitioners choose between them.
Problem
The paper addresses whether zero-shot LLMs can replace fine-tuned NLU classifiers when production deployments have varying intent schemas and little or no labeled data.
Method
The study compares four intent-detection approaches on full ATIS and CLINC150, using bootstrap confidence intervals, paired significance tests, and production stress tests.
Results
The results are conditional: fine-tuned RoBERTa wins on stable, data-rich domains, while LLMs match broad-schema accuracy and outperform on OOS detection, noisy ASR, and unseen schemas.
Takeaways & Limitations
Practitioners should select the tool according to intent-space stability, label availability, OOS requirements, ASR noise, and whether deployments introduce new schemas.
Takeaways & Limitations
The study evaluates one LLM and one encoder, uses controlled TTS with additive noise rather than a fully natural spoken corpus, and tests cleanly disjoint schemas.
Abstract
from arXiv · showhide
A common claim is that zero-shot large language models (LLMs) can replace fine-tuned NLU classifiers for intent detection. We test this claim head-to-head and find that the honest answer is: it depends on the intent space. On full ATIS and CLINC150 we compare a fine-tuned RoBERTa, a TF-IDF+logistic-regression baseline, sentence-embedding kNN, and Claude Haiku zero-shot, reporting bootstrap 95% confidence intervals and paired significance tests. When abundant in-domain labels exist, fine-tuned RoBERTa is as good or better and three orders of magnitude cheaper and faster: on ATIS it beats Claude zero-shot by 11.8 points (95.9 vs. 84.1, p<0.001). On the broad 150-intent CLINC150 schema the two are statistically tied (89.1 vs. 88.5, p=0.24): the LLM matches a fully supervised model with no training data. The LLM's advantages appear in three production-relevant regimes: out-of-scope detection (OOS recall 85.6 vs. 58.1 for RoBERTa); robustness to realistic ASR noise via a controlled text-to-speech to noise to Whisper pipeline (92.5 vs. 80.0 at 0 dB); and dynamic per-deployment schemas, where a classifier trained on one app's intents scores 0% on a new app's intents while the schema-prompted LLM serves both at ~94% with zero retraining. We distill these findings into a decision framework for practitioners.
1 Introduction
The paper reframes LLM replacement as a conditional production decision: performance depends on the intent space and deployment requirements. It compares supervised and zero-shot approaches across benchmarks and production-relevant stress tests.
- Motivation: The study asks when fine-tuned NLU or zero-shot LLMs win across accuracy, latency, cost, robustness, and adaptability.The motivating setting is customer-specific intent schemas with little or no labeled data.
- Evaluation: The evaluation head-to-head compares fine-tuned RoBERTa, TF-IDF+LR, sentence-embedding kNN, and Claude Haiku zero-shot on full ATIS and CLINC150.It reports bootstrap 95% confidence intervals and paired significance tests rather than single-run point estimates.
- Production stress tests: The stress tests target out-of-scope detection, realistic WER-stratified ASR robustness, and dynamic schemas on previously unseen applications.These tests isolate production conditions where an LLM may provide value beyond closed-set accuracy.
- Decision framework: The paper contributes a decision framework mapping intent-space properties to the appropriate tool, including cases where fine-tuned models are inapplicable.The framework is intended to distinguish conditional strengths rather than assume universal replacement.
- Headline finding: The headline finding is conditional: LLMs are significantly worse and more expensive on stable, data-rich narrow domains but valuable in open, dynamic, or label-scarce settings.Out-of-scope handling is another regime where the paper identifies an LLM advantage.
2 Related Work
Prior work established fine-tuned encoders as strong intent classifiers while showing that LLMs can compete zero-shot. This paper extends that literature to OOS detection, dynamic schemas, and ASR robustness.
- Fine-tuned NLU: BERT-based intent classifiers remain state of the art on ATIS and SNIPS after replacing earlier bag-of-words systems.CLINC150 added an explicit out-of-scope class to isolate a recognized weakness of closed-set classifiers.
- LLMs for intent detection: Instruction-tuned LLMs established strong zero-shot and few-shot classification, with prior comparisons finding zero-shot performance competitive but below fine-tuning in full-data regimes.Related work also links OOS quality to label-space size and scope.
- Dynamic schemas: Schema-guided dialogue supplies natural-language intent schemas at inference, matching the per-application setting studied here.The paper operationalizes this paradigm as a disjoint-schema transfer test.
- Latency and ASR: Prior latency and ASR work frames sub-second response as conversationally important and uses synthetic speech, graded noise, and Whisper for robustness evaluation.SLURP is discussed as the real-audio alternative.
3 Datasets and Methods
The paper evaluates four intent-detection approaches on full ATIS and CLINC150, using uncertainty-aware statistical comparisons and two controlled production stress tests. It also constructs a disjoint dynamic-schema transfer setting.
- Datasets: ATIS contains 4,978 training and 893 test examples across 26 intents, while CLINC150 contains 15,250 training and 5,500 test examples across 150 intents plus 1,000 OOS items.ATIS represents a narrow, stable, data-rich regime; CLINC150 represents a broad schema with an explicit OOS axis.
- Models: The systems are TF-IDF+LR, sentence-embedding kNN, fine-tuned RoBERTa, and zero-shot Claude Haiku prompted with intent labels and JSON output.A 5-shot 1-NN variant is included as a low-data deployment competitor.
- Evaluation protocol: Every accuracy includes a percentile bootstrap 95% CI from 10,000 resamples, with paired bootstrap and McNemar tests for identical test items.Macro-F1 accompanies accuracy because ATIS is highly imbalanced.
- OOS evaluation: CLINC150 evaluation reports in-scope accuracy, OOS recall, precision, and F1, treating OOS as the positive class.The LLM may answer OOS, while classifiers learn the class from training data.
- Dynamic schemas: For dynamic schemas, CLINC150 intents are split into disjoint 75-intent App A and App B schemas, with RoBERTa trained only on App A and the LLM given each app’s labels at inference.This tests transfer to a previously unseen application schema.
- ASR robustness: The ASR pipeline synthesizes 120 CLINC utterances, adds white noise at five SNR levels, transcribes with Whisper-base, stratifies by measured WER, and classifies the transcripts.This preserves known ground truth while modeling noisy speech inputs.
4 Results
Results depend on the deployment regime: fine-tuning wins on narrow, data-rich accuracy and operational efficiency, while LLMs excel at OOS detection, noisy ASR, and unseen schemas. The dynamic-schema result is categorical because the locked classifier cannot emit unseen labels.
- In-domain accuracy: 11.8 points: RoBERTa beats Claude on ATIS accuracy, scoring 95.9 versus 84.1 with p = 2 × 10^-4.The paired-bootstrap confidence interval for the gap is [9.2, 14.4].
- In-domain accuracy: 0.6 points: RoBERTa and Claude differ on CLINC150 accuracy, 89.1 versus 88.5, with p = 0.24.The result is statistically indistinguishable, and Claude reaches it with zero training data.
- Out-of-scope detection: 85.6 OOS recall: Claude exceeds RoBERTa’s 58.1 and TF-IDF’s 36.4 at comparable in-scope accuracy.Claude’s OOS F1 is 85.0, versus 73.0 for RoBERTa and 51.7 for TF-IDF.
- Dynamic schemas: 0%: locked RoBERTa scores on unseen App B intents, while the schema-driven LLM serves both apps at approximately 94% without retraining.RoBERTa scores 96.8 on its own App A schema, but its head cannot emit labels absent from training.
- ASR robustness: 12.5 points: at 0 dB and mean WER 28.9%, Claude scores 92.5 while TF-IDF scores 80.0.As SNR falls, the lexical classifier degrades faster than the LLM.
- Latency and cost: 981 ms median latency: Claude exceeds the 500 ms conversational bar, whereas on-device classifiers respond in microseconds to milliseconds at zero marginal cost.The LLM costs $0.25 per 1K calls at the observed token counts.
5 A Decision Framework
The decision framework recommends choosing between fine-tuned classifiers and LLMs according to schema breadth, label availability, out-of-scope requirements, ASR noise, latency, and cost. Fine-tuned models suit stable, data-rich settings, while LLMs are favored for dynamic schemas, OOS handling, and noisy ASR.
- Stable schema, abundant labels: Stable schemas with abundant labels favor fine-tuned encoders because they match or beat LLMs at lower latency and cost.The framework specifically recommends fine-tuned models for mature single-domain assistants.
- Broad schema, moderate labels: Broad schemas with moderate labels support either approach, with the LLM avoiding a training pipeline while latency and cost determine the choice.The framework describes CLINC150 as a parity regime.
- Dynamic schemas: Dynamic per-deployment schemas with little or no labeled data favor a schema-prompted LLM because a locked classifier cannot emit unseen labels.The recommended LLM serves new schemas without retraining, whereas the classifier requires fresh annotation and training.
- Out-of-scope handling: Out-of-scope handling favors LLMs, which achieve higher OOS recall than RoBERTa on CLINC150.The framework also permits LLM-based OOS routing on top of a fast classifier.
- Noisy ASR front-end: Noisy ASR favors LLMs, with a hybrid fallback recommended when classifier confidence is low or WER is high.The cited framework describes the LLM as degrading more gracefully under acoustic noise.
6 Limitations
The evaluation is limited to English ATIS and CLINC150, excludes multilingual and multi-intent or multi-turn settings, and uses controlled synthetic-noise ASR conditions. It also evaluates one LLM and one encoder, while few-shot prompting and overlapping production schemas remain scope boundaries.
- Scope: The study evaluates English ATIS and CLINC150, leaving multilingual and multi-intent or multi-turn settings out of scope.These are explicit evaluation boundaries.
- ASR evaluation: The ASR study uses controlled TTS with additive noise rather than a fully natural spoken corpus, although transcripts come from a real ASR model and are WER-stratified.The authors identify the synthetic-noise setup as a limitation while retaining measured-WER evaluation.
- Model and prompting scope: The reported comparison covers zero-shot prompting, one LLM, and one encoder; few-shot prompting and larger models could change absolute results.The paper states that few-shot prompting is a natural extension and larger models may shift absolute numbers.
- Dynamic schemas: The dynamic-schema test uses disjoint intent partitions, whereas production schemas may overlap and soften but not eliminate the locked-classifier ceiling.This bounds how directly the transfer result maps to deployments with overlapping schemas.
7 Ethical Considerations
The study uses public benchmarks and synthesized speech, without human subjects or proprietary user data, while withholding customer-identifying and proprietary system details.
- The experiments use public ATIS and CLINC150 benchmarks plus synthesized speech, with no human subjects or proprietary user data released.
8 Conclusion
The paper concludes that intent detection requires choosing between fine-tuned NLU and LLMs based on the intent space and deployment conditions. It releases a decision framework and evaluation harness to support deliberate tool selection.
- Fine-tuned models win on accuracy, latency, and cost when labels are abundant and the schema is stable.
- LLMs are favored when intent spaces are dynamic or label-scarce, out-of-scope detection matters, or ASR noise is heavy.
- A fine-tuned classifier cannot operate when each deployment introduces a new intent schema.
- The authors release a decision framework and full evaluation harness so practitioners can choose tools deliberately rather than by default.