Source-linked AI summary
AutoIntent: AutoML for Text Classification
Ilya Alekseev, Roman Solomatin, Darina Rustamova, Denis Kuznetsov
TL;DR
Existing NLP AutoML frameworks often lack comprehensive support for intent-classification needs such as embedding selection, multi-label classification, and out-of-scope detection. AutoIntent automates embedding, classifier, and decision-module selection, and its evaluations report competitive performance and a favorable quality-cost balance alongside support for OOS and few-shot settings.
Problem
Method
AutoIntent uses an end-to-end modular pipeline that optimizes embedding, scoring, and decision modules and supports multi-label classification and OOS detection.
Source: §1 · p. 1, §3.1 · p. 3, §5 · p. 6Results
AutoIntent and AutoGluon are highly competitive across five intent-classification datasets, while AutoIntent offers the most affordable quality-computational-cost balance; it also demonstrates superiority on OOS detection and few-shot scenarios.
Source: §4.1 · p. 5, §4.2 · p. 5, §4.3 · p. 6Takeaways & Limitations
AutoIntent’s sklearn-like interface and end-to-end automation are intended to make intent classification more accessible while supporting multi-label and OOS tasks.
Source: §1 · p. 1, §5 · p. 6The evaluation focuses on intent-classification datasets and includes no user studies.
Source: §5 · p. 6
Abstract
from arXiv · showhide
AutoIntent is an automated machine learning tool for text classification tasks. Unlike existing solutions, AutoIntent offers end-to-end automation with embedding model selection, classifier optimization, and decision threshold tuning, all within a modular, sklearn-like interface. The framework is designed to support multi-label classification and out-of-scope detection. AutoIntent demonstrates superior performance compared to existing AutoML tools on standard intent classification datasets and enables users to balance effectiveness and resource consumption.
1 Introduction
Text classification with transformer embeddings requires choices about embeddings, classifiers, and hyperparameters, but existing NLP AutoML often lacks comprehensive support for these decisions and confidence thresholds. AutoIntent introduces an automated intent-classification framework that supports multi-label classification and out-of-scope detection.
1 Introduction
- Transformer embeddings provide contextual text representations, but classification still requires choosing an embedding model, classification algorithm, and hyperparameters.The paper frames these choices as considerations for effectively using transformer-based embedding models.Source: §1 · p. 1
- Existing NLP AutoML often lacks comprehensive embedding-model selection and confidence-threshold tuning for multi-label classification and out-of-scope detection.Traditional approaches require manual tuning of components, which can be time-consuming and require significant expertise.Source: §1 · p. 1
- AutoIntent provides an sklearn-like interface for intent classification, including support for multi-label classification and out-of-scope detection.The paper presents these capabilities as bridging AutoML and conversation systems.Source: §1 · p. 1
2 Background
The background describes AutoML as automating routine model-development tasks and distinguishes tabular and NLP AutoML by their typical capabilities. It also outlines common approaches to model selection and presents AutoIntent’s hierarchical optimization in Figure 2.
2 Background
- AutoML automates routines such as validation data splitting, hyperparameter tuning, ensembling, and model selection, and is relevant when non-experts need to solve machine-learning tasks.The passage also connects AutoML with no-code machine learning, sometimes called “ML as a service.”Source: §2 · p. 1
- Tabular AutoML commonly focuses on feature engineering, feature selection, and model ensembling, often using classical models and budget-aware tuning.The passage notes that some open-source solutions produce large ensembles that can be difficult to transfer into production-ready systems.Source: §2 · p. 2
- NLP AutoML frameworks natively support text inputs and often use transformer backbones, while primarily targeting classification and regression rather than generation or named entity recognition.Native text support removes the need for manual tokenization and vectorization.Source: §2 · p. 2
- Model selection may use preset hyperparameter tuning, empirically chosen generic hyperparameters, or meta-learning that predicts hyperparameters from a dataset.Preset choices can vary in their time and computational-resource requirements.Source: §2 · p. 2
- Figure 2 depicts sequential module-level optimization, comparisons among classification approaches, and individual hyperparameter tuning, alongside the resulting inference pipeline.Its caption identifies Optuna samplers for instance-level tuning.Source: Figure 2 · p. 3
3 AutoIntent
AutoIntent combines modular text-classification components with hierarchical optimization of embeddings, scoring models, and decision strategies. Its design spans embedding-selection options, diverse classifiers, thresholding methods, and pipeline safeguards for tuning and training.
3.1 Design Principles
- AutoIntent’s modular design supports deep-learning and classical models on precomputed transformer embeddings, alongside flexible optimization, multi-label classification, OOS detection, and few-shot learning.The embedding-centric design uses the Hugging Face model repository and avoids complex feature engineering.Source: §3.1 · p. 3
3.2 Separation of Concerns
- Separating scoring from decision-making allows class probabilities from one scoring model to be reused with different threshold strategies without recomputation.The scoring module outputs class probabilities; the decision module applies thresholds to make final predictions.Source: §3.2 · p. 3
3.3 Embedding Module
- Embedding selection can use pipeline-level optimization, scoring-level optimization for each candidate model, or a user-specified fixed embedding.Pipeline-level selection uses retrieval metrics or a simple downstream classifier; scoring-level optimization is more computationally intensive but may yield better performance.Source: §3.3 · pp. 3–4
3.4 Scoring Module
- Scoring models include nearest-neighbor methods, BERT-based classifiers, sklearn classifiers on embeddings, and zero-shot approaches using class descriptions or LLM prompting.The available BERT-based approaches include full fine-tuning, LoRA, and P-Tuning.Source: §3.4 · p. 4
3.5 Decision Module
- The decision module converts class scores into predictions using strategies for sample-specific, universal, fixed, or automatically tuned thresholds.JinoosDecision balances in-domain and out-of-scope accuracy; TunableDecision uses Optuna to maximize F1 score.Source: §3.5 · pp. 4–5
3.6 AutoML Pipeline
- Hierarchical optimization sequentially selects embedding, scoring, and decision modules, then samples models and their hyperparameters while separating tuning from model training.The greedy pipeline builds each module on the best preceding module, and separate data subsets plus out-of-fold predictions address validation and target-leakage concerns.Source: §3.6 · p. 5
4 Experiments
Across five intent-classification datasets, AutoIntent and AutoGluon were highly competitive, while H2O was moderate and LAMA failed; AutoIntent also performed strongly in OOS and scarce-data evaluations. The experiments identify baseline limitations and describe the settings and methods used in those evaluations.
4.1 Baselines
- AutoIntent and AutoGluon were highly competitive across five intent-classification datasets, H2O achieved moderate performance, and LAMA failed on these tasks.AutoIntent offered the most affordable balance between quality and computational cost.Source: §4.1 · p. 5
- Baseline limitations included AutoGluon’s fixed training recipe, H2O’s lack of native text features, and LAMA’s support for only three predefined transformer models.AutoGluon’s HPO presets were considered too time- and disk-space-consuming to test; unlike its fixed fine-tuned transformer output, AutoIntent could select lighter models for comparable performance and offered both ML and DL models.Source: §4.1 · p. 5
4.2 OOS Detection
- AutoIntent’s dedicated confidence-threshold tuning was associated with superior OOS results on CLINC150, where baselines treated OOS as an additional class.The evaluation used in-domain accuracy for the balanced dataset and F1-score for the OOS class.Source: §4.2 · p. 5
4.3 Few-shot Scenario
- With 4–128 shots per class, AutoIntent showed robustness and superiority over AutoGluon medium_quality and H2O word2vec in the scarce-data comparison.The comparison also included AutoIntent preset classic-light; neighbor-based classification methods were cited as the reason for its performance.Source: §4.3 · p. 6, Figure 3 · p. 6
5 Conclusion
AutoIntent is presented as addressing gaps in intent-classification AutoML through end-to-end automation and an optimization strategy with an embedding-centric design. Its evaluation focused on intent-classification datasets and included no user studies.
5 Conclusion
- AutoIntent targets gaps in intent-classification AutoML, including embedding-model selection, multi-label classification, OOS detection, and few-shot learning.The conclusion describes end-to-end automation as a way to democratize intent classification.Source: §5 · p. 6
- The framework uses a three-stage embedding, scoring, and decision pipeline with hierarchical optimization using Optuna.Its stated target users include NLP practitioners, conversational AI developers, and ML-as-a-service platforms.Source: §5 · p. 6
- The system was evaluated on five intent-classification datasets, with no user studies conducted.It was released under the Apache-2.0 license to encourage community adoption.Source: §5 · p. 6
A Computational Efficiency
Scoring methods varied in computational resource use, with KNN-based methods most efficient, logistic regression a middle-ground trade-off, and BERT-based methods most demanding.
A Computational Efficiency
- The resource analysis used Code Carbon and reported Table 4 results for different scoring modules.A single trial comprised training and evaluation of one model configuration.Source: A Computational Efficiency · p. 9
- KNN-based methods had minimal emissions and runtime, logistic regression showed moderate resource use with high performance, and BERT-based methods required the most resources.The analysis measured resource consumption for a single trial on a model configuration.Source: A Computational Efficiency · p. 9
- The reported resource-efficiency differences across scoring methods inform deployment scenarios with varying resource constraints and the construction of AutoIntent presets.Source: A Computational Efficiency · p. 9
A.1 Embedding Module Effectiveness
The embedding-selection heuristic’s approximate NDCG ranking was imperfect but identified the best model, stella_en_400M_v5. The supplied visual descriptions distinguish this ranking from full-pipeline ranking and identify the datasets covered by the embedding-performance table.
A.1 Embedding Module Effectiveness
- The NDCG-based approximate ranking was imperfect but identified stella_en_400M_v5 as the best model against full-pipeline accuracy.The authors state that the heuristic balances computational cost and selection quality.Source: §A.1 · p. 9
- Table 5 reports embedding-model performance averaged over hwu64, massive, minds14, and snips.The candidate models were selected from the MTEB English leaderboard.Source: §A.1 · p. 9, Table 5 · p. 10
- Figure 4 contrasts precise rankings from the full AutoML pipeline for one model at a time with approximate rankings based on retrieval-quality NDCG.Source: Figure 4 · p. 10