Source-linked AI summary
OpenAutoNLU: Open Source AutoML Library for NLU
Grigory Arshinov, Aleksandr Boriskin, Sergey Senichev, Ayaz Zaripov, Daria Galimzianova, Daniil Karpov, Leonid Sanochkin
TL;DR
NLP AutoML lacks simple, unified interfaces and native integration of data-aware training selection, text-specific quality assessment, and classification–NER support. OpenAutoNLU addresses these gaps with low-code pipelines that select methods from label distributions and integrate diagnostics, OOD detection, and LLM features. It achieves the best or tied performance on three of four datasets, while AutoGluon outperforms it on Banking77 at higher computational cost.
Problem
Existing NLP AutoML systems lack simple unified interfaces and native integration of data-aware training selection, text-specific data-quality assessment, and classification–NER support.
Method
OpenAutoNLU uses low-code automatic pipelines that deterministically select training regimes from label distributions and integrate data-quality diagnostics, OOD detection, NER, and LLM-assisted generation.
Results
OpenAutoNLU achieves the best or tied performance on three of four datasets; AutoGluon is the only competitor to outperform it, on Banking77 only and at considerably higher computational cost.
Takeaways & Limitations
The library provides an all-in-one low-code NLU solution balancing model quality and cost while offering configurable OOD detection.
Takeaways & Limitations
The data-driven deterministic optimization strategy is planned for extension with a metamodel using more abstract dataset features such as dataset2vec.
Abstract
from arXiv · showhide
OpenAutoNLU is an open-source automated machine learning library for natural language understanding (NLU) tasks, covering both text classification and named entity recognition (NER). Unlike existing solutions, we introduce data-aware training regime selection that requires no manual configuration from the user. The library also provides integrated data quality diagnostics, configurable out-of-distribution (OOD) detection, and large language model (LLM) features, all within a minimal lowcode API. The demo app is accessible here https://openautonlu.dev.
1 Introduction
NLU classification and NER are widely used but difficult to deploy because practitioners must choose among data- and resource-sensitive approaches while handling data quality, hyperparameters, and OOD inputs. OpenAutoNLU addresses gaps in ease of use and NLP-specific automation through a unified, low-configuration library.
- NLU classification and NER support applications including intent detection, sentiment analysis, information extraction, and document categorization.
- Practitioners must navigate full transformer fine-tuning, few-shot learning, and classical embedding-based machine learning across different data regimes and resource constraints.
- Deployment is further complicated by data quality issues, hyperparameter sensitivity, and the need for OOD detection in production settings.
- Existing AutoML systems often require substantial configuration and lack a simple unified interface for common NLP tasks.
- OpenAutoNLU integrates automatic training-regime selection, text-specific data-quality assessment, and unified classification and NER support.
2 Background
Prior AutoML and LLM-based approaches provide useful alternatives for NLP, but they differ in task specialization, computational requirements, and handling of text inputs. The paper situates OpenAutoNLU among intent-focused and general-purpose frameworks while emphasizing performance–cost trade-offs.
- Figure 2 compares averaged classification macro F1 performance with training time across four text classification datasets.
- LLMs can provide strong zero- and few-shot text classification performance but typically incur high computational costs, monetary costs, and inference latency.
- AutoIntent targets intent classification with sentence embeddings, classical or neural classifiers, threshold optimization, supervised OOD detection, and multiple quality–cost presets.
- AutoGluon is a general-purpose AutoML framework that relies on ensembling, stacking, and bagging across heterogeneous models.
- LightAutoML and H2O AutoML primarily target tabular data, processing text through Word2Vec-based vectorization and standard tabular models.
3 OpenAutoNLU
OpenAutoNLU combines low-code pipelines with data-aware method selection, integrated diagnostics, configurable OOD detection, and LLM-assisted data generation for classification and NER. Its design adapts training to label availability while supporting evaluation and deployment workflows.
- OpenAutoNLU provides a simple Python API for low-code NLU prototyping and production training through chained automatic pipelines.
- The method resolver deterministically selects training from label distribution, using AncSetFit for 2 ≤ nmin ≤ 5, SetFit for 5 < nmin ≤ 80, and full fine-tuning for nmin > 80.
- OOD variants pair training methods with configurable detectors, while LLM pipelines generate augmentation data, synthetic tests, and domain analyses.
- The pipelines include data loading, processing, optional quality evaluation, method resolution, training, evaluation, and model export.
- OpenAutoNLU supports classification methods, anchor-augmented few-shot learning, transformer fine-tuning, and BIO-tagging-based NER with entity-level evaluation.
- Data-quality evaluators identify potentially problematic or ambiguous samples using confidence, variability, uncertainty, usable information, and retagging signals.
- Adaptive rebalancing upsamples low-resource classes or downsamples overrepresented classes before re-evaluating the selected training regime.
- Generated test-set scores differed from real held-out scores by less than 5 percentage points for up to 80 examples per class.
4 Experiments
The experiments evaluate OpenAutoNLU across intent-classification datasets, data regimes, competing AutoML frameworks, and supervised or unsupervised OOD settings. Results assess both classification quality and OOD detection under controlled representation and evaluation protocols.
- Evaluation covers four intent-classification datasets spanning binary and multi-class labels, low-, medium-, and full-data regimes, with results averaged over three random seeds.
- The study compares F1-macro, in-scope F1, and OOD F1 against AutoIntent, AutoGluon, LightAutoML, and H2O.
- 4.1 OOD Unaware Regime: The OOD-unaware protocol places OOD samples only at test time, so F1-Macro measures in-domain classification under distributional shift.
- 4.1 OOD Unaware Regime: OpenAutoNLU achieves the best or tied performance on three of four datasets; AutoGluon surpasses it only on Banking77 and at considerably higher computational cost.
- 4.1 OOD Unaware Regime: OOD experiments include supervised training with labeled OOD samples and unsupervised training without explicit OOD labels, using semantic-distance categories for OOD analysis.
5 Conclusion
The paper presents OpenAutoNLU as an easy-to-use, low-code library for producing and serving NLU models. It reports a balance between model cost and quality and highlights configurable OOD detection as a distinctive capability.
- OpenAutoNLU is presented as an easy-to-use NLU-model producing and serving library.
- The resulting models are reported to be well-balanced in cost and quality.
- The configurable OOD layer is described as unique among AutoML libraries with OOD support and state of the art among implemented alternatives.
Limitations and Future work
A limitation is that the current data-driven deterministic optimization strategy does not yet use a metamodel to select combined training, augmentation, and OOD choices from abstract dataset features.
- Limitations and Future work: Future work will analyze the deterministic optimization strategy by implementing a metamodel based on abstract dataset features such as dataset2vec.
- Limitations and Future work: The proposed metamodel would select the best combination of training method, augmentation, and OOD method.
- Limitations and Future work: The metamodel is described as future work rather than part of the reported optimization strategy.
A LLM-generated test set evaluation
The appendix evaluates LLM-generated test sets against original test sets using Macro-F1. It reports a reliability boundary based on a five-percentage-point difference and describes the backend as replaceable.
- A LLM-generated test set evaluation: The experiments use GPT-4o-mini as the generative backend, while OpenAutoNLU supports any OpenAI API-compatible endpoint, including locally hosted models.
- A LLM-generated test set evaluation: Table A.1 compares Macro-F1 on original and LLM-generated test sets.
- A LLM-generated test set evaluation: Generated test sets are treated as reliable proxies when the absolute Macro-F1 difference is below 5 percentage points.
- A LLM-generated test set evaluation: The regime boundaries nmin = 5 and nmin = 80 were determined empirically across public English and internal non-English datasets.
C In-domain classification evaluation
OpenAutoNLU achieves the best or tied in-domain classification performance on three of four datasets and remains highly competitive on the fourth. Under clean test conditions, it performs strongly in medium- and full-data regimes.
- OpenAutoNLU achieves the best or tied F1-Macro performance on HWU64, MASSIVE, and SNIPS.
- 0.914 vs. AutoGluon’s 0.920 on Banking77 shows OpenAutoNLU remains highly competitive on the remaining dataset.
- AutoIntent falls short of OpenAutoNLU on all four datasets, with the largest gap on MASSIVE: 0.755 vs. 0.880.
- OpenAutoNLU performs strongly in medium- and full-data regimes under a controlled test condition with no OOD samples.
D OOD detection description
OpenAutoNLU provides joint in-domain classification and OOD detection across datasets and data regimes. Its unsupervised OOD mechanism performs effectively without explicit OOD supervision, while supervised OOD samples help inconsistently.
- OpenAutoNLU’s unsupervised regime achieves the best overall balance between in-domain and OOD detection performance on Banking77, HWU64, and MASSIVE.
- Supervised OOD training yields the highest OOD F1 on SNIPS at 0.782 but reduces scores on the remaining datasets.
- AutoIntent requires supervised OOD samples to produce predictions and trails OpenAutoNLU by up to 0.304 OOD F1 on Banking77 and 0.144 on MASSIVE.
- OpenAutoNLU provides a more robust and consistent solution for joint in-domain classification and OOD detection across evaluated datasets and regimes.
E Sampling Implementation Details
The evaluation uses four English intent datasets with consistent in-distribution/OOD sampling and evaluation protocols. OOD tests include held-out, cross-domain, and synthetic gibberish examples, with experiments run on specified CPU/GPU hardware.
- Experiments evaluate four English intent-classification datasets using the same in-distribution versus OOD setup and evaluation protocol.
- Only classes meeting nmin are retained; 80% are sampled as in-distribution, with each selected class split 90% for training and 10% for testing.
- The test set includes mid OOD from held-out classes, far OOD from a semantically related but distributionally distinct dataset, and very far OOD synthetic gibberish.
- Far OOD sources are paired separately with each primary intent benchmark.
- Training-time measurements use one machine with a 64-core Intel Xeon processor, an NVIDIA H100 with 80 GB VRAM, and 756 GB RAM.