Source-linked AI summary
DeepOpen: Decision-Oriented Intent Classification on BANKING77 and CLINC150
Shuang Song, Muhammad Luqman Naseem, Zhongqiao Yuan
TL;DR
DeepOpen addresses fixed-label intent classification on BANKING77 and CLINC150 by adapting Laya’s candidate interface and using task-specific systems. It reports 94.1234% BANKING77 accuracy and 98.0222% in-scope CLINC150 accuracy, but the matched-path BANKING77 comparison does not establish a reliable gain.
Problem
Laya’s flexible candidate interface creates an input-representation challenge for fixed-label benchmarks, where serializing candidate descriptions can compete with utterance text for encoder tokens.
Source: §1 · p. 1Method
DeepOpen replaces candidate scoring with fixed-label classification heads and evaluates SAM continuation for BANKING77 separately from probability fusion for CLINC150.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The report documents task-specific benchmark results, but stronger causal, efficiency, or generalization claims require additional controlled evidence.
Source: §6 · p. 7The report covers two English benchmarks; it does not establish multilingual generalization, out-of-scope performance, robustness, or deployment performance.
Source: §5 · p. 7
Abstract
from arXiv · showhide
We present DeepOpen, a decision-Oriented intent-classification framework with dataset- specific systems for BANKING77 and CLINC150. On BANKING77, a single English classi- fier continued with sharpness-aware minimization (SAM) achieves 94.1234% test accuracy (2,899/3,080). This is the best observed result among three fixed SAM seeds, selected for display after test evaluation; their mean is 94.0368%. On the 150 in-scope CLINC150 classes, probability fusion of one Laya specialist and three DeBERTa-v3-large classifiers achieves 98.0222% (4,411/4,500). We describe the two configurations, reproduce the reference com- parisons displayed in the accompanying evaluation explorer, and separate headline scores from evidence about improvement. Under a uniform BANKING77 evaluation path, the SAM mean exceeds the historical checkpoint by 0.0433 percentage points, with a shared-row 95% interval of [-0.2273, +0.3247]. Thus, the continuation does not establish a reliable accuracy gain. The report documents strong observed task-specific performance, selection effects, and unresolved historical execution-path differences, rather than a new backbone or a unified algorithmic advantage.
1 Introduction
DeepOpen adapts Laya’s flexible candidate interface for fixed-label intent classification and investigates separate task-specific optimization and fusion strategies. It reports strong observed benchmark accuracies, while noting that the BANKING77 mean gain remains inconclusive and CLINC150 fusion uses additional inference resources.
Fixed-label adaptation
- DeepOpen replaces Laya’s flexible candidate-scoring interface with task-specific classification heads that represent every class explicitly.This fixed-label adaptation avoids repeatedly serializing the label inventory but trades away arbitrary candidate flexibility and does not preserve all capabilities of the original typed-decision model.Source: §1 · pp. 1–2
Task-specific strategies
- The BANKING77 system tests SAM continuation of a single classifier, while the CLINC150 system tests probability fusion of an R-Drop-trained Laya specialist and three DeBERTa-v3-large classifiers.SAM changes optimization; fusion combines predictors at inference, so the systems address different questions.Source: §1 · p. 2
Contributions
- 94.1234% on BANKING77 and 98.0222% on in-scope CLINC150 are the systems’ observed test accuracies.The BANKING77 figure is the best observed among three SAM seeds; the CLINC150 figure uses a four-encoder ensemble.Source: §1 · p. 2
2 Methodology
DeepOpen adapts Laya’s flexible candidate interface to fixed-label classification and uses separately trained systems for BANKING77 and CLINC150. SAM and R-Drop affect training, while the CLINC150 system fuses classifier probabilities at inference.
2 Methodology
- Figure 1 depicts fixed-label adaptation and two separately trained systems: a single SAM-trained BANKING77 classifier and weighted probability fusion for CLINC150.SAM and R-Drop are training choices; the arrows in Panels B and C describe inference.Source: §2 · p. 2, Figure 1 · p. 2
2.1 SAM-Based Optimization for BANKING77
- The BANKING77 system starts from a Laya-based English encoder and 77-class head, then continues training from historical full-training weights using three fixed seeds.The protocol uses two epochs, 626 updates, 10,003 training rows, encoder learning rate 2e-6, head learning rate 2e-5, and BF16; final checkpoints are evaluated without validation-based selection, while seed 13 is chosen for the headline after testing.Source: §2.1 · p. 3
2.2 Probability Fusion for CLINC150
- The CLINC150 system fuses probabilities from a Laya-based specialist and three DeBERTa-v3-large classifiers, with the predicted intent selected by argmax.The final system uses Laya weight 0.5 and equal weighting across the DeBERTa members; all four encoders are required at inference, and earlier rounds exposed benchmark test summaries.Source: §2.2 · p. 3
2.3 Relationship to Existing Methods
- Laya, R-Drop, SAM, and DeBERTa-v3 are established components used for task-specific systems, not methods introduced by this paper.The paper does not isolate each component’s contribution through a common cross-dataset ablation.Source: §2.3 · p. 3
3 Experiment
The experiment reports benchmark results and model-size comparisons, while distinguishing parameter footprint from other measures of deployment cost.
3.1 Dataset for Intent Classification
- BANKING77 evaluation uncertainty uses 10,000 shared-row bootstrap resamples of fixed checkpoints, excluding model-selection effects and broader training-seed uncertainty.These intervals do not make the observed tests a new independent confirmation set.Source: §3.1 · p. 4
3.2 Benchmark Results
- BANKING77 headline accuracy was one correct prediction above the archived 94.0909% result, but execution settings differ; matched-path analysis is deferred to Section 5.The headline comparison is not a like-for-like execution-path comparison.Source: Table 2 · p. 4, §3.2 · p. 4
3.3 Separate Benchmark and Model Size Comparisons
- The BANKING77 classifier uses approximately 17.7 times fewer parameters than the nominal 7B reference, with accuracy 0.2266 percentage points lower.This supports a parameter-footprint comparison for that reference pair, not an equal-compute or latency claim.Source: Table 3 · p. 4, §3.3 · p. 4
- The CLINC150 ensemble totals 1,697,433,176 parameters across a 394,935,446-parameter Laya specialist and three 434,165,910-parameter DeBERTa classifiers.The displayed model-size comparisons are contextual, not reruns under a shared data, selection, or compute protocol.Source: §3.3 · p. 4
- Adding Laya increases CLINC150 accuracy by 0.5778 percentage points over the component ensemble, while the final system exceeds the specialist by 0.2667 points.The archived hybrid report gives conditional paired 95% intervals of [0.2444, 0.9111] and [0.0667, 0.4667] points, respectively, conditional on selected weights.Source: Table 4 · p. 5, §3.3 · p. 5
3.4 Parameter Accounting and Interpretation
- The BANKING77 total of 395,910,221 parameters is derived from the shared encoder, added dense and LayerNorm layers, and 77-class head; SAM adds no inference parameters.The accounting is documented in a companion parameter-accounting file.Source: §3.4 · p. 5
- Parameter count measures model footprint, not latency, peak memory, FLOPs, or energy; four-model CLINC150 fusion illustrates an accuracy-resource trade-off.Sequence length, precision, batching, software kernels, and model-evaluation count also affect deployment cost.Source: §3.4 · p. 5
4 BANKING77 Continuation and Execution Audit
The continuation audit finds no reliable BANKING77 gain over the historical checkpoint under a shared evaluation path. A separate audit finds unresolved prediction differences for the historical checkpoint.
4 BANKING77 Continuation and Execution Audit
- 94.0368% SAM mean versus 93.9935% historical checkpoint; the +0.0433-point difference has an interval spanning zero, establishing no reliable gain.The 94.1234% headline is the post-test-selected best of three SAM seeds, an effect not modeled by these intervals.Source: Table 6 · p. 6, §4 · p. 6
4.1 Historical Prediction Drift
- Matching data and weight hashes do not resolve exact prediction reproduction: the current original-script replay differs from the archive on three labels.Eight audited eager conditions yield 2,895 correct predictions, but the responsible operator and remaining historical drift have not been isolated.Source: Table 7 · p. 6, §4.1 · p. 6
5 Reproducibility and Limitations
The report preserves the archived evaluation records while documenting limits on reproducibility, comparison, and scope. Its evidence does not establish multilingual generalization or a uniformly rerun cross-model benchmark.
5 Reproducibility and Limitations
- The evaluation explorer and linked JSON records are the result source; the continuation protocol records hashes, training configuration, precision, and bootstrap scope.No new training or model inference was performed for this editorial revision.Source: §5 · p. 6
- A full 4,500-example rerun of the exported CLINC150 system had zero prediction differences, while synthetic package checks establish loading behavior only.Historical hardware provenance and a uniformly rerun cross-model benchmark remain incomplete.Source: §5 · pp. 6–7
- Different task architectures and objectives, post-test BANKING77 headline selection, and hybrid-selected CLINC150 comparisons constrain interpretation of the reported results.The BANKING77 bootstrap intervals exclude selection and seed-population uncertainty; parameter counts do not establish compute-matched efficiency.Source: §5 · p. 7
- The report covers two English benchmarks, not all project experiments, and does not establish multilingual generalization.The public page is a result display, not proof of accepted leaderboard submission or verified delivery of every model package.Source: §5 · p. 7
6 Conclusion
DeepOpen reports task-specific intent-classification results on BANKING77 and CLINC150. The BANKING77 continuation's mean gain over the uniform-path historical checkpoint remains statistically inconclusive, so stronger causal, efficiency, or generalization claims require additional controlled evidence.
6 Conclusion
- 94.1234% BANKING77 accuracy comes from a single SAM-continued classifier, while 98.0222% in-scope CLINC150 accuracy comes from a four-encoder ensemble.The BANKING77 three-seed mean is 94.0368%, and its gain over the uniform-path historical checkpoint remains statistically inconclusive.Source: §6 · p. 7