PaperlayerView exact revision on arXiv

Source-linked AI summary

How Well Do Decision Models Transfer to Vietnamese? An Empirical Study of Text Classification

Phat Huu Ha, Nguyen Xuan Phuc, Nong Nhut Quy, Trinh Tran Tran, Nguyen Minh Triet, Dang Van Thin

arXiv:2610.31422v1cs.CL

TL;DR

Evidence on specialized decision models for Vietnamese classification remains limited, motivating evaluation across tasks, label-description languages, and linguistic variations. Across the evaluated settings, JEV ranks first most often under English descriptions, while diacritic removal produces the largest degradation. The results show that model behavior depends on evaluation conditions, and the reported compound-intent examples are too few for broad conclusions.

  • Problem

    Performance of specialized decision models across Vietnamese classification tasks with varied label inventories and linguistic characteristics remains insufficiently characterized.

    Source: §1 · p. 1

  • Method

    The study evaluates four specialized decision models across 13 Vietnamese settings, examining zero-shot performance, label-description language, and linguistic perturbations.

    Source: §1 · pp. 1–2

  • Results

    JEV ranks first in 11 of 13 settings under English descriptions, while diacritic removal causes the largest degradation among evaluated perturbations.

    Source: §1 · pp. 1–2, §6 · p. 8

  • Takeaways & Limitations

    Model behavior varies across tasks and evaluation conditions, and bilingual descriptions offer no consistent advantage.

    Source: §6 · p. 8

    Only 20 compound-intent examples were annotated, limiting broader conclusions about compound-intent recognition.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

Recent advances in specialized decision models have expanded the possibilities for zero-shot classification, yet their effectiveness on Vietnamese remains underexplored. This study evaluates four specialized decision models across 13 classification settings spanning six task groups, examining zero-shot performance, sensitivity to label-description language, and robustness to Vietnamese linguistic variations. The evaluation considers English, Vietnamese, and bilingual label descriptions, alongside common linguistic perturbations involving teencode, typographical errors, and diacritic removal. Under English label descriptions, JEV achieves the strongest overall performance, ranking first in 11 classification settings. The effects of label-description language vary across models and datasets, with bilingual descriptions providing no consistent improvement. Robustness experiments reveal that diacritic removal causes the most substantial performance degradation, reducing mean macro-$F_1$ by 9.51--15.47 points across model configurations, considerably more than teencode or typographical errors. These findings highlight the importance of label-description choices and Vietnamese orthographic variation when evaluating specialized decision models for zero-shot text classification.

1 Introduction

Specialized decision models provide structured predictions over predefined options, but their performance across Vietnamese classification tasks remains insufficiently characterized. This study evaluates four models across 13 settings and examines label-description language and linguistic robustness.

Research gap

  • Despite prior evaluations of specialized decision models, evidence on Vietnamese text classification remains limited across varied label inventories and linguistic characteristics.Prior work reports strong performance on some tasks alongside considerable variation in results.Source: §1 · p. 1

Motivation

  • Vietnamese label descriptions and input forms merit separate examination because label wording can affect predictions and user text may omit diacritics or use nonstandard spellings.Prior Vietnamese language-model evaluations report substantial performance degradation under diacritic removal.Source: §1 · p. 1

Study overview

  • Across 13 Vietnamese classification settings, JEV ranks first in 11 under English label descriptions, while language effects vary by model and dataset and diacritic removal causes the largest degradation.The study also examines zero-shot performance and robustness to Vietnamese linguistic variations.Source: §1 · pp. 1–2

Contributions

  • The study systematically compares decision models across diverse Vietnamese classification tasks, including differences at both task and class levels.The evaluation also examines how label-description language and Vietnamese linguistic variations affect performance.Source: §1 · p. 2

2 Related Work

Prior work finds that specialized decision-model performance varies across tasks and conditions, while evidence across Vietnamese classification tasks remains limited. Research on label specifications and Vietnamese text transformations motivates examining both factors in this study.

2.1 Specialized Decision Models and Their Evaluation

  • Specialized decision models show varied performance across tasks and conditions, and focused studies report sensitivity to request configuration, robustness differences, and label-selection biases.Evidence across diverse Vietnamese classification tasks remains limited.Source: §2.1 · p. 2

2.2 Label Specifications and Vietnamese Linguistic Robustness

  • Studies of generative and typed decision models find that label wording can affect classification, motivating language-specific evaluation of label descriptions.For typed decision models, option names may exert greater influence than accompanying definitions.Source: §2.2 · p. 2
  • Robustness methods test controlled linguistic transformations, and prior Vietnamese evaluations report degradation under diacritic removal, motivating tests on specialized decision models.DecisionRobust emphasizes checking that transformed inputs retain their original labels.Source: §2.2 · p. 2

3 Experimental Setup

The experiments evaluate four specialized decision models on 13 settings from 12 Vietnamese datasets, using frozen models and English, Vietnamese, or bilingual label descriptions. Macro-F1 is primary, with robustness measured by its change on perturbed inputs.

3.1 Tasks and Datasets

  • The evaluation spans 13 settings from 12 Vietnamese datasets across six task groups, including UIT-VSFC in both sentiment and topic classification.The groups are topic classification, emotion recognition, hate speech detection, sentiment analysis, intent detection, and natural language inference.Source: §3.1 · p. 2

3.2 Models

  • Four decision models are evaluated: JEV, LAYA Multilingual, GLiNER2.5-multi-Decide, and Quyet-1.0-Small; Quyet is excluded from PhoATIS because its label limit is ten.PhoATIS contains 28 intent labels.Source: §3.2 · pp. 2–3

3.3 Zero-Shot Evaluation Protocol

  • Models use frozen weights without task-specific fine-tuning or labeled demonstrations, with English, Vietnamese, and bilingual descriptions tested while label identities and order stay fixed.Macro-F1 is primary; robustness is the macro-F1 change between original and perturbed inputs under the same model configuration.Source: §3.3 · p. 3

4 Results and Analysis

Across the evaluated Vietnamese classification settings, JEV leads the specialized models under English label descriptions, but performance varies by task and metric. Error patterns and label-description effects reveal important differences across classes, models, and tasks.

4.1 Overall Performance Across Tasks

  • JEV ranks first in 11 of 13 settings and averages 69.22 macro-F1 across 12 settings shared by all models, ahead of Quyet (53.61), LAYA (43.23), and GLiNER (31.26).Quyet slightly exceeds JEV on UIT-VSMEC and UIT-ViCTSD, and on NEU-ESC GLiNER leads in accuracy (62.30%) while JEV leads in macro-F1 (37.31%).Source: §4.1 · pp. 3–4

4.2 Comparison with LLMs

  • JEV outperforms all three open-weight LLMs on five of 11 overlapping settings, while relative performance varies across tasks.JEV exceeds Qwen3-32B on ViHSD (59.34 versus 57.56) and GPT-4.1 on UIT-VSFC sentiment (73.44 versus 73.14), but GPT-4.1 exceeds JEV by 7.66 points on UIT-VSMEC.Source: §4.2 · p. 4, Figure 1 · p. 4

4.3 Error Analysis

  • On PhoATIS, 140 of JEV’s 169 incorrect predictions include the gold intent plus an extra intent, most often flight → flight#flight_time (109 cases).JEV correctly identifies 19 of 20 compound-intent examples, so the small number of such examples limits broader conclusions about compound-intent recognition.Source: §4.3 · pp. 4–5
  • On UIT-VSMEC, JEV misclassifies 80 of 132 disgust examples as anger, while its disgust F1 is 21.94 versus Quyet’s 49.09; JEV’s anger recall is 85.00% versus 32.50%.Figure 2 shows these contrasting confusion profiles despite relatively close overall macro-F1 scores.Source: §4.3 · p. 5, Figure 2 · p. 5
  • On UIT-VSFC sentiment classification, all four models overpredict neutral, assigning it to 395–1,875 examples.Source: §4.3 · p. 5

4.4 Effects of Label-Description Language

  • Label-description effects vary by model and task: Quyet gains 21.06 macro-F1 points on ViNLI with Vietnamese descriptions but loses 14.23 on ViHSD, while LAYA loses 18.70 on ViMQ with bilingual descriptions.GLiNER performs best with English descriptions and shows its largest average degradation under Vietnamese and bilingual descriptions; the effects also alter predicted-label distributions.Source: §4.4 · pp. 5–6, Table 2 · p. 6

5 Robustness to Vietnamese Linguistic Variation

The section defines three independent Vietnamese text perturbations and evaluates their effects on classification. Diacritic removal is the most damaging on average, while effects of teencode and typographical noise are smaller and vary across tasks.

5.1 Perturbation Design

  • The study independently applies teencode substitutions, Telex-related and character-level typographical noise, and diacritic removal while retaining original labels.For NLI, teencode and diacritic removal affect both premise and hypothesis, while typographical noise affects only the hypothesis.Source: §5.1 · p. 6
  • Table 3 covers perturbation coverage and character edit burden across 13 settings; coverage is the share of modified inputs, and diacritic removal has the highest coverage and edit burden.Edit burden is character-level Levenshtein distance normalized by original input length.Source: Table 3 · p. 6, §5.1 · p. 6

5.2 Performance Under Linguistic Perturbations

  • Diacritic removal reduced mean macro-F1 by 9.51 points for JEV, 13.77 for LAYA, 14.75 for GLiNER, and 15.47 for Quyet.It caused decreases in 47 of 48 shared model–setting comparisons.Source: Table 4 · p. 7, §5.2 · p. 7
  • JEV’s mean macro-F1 was 69.44 on original inputs, 59.93 under diacritic removal, 68.34 under teencode, and 66.37 under typographical noise.It remained above the other models in every perturbation condition.Source: §5.2 · pp. 6–7, Table 4 · p. 7
  • Teencode and typographical noise had smaller mean reductions of 0.47–1.10 and 2.87–4.54 points, respectively, but teencode reduced JEV and Quyet by 5.13 and 4.93 points on UIT-VSMEC.On UIT-ViOCD, LAYA improved under all three perturbations, gaining 9.58 points under typographical noise.Source: §5.2 · p. 7

6 Conclusion

The evaluation finds that model performance depends on task and evaluation conditions, despite JEV’s strong results across the tested settings. The paper also describes instance-level prediction changes as a way to examine sensitivity to label descriptions and input perturbations.

6 Conclusion

  • Under English label descriptions, JEV achieved the highest macro-F1 in 11 of 13 settings and maintained the strongest average performance across all three linguistic perturbations.Diacritic removal caused the largest average degradation for all four models, while JEV had the smallest mean reduction.Source: §6 · p. 8
  • Label-description language had model- and task-dependent effects, and bilingual descriptions offered no consistent advantage.The paper concludes that strong zero-shot performance does not necessarily imply consistent behavior across label configurations and linguistic variations.Source: §6 · p. 8

A Prediction-Level Sensitivity

  • The prediction-level analysis compares paired label-description or input-perturbation conditions; a prediction flip is a change in the predicted label.W→C, C→W, and W→W denote incorrect-to-correct, correct-to-incorrect, and two-incorrect-label transitions.Source: A Prediction-Level Sensitivity · p. 9

A.1 Label-Description Language

Label-description changes produced different instance-level effects across models and datasets.

A.1 Label-Description Language

  • On ViNLI, switching from English to Vietnamese descriptions changed 123 JEV predictions and 930 Quyet predictions; for Quyet, 682 errors were corrected and 151 new errors introduced.On ViHSD, the same change introduced 1,247 Quyet errors and corrected 147.Source: §A.1 · pp. 9–10
  • On ViMQ, switching LAYA from Vietnamese to bilingual descriptions produced 393 newly incorrect predictions and 88 corrections; GLiNER had 987 of 2,266 ViNLI flips between two incorrect labels.Source: §A.1 · p. 10

A.2 Linguistic Perturbations

Prediction-level changes under diacritic removal differ across models, and low flip rates do not always indicate strong classification performance.

A.2 Linguistic Perturbations

  • Under diacritic removal, 11.73% of JEV predictions changed from correct to incorrect, compared with 23.33% for Quyet and 27.95% for LAYA.Results are averages across the 12 shared settings, weighted equally.Source: §A.2 · p. 10
  • GLiNER’s 19.83% flip rate coexisted with lower classification performance and predictions dominated by a single label in several settings.Its apparent prediction stability therefore partly reflects persistent label concentration.Source: §A.2 · p. 10

A.3 Negation-Cue Sensitivity under Teencode

Informal rewrites of Vietnamese negation affect NLI predictions differently across models. The evaluation also uses model-adapted prompts and label descriptions in English, Vietnamese, or both.

A.3 Negation-Cue Sensitivity under Teencode

  • Replacing không with “ko” or “k” reduced contradiction predictions by 20.62 and 14.43 points for LAYA and GLiNER, versus 3.09 and 2.06 for JEV and Quyet.Across 194 instances where không appeared in the hypothesis but not the premise, LAYA and GLiNER also had larger accuracy reductions.Source: §A.3 · p. 10

B Prompt and Schema Templates

  • Prompts specify task instructions, input fields, and ordered candidate labels with descriptions; NLI uses separate premise and hypothesis fields.Gold labels and example identifiers are excluded from model inputs.Source: B Prompt and Schema Templates · p. 10
  • JEV, LAYA, and Quyet use typed-choice questions, while GLiNER uses a classification schema with label descriptions and single-label prediction.English, Vietnamese, and bilingual descriptions are evaluated, with English preceding Vietnamese in the bilingual configuration.Source: B Prompt and Schema Templates · p. 10

C Perturbation Implementation Details

Perturbations are generated with separate procedures for teencode, typographical noise, and diacritic removal. The implementation also specifies randomization, protected text, and instance-level prediction tables.

C.1 Perturbation Generation

  • Teencode uses dictionary-based slang substitutions, while typographical noise combines Telex-related errors, diacritic modifications, confusion-set substitutions, and character-level operations.Diacritic removal uses a character mapping; inputs are NFC-normalized before transformation, and OCR and word-segmentation perturbations are disabled.Source: §C.1 · p. 10
  • Character-level operations use a shared random draw sequentially, so Table 8 thresholds are not independent mutation probabilities.The max_edit_ratio parameter limits an internal operation budget per unprotected segment, with a minimum budget of one, rather than strictly bounding Levenshtein distance.Source: §C.1 · pp. 10–11
  • Table 5 presents selected instance-level prediction changes across label-description configurations.Table 6 covers instance-level prediction transitions under diacritic removal, and Table 7 covers effects of rewriting không on NLI predictions across 194 paired instances.Source: Table 5 · p. 11, Table 6 · p. 11, Table 7 · p. 11
  • Both stochastic transformations protect URLs, mentions, hashtags, digit-containing expressions, and emoji or symbol sequences; typographical noise also protects selected Vietnamese negation forms.Random generation uses base seed 42, with seeds derived from example identifiers, input fields, and segment positions.Source: §C.1 · p. 11

C.2 Perturbation Statistics

Perturbation statistics report input coverage and character edit burden across evaluation settings. The tables identify the measures and their scope, but the supplied passages do not provide their values.

C.2 Perturbation Statistics

  • Coverage is the proportion of inputs modified, and edit burden is the sum of field-level Levenshtein distances divided by total original input length.For NLI, the edit denominator includes premise and hypothesis, including an unchanged premise under typographical perturbations.Source: §C.2 · p. 11
  • Table 9 reports perturbation coverage and character edit burden by evaluation setting, across 13 settings and 33,350 setting–example pairs.UIT-VSFC is counted separately for topic and sentiment classification.Source: §C.2 · p. 11, Table 9 · p. 12
  • Table 10 compares zero-shot macro-F1 on 11 overlapping classification settings, with decision models using English descriptions and LLM results drawn from prior zero-shot experiments.Unreported results are marked with a dash; bold and underlined values denote the highest and second-highest scores per setting.Source: Table 10 · p. 12

Related papers

Loading 2610.31422v1…