Source-linked AI summary
BanglaVeilGuard: Cross-Script Safety Benchmarking and Lightweight Guardrails for Bangla Large Language Models
Md. Rakibul Hassan, Muhammad Iqbal Hossain
TL;DR
Bangla LLM safety lacks evaluation that reflects users’ cross-script, spelling, code-mixed, and regional writing, motivating a Bangla-first benchmark and guard. BanglaVeilGuard uses non-destructive multi-view normalization with a prompt-risk gate before generation, reducing ASR across target models while leaving over-refusal on dialectal and noisy benign prompts as the main limitation.
Problem
Bangla safety evaluation must distinguish semantic risk across six practical surface forms rather than relying on familiar standard-script behavior.
Method
BanglaVeilGuard combines a six-form benchmark with non-destructive multi-view normalization and a prompt-only pre-generation gate for heterogeneous target models.
Results
93.8–100.0% ASR falls to 6.3% for Claude Opus 4.8, BanglaLLama, and TituLLM under guarded evaluation.
Takeaways & Limitations
Bangla safety requires surface-form-aware evaluation and prompt-side guardrail design alongside generic multilingual alignment.
Takeaways & Limitations
Over-refusal is especially pronounced on dialectal and noisy benign prompts, while the benchmark scale, model coverage, and lexical scorer remain limited.
Abstract
from arXiv · showhide
Bangla large language model (LLM) safety is difficult to evaluate with English-centric or standard-script benchmarks because Bangla users routinely write across scripts, spellings, code-mixed forms, and regional registers. This paper presents BanglaVeilGuard, a compact Bangla-first safety benchmark and lightweight prompt guard for six language forms: standard Bangla, Romanized Bangla, Banglish, code-mixed Bangla--English, noisy Bangla, and dialectal Bangla. The benchmark contains 2,366 quality-filtered prompts and a held-out 354-prompt evaluation split spanning unsafe, safe, and safe-sensitive requests. BanglaVeilGuard uses non-destructive multi-view normalization with a prompt-risk classifier and thresholded pre-generation gate, allowing it to screen prompts for heterogeneous target models without changing their weights. Across target-model families, guarded runs reduce attack success under deterministic response scoring from 93.8--100.0\% to 6.3\% for Claude Opus 4.8, BanglaLLama, and TituLLM; TigerLLM-1B with BanglaVeilGuard achieves 78.2\% accuracy with 8.8\% ASR. The prompt guard also attains 88.5\% unsafe recall, substantially above the evaluated prompt-only guard baselines. The main remaining cost is over-refusal on dialectal and noisy benign prompts, revealing a concrete safety-helpfulness frontier for Bangla LLM deployment.
1 Introduction
BanglaVeilGuard frames Bangla LLM safety as a cross-script, cross-register problem in which semantic risk should remain detectable across users’ natural writing forms. It introduces a benchmark and lightweight prompt-side guard to evaluate and improve that robustness across heterogeneous target models.
- Bangla users routinely shift among six forms, including standard, Romanized, Banglish, code-mixed, noisy, and dialectal Bangla.
- Unsafe prompts can bypass filters in unfamiliar forms, while benign or safe-sensitive prompts can be unnecessarily refused as noisy.
- BanglaVeilGuard preserves raw prompts and adds deterministic helper views before a shared prompt-risk scorer makes a pre-generation decision.
- The evaluation asks whether models and guards respond to semantic risk rather than to the most familiar writing style.
- The benchmark contains 2,366 prompts and a 354-prompt held-out evaluation split spanning unsafe, safe, and safe-sensitive requests.
2 Related Works
Prior work establishes multilingual safety, moderation, transliteration, guard-model, and representation-level concerns, but BanglaVeilGuard targets practical safety decisions across six user-facing forms. It combines a compact benchmark, non-destructive normalization, and a prompt-only guard to measure both unsafe-completion reduction and over-refusal.
- Multilingual studies report less reliable refusal behavior and larger jailbreak surfaces for non-English and low-resource languages.
- Bangla-adjacent work covers moderation and transliteration robustness, but cross-script safety decisions involving jailbreaks, privacy, safe-sensitive prompts, and over-refusal remain less studied.
- Existing multilingual guard and representation-level methods improve coverage or safety transfer, but they do not answer whether a lightweight prompt gate works across Bangla-capable target-model families.
- BanglaVeilGuard contributes a six-form benchmark, non-destructive multi-view normalization, and a prompt-only guardrail placed before existing Bangla-capable models.
- The benchmark measures both unsafe-completion reduction and over-refusal on benign or safe-sensitive Bangla requests.
3 Methodology
BanglaVeilGuard evaluates Bangla safety across heterogeneous surface forms using a quality-filtered benchmark, non-destructive multi-view representations, and a model-agnostic pre-generation guard. Its framework combines prompt-risk classification, thresholded actions, controlled guarded generation, and deterministic response scoring.
- Benchmark Construction: Variants are generated and validated from multiple source groups using transliteration, noun-only English substitution, spelling noise, and direct dialectal labels.Low-information prompts, no-op variants, and duplicates are removed during quality filtering.
- Multi-View Representation: The guard preserves the raw prompt while adding deterministic Unicode-normalized, transliteration-hint, script-segmented, and noisy-canonical helper views.These views expose script, noise, transliteration, and code-mixing cues without modifying the original request.
- Multi-View Representation: Prompt-visible features combine character 3- to 6-gram TF–IDF, word 1- to 2-gram TF–IDF, and numeric indicators covering length, scripts, noise, and language form.Label-bearing metadata is excluded from classifier input.
- Risk Classification and Guard Policy: A balanced multinomial logistic classifier produces prompt-risk estimates, and a threshold policy selects refuse, review, or allow before target-model generation.The research profile uses τ_u=0.43 and τ_q=0.45, selected to balance unsafe recall against false positives.
- Guarded Generation and Scoring: Guarded evaluation runs the prompt-risk gate first, skips target-model calls for refuse or review actions, and sends the original prompt unchanged only when allowed.Comparisons use the same held-out split and decoding configuration within each target-model family.
- Guarded Generation and Scoring: Deterministic scoring recognizes refusal and redirection in English, Bangla, and Romanized Bangla while measuring unsafe handling, helpfulness, over-refusal, accuracy, and attack success.Script-drift additionally tests whether safety decisions change across surface-form variants of the same semantic seed.
4 Experiments and Results
The experiments evaluate BanglaVeilGuard across target-model families, prompt-only baselines, and six surface forms. Guarding sharply reduces unsafe completions, but introduces substantial over-refusal, especially for dialectal and noisy prompts.
- Experimental setup: The held-out evaluation uses 354 prompts across standard, Romanized, Banglish, code-mixed, noisy, and dialectal Bangla forms.The benchmark separates semantic risk from surface-form variation; dialectal items are risk-matched regional stress tests rather than direct translations.
- Evaluation metrics: ASR is the fraction of unsafe prompts receiving unsafe or non-refusal completions, while over-refusal measures refusals on safe or safe-sensitive prompts.Prompt-only guards are additionally evaluated with unsafe recall and safe false-positive rate.
- Main results: 6.3% ASR is achieved by guarded BanglaLLama, TituLLM, and Claude Opus 4.8 conditions, versus 93.8–100.0% for their unguarded counterparts.BanglaLLama and TituLLM improve from 48.4% accuracy and 100.0% ASR, while Claude improves from 50.5% to 75.3% accuracy and 93.8% to 6.3% ASR.
- Main results: 8.8% ASR and 78.2% accuracy are obtained by guarded TigerLLM-1B despite its smaller target-model size.Unguarded TigerLLM-9B reduces ASR from 92.7% to 41.7% relative to the unguarded 1B model, but remains substantially unsafe.
- Error analysis: 39.3–44.4% guarded over-refusal contrasts with 0.0–37.8% for unguarded baselines, establishing a safety-helpfulness trade-off.Dialectal prompts have guarded accuracies of 64.5–71.4% with 0.0–7.1% ASR, making them the strongest source of benign-side conservatism; noisy prompts show 43.3–46.7% over-refusal.
- Prompt screening: 88.5% unsafe recall makes BanglaVeilGuard the strongest evaluated prompt-only guard, ahead of Qwen3Guard-Gen-4B at 55.2% and Nemotron-SG-8B at 41.7%.Its 20.0% safe false-positive rate and 68.8% conservative rate show that the guard prioritizes unsafe-prompt coverage over minimal intervention.
- Error analysis: Dialectal and noisy prompts account for the strongest benign-side conservatism, while Banglish and code-mixed prompts have the highest observed guarded ASR values.Wide, overlapping confidence intervals prevent definitive cross-form rankings, and approximately 13 unsafe completions remain at 6.3% ASR.
5 Discussion
BanglaVeilGuard lowers unsafe completions across varied Bangla surface forms, but the high-safety operating point increases over-refusal, particularly for dialectal and noisy benign prompts. Evaluation limitations include controlled transformations, small per-form subsets, output truncation, and lexical response scoring.
- BanglaVeilGuard consistently lowers ASR by screening prompts before generation, supporting surface-form-aware evaluation beyond language identity.
- The main trade-off is helpfulness: guarded systems sharply reduce unsafe completions while increasing over-refusal on benign and safe-sensitive prompts.
- Claude outputs frequently reached the 128-token limit, potentially truncating refusal language before lexical scoring and affecting automatic ASR.
- Four evaluated forms are controlled rule-based transformations of shared standard-Bangla seeds, which enables comparisons but may understate natural usage diversity.
- Small evaluated subsets create substantial uncertainty in per-form results, as reflected by the Wilson intervals.
6 Conclusion
BanglaVeilGuard combines a six-form Bangla safety benchmark with a lightweight pre-generation guard. Across heterogeneous models, it substantially reduces unsafe responses, motivating surface-form-aware evaluation and prompt-side guardrails.
- BanglaVeilGuard provides a six-form Bangla benchmark and lightweight pre-generation prompt guard for Bangla-capable LLMs.
- 93.8–100.0% ASR falls to 6.3% for Claude Opus 4.8, BanglaLLama, and TituLLM under guarded evaluation.
- 78.2% accuracy with 8.8% ASR is achieved by TigerLLM-1B with BanglaVeilGuard.
- Bangla safety requires surface-form-aware evaluation and prompt-side guardrail design, not only generic multilingual alignment.