Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation

Ismail Hossain; Nafi Ullah Shafin; Mohammad Abdullah Al Mumin

Source-linked AI summary

Quantization Effects on Bangla Language Understanding in Large Language Models: A Systematic Evaluation

Ismail Hossain, Nafi Ullah Shafin, Mohammad Abdullah Al Mumin

arXiv:2608.24615v1Aug 25, 2026cs.CL

TL;DR

The paper asks whether post-training quantization affects Bangla understanding differently from English, especially given Bangla’s linguistic complexity and limited resources. It evaluates three model families across five Bangla benchmarks and finds that quantization effects depend more on architecture and format than bit width alone, with important implications for constrained-hardware deployment.

Problem
Most quantization evaluations use English benchmarks, leaving its effects on morphologically complex, low-resource Bangla unclear.
Method
The study compares three LLM families in full precision and INT8-range quantized formats across five Bangla NLU benchmarks using zero-shot evaluation.
Results
Quantization costs at most 1.5% absolute accuracy for GPTQ-compressed Qwen and LLaMA, but GGUF-W8A16 reduces GPT-OSS reasoning and commonsense accuracy by up to 57.4%, while reading comprehension remains relatively stable.
Takeaways & Limitations
Quantization can support resource-constrained Bangla NLP deployment, but practitioners should select the model architecture and quantization format carefully.
Takeaways & Limitations
Family, quantization format, and model size are not fully crossed, so architecture effects cannot be separated from possible format or model-size effects.

Abstract

from arXiv · show

Post-training quantization lowers the memory footprint of Large Language Models (LLMs) and speeds up inference, which is why it is now common for on-device deployment. Most of what we know about its effects, however, comes from English benchmarks. It is not clear whether the same holds for morphologically complex, low-resource languages such as Bangla, and this gap is what we address here. We evaluate three model families---Qwen-2.5-7B, LLaMA-3.1-8B, and GPT-OSS-20B---in full precision and in three quantized formats (GPTQ-Int8, GPTQ-Q8, GGUF-W8A16) across five Bangla natural language understanding benchmarks (Bangla MMLU, CommonsenseQA-BN, OpenBookQA-BN, PIQA-BN, and BoolQ-BN), using zero-shot evaluation through lm-evaluation-harness. To our knowledge this is the first controlled comparison of quantization formats on Bangla NLU. The three families do not respond the same way: GPT-OSS loses up to 57.35% accuracy on reasoning-heavy tasks under GGUF-W8A16, while Qwen and LLaMA hold steady under GPTQ, and in a few cases the quantized version edges out the full-precision one. BoolQ-BN, a comprehension task, stays stable across all three families regardless of format. Taken together, these results suggest quantization can work well for Bangla deployment, but the choice of architecture and quantization method matters more than the bit width alone. We discuss what this means for practitioners choosing a model to run on constrained hardware.

1 Introduction

The paper addresses whether post-training quantization affects Bangla NLU differently from English-centered evaluations, and frames this as a practical question for constrained-hardware deployment. It systematically compares model families, quantization formats, and Bangla task sensitivities.

Motivation: Post-training quantization can reduce memory use and speed inference, making on-device LLM deployment more realistic on constrained hardware.The motivation is especially relevant where computing infrastructure is limited, including much of South Asia.
Research gap: English benchmarks dominate prior quantization evaluations, leaving Bangla’s low-resource and morphologically complex setting insufficiently studied.The paper highlights tokenization complications from morphology, conjunct consonants, and Unicode grapheme composition.
Research questions: The study asks how quantization affects Bangla accuracy, which formats preserve it best, and whether reasoning, commonsense, and comprehension differ in sensitivity.These questions organize the paper’s controlled comparison of post-training quantization effects.
Contributions: The paper presents the first controlled, multi-family comparison of post-training quantization effects on Bangla NLU across five task categories.It covers 15 model–benchmark pairs and translates the findings into deployment recommendations.

2 Related Work

Prior work identifies Bangla-specific challenges in multilingual LLMs and studies quantization mainly on English benchmarks. This paper fills that gap by evaluating quantized models across diverse Bangla NLU tasks.

Multilingual LLMs and low-resource languages: Bangla performance is challenged by sparse token coverage, tokenization mismatches, morphological complexity, and a comparatively small web-scale footprint.Bangla’s agglutinative morphology and conjunct-consonant orthography contribute to these challenges.
Multilingual LLMs and low-resource languages: BanglaBERT, IndicLLMs, and TituLLMs address low-resource language gaps through curated data, broader South Asian pretraining, or Bangla-aware tokenizers.TituLLMs specifically benchmark Bangla NLU across several of the tasks used here.
Quantization methods: Post-training quantization lowers weight precision after training, while GPTQ and AWQ use reconstruction or activation information to limit accuracy loss.These methods avoid the cost of quantization-aware retraining and support more efficient deployment.
Quantization methods: Quantization typically reduces memory use by 2×–8× and speeds inference by 2×–5× across the methods discussed.These efficiency gains make consumer laptops and edge hardware viable deployment targets.
Quantized LLM evaluation: English-only studies generally find complex reasoning more vulnerable to quantization than factual retrieval, especially at lower bit widths.The reviewed literature does not address low-resource or morphologically complex languages.
Quantized LLM evaluation: This paper fills the literature gap by evaluating quantized LLMs on five Bangla benchmarks spanning reasoning, commonsense inference, and reading comprehension.The authors describe this combination as not previously evaluated to their knowledge.

3 Methodology

The methodology compares full-precision and quantized variants of three LLM families on five Bangla classification benchmarks under controlled zero-shot evaluation. Accuracy and absolute or relative degradation quantify the effects of quantization.

Experimental design: The study holds prompt format, decoding strategy, and harness version fixed while comparing full-precision and quantized variants within three model families.Within each family, quantization is the intended model difference.
Benchmarks: Five publicly available Bangla benchmarks cover reasoning, commonsense inference, and reading comprehension.The suite includes Bangla MMLU, CommonsenseQA-BN, OpenBookQA-BN, PIQA-BN, and BoolQ-BN.
Benchmarks: Bangla MMLU tests broad encyclopaedic reasoning across 57 subjects using four-choice questions.Its subjects span science, humanities, and social sciences.
Benchmarks: CommonsenseQA-BN uses five-choice questions requiring implicit real-world reasoning, while OpenBookQA-BN uses four-choice questions requiring multi-step elementary science reasoning.The latter combines reasoning with external factual knowledge.
Benchmarks: PIQA-BN evaluates physical and procedural commonsense reasoning with two-option questions, while BoolQ-BN tests binary reading comprehension from short passages.BoolQ-BN requires extracting and verifying a factual claim.
Evaluation procedure: Evaluations use lm-evaluation-harness in zero-shot mode, selecting answers by log-likelihood over candidate choices or BoolQ-BN Yes/No tokens.Random seeds are fixed and greedy selection minimizes sampling variance.
Evaluation metrics: Accuracy is the primary metric, and quantization impact is reported using absolute and relative performance degradation.Negative Δ values are retained when quantized models outperform their full-precision baselines.

4 Results

Across five Bangla benchmarks, full-precision models show complementary strengths, while quantization affects families differently: GPT-OSS degrades sharply on reasoning tasks, whereas Qwen and LLaMA remain close to baseline. BoolQ-BN is comparatively stable across families.

4.1 Full-Precision Baseline Performance: Full-precision models show no across-the-board winner: Qwen leads Bangla MMLU, GPT-OSS leads PIQA-BN and OpenBookQA-BN, and LLaMA leads BoolQ-BN.Qwen-2.5-7B scores 0.440 on MMLU, GPT-OSS-20B scores 0.641 on PIQA-BN and 0.584 on OpenBookQA-BN, while LLaMA-3.1-8B scores 0.914 on BoolQ-BN.
4.3 Performance Degradation Analysis: 57.4% and 54.1% are GPT-OSS’s accuracy losses on CommonsenseQA-BN and OpenBookQA-BN after quantization, with PIQA-BN falling 22.4%.BoolQ-BN is the exception, with a 5.6% drop.
4.2 Quantized Model Performance: Under GPTQ, LLaMA stays within 1% degradation on every benchmark, while Qwen’s GPTQ-Int8 incurs no measurable loss and slightly improves three benchmarks.Qwen improves on MMLU, CSQA, and PIQA by less than 1.5%; LLaMA’s OpenBookQA-BN is unchanged and PIQA-BN rises by 0.4%.
4.3 Performance Degradation Analysis: BoolQ-BN remains the most stable benchmark: GPT-OSS drops 0.030, while LLaMA and Qwen drop by 0.007 or less.This stability contrasts with the larger losses observed on reasoning and commonsense benchmarks.
4.3 Performance Degradation Analysis: Radar charts show severe contraction of GPT-OSS on CommonsenseQA-BN and OpenBookQA-BN, versus near-unchanged Qwen and LLaMA profiles.The charts visualize the same five-benchmark accuracy profiles before and after quantization.

5 Discussion

Quantization robustness differs sharply by model family, while reasoning-heavy Bangla tasks are more fragile than reading comprehension. The findings support format-specific deployment choices, but proposed explanations remain partly confounded.

Task Sensitivity: Reasoning vs. Comprehension: Reasoning-heavy benchmarks degrade more than BoolQ-BN reading comprehension across all three model families.The reasoning tasks include CommonsenseQA-BN, OpenBookQA-BN, and Bangla MMLU.
Linguistic Interaction and the Bangla Factor: Bangla’s morphology and Unicode grapheme clusters may amplify precision-error accumulation, but the study treats this language-specific effect as a hypothesis rather than confirmation.GPT-OSS loses 57.4% on CommonsenseQA-BN, above the 10–20% range typically reported for GGUF on English MMLU.
Architectural Resilience to Quantization: Small Qwen-Int8 gains over Qwen-FP16 remain below 1.5% and are interpreted as measurement noise or possible mild regularization.Reported gains occur on MMLU (+0.6%), CSQA (+0.5%), and PIQA (+0.1%).
Practical Implications for Bangla NLP Deployment: The authors recommend GPTQ-Int8 or GPTQ-Q8 for on-device deployment and advise against GGUF-W8A16 for reasoning-sensitive use cases.Qwen and LLaMA configurations stay near full-precision accuracy at roughly half the memory footprint, while GPT-OSS collapses on multi-step reasoning tasks.

6 Conclusion

The study finds that quantization effects on Bangla NLU depend more on architecture and format than on bit width alone. GPTQ preserves Qwen and LLaMA accuracy, whereas GPT-OSS under GGUF suffers large reasoning losses, and reasoning remains more sensitive than comprehension.

6 Conclusion: GPTQ-Int8 and GPTQ-Q8 reduce Qwen and LLaMA accuracy by at most 1.5% across five Bangla benchmarks, while GPT-OSS GGUF-W8A16 loses up to 57.4% on reasoning and commonsense tasks.Reading comprehension remains reasonably stable even for GPT-OSS.
6 Conclusion: Reasoning tasks are more sensitive to quantization than reading comprehension across all three tested model families.This pattern extends prior English-benchmark observations to Bangla, a low-resource, morphologically complex language.
Future Work: Future work should test INT4 formats, Bangla-native models, quantization-aware training, and generative Bangla tasks.These directions would broaden the accuracy–compression and task coverage beyond the present evaluation.

Ethics Statement

The authors report no expected direct harm from the study, while warning that degraded reasoning in quantized models could create risks in high-stakes decision-support settings without validation.

Quantized models with degraded reasoning accuracy could cause real problems in high-stakes decision-support settings without appropriate validation.The evaluations use public, openly licensed models and datasets and involve no private or personally identifiable data.

Limitations

The study’s limitations constrain how broadly its quantization findings can be interpreted. Key boundaries involve evaluated formats and settings, incomplete experimental crossing, and an undocumented calibration dataset.

Scope of evaluation: Only INT8-range formats are evaluated; INT4 compression, few-shot or fine-tuned settings, latency, and peak memory are not covered.The study also excludes quantized Bangla-native models and generative Bangla tasks, limiting conclusions about language-specific resilience and broader deployment behavior.
Experimental design: Family, quantization format, and model size are not fully crossed, so architecture effects cannot be separated from GGUF-W8A16 or model-size effects.A full 3×3 grid would be needed to disentangle these factors.
Calibration transparency: The GPT-OSS GGUF calibration dataset is not publicly documented, adding a confound to cross-family comparisons involving that model.

A Dataset Statistics and Compute Budget

The evaluation uses benchmark-specific evaluation splits under a zero-shot protocol, with dataset example counts reported in Table 5. Total study compute was approximately 40 GPU-hours.

Dataset statistics: Table 5 reports the number of examples in the evaluation split for each benchmark.The supplied passage identifies the statistic but does not provide the benchmark-specific counts.
Evaluation protocol: All evaluation is zero-shot, using only the test or full split.
Compute budget: Approximately 40 GPU-hours covered all six model–precision configurations across the five benchmarks.GPT-OSS-20B consumed the largest share because of its parameter count.