Source-linked AI summary
To What Extent Do Large Language Models Understand Bangla Idioms?
Mousumi Akter, Md. Faiyaz Abdullah Sayeedi, Nurul Labib Sayeedi, Swakkhar Shatabda
TL;DR
Bangla idiom understanding lacks large-scale resources and systematic LLM evaluation. The paper introduces benchmark and synthetic MCQ datasets and evaluates recent LLMs on paraphrasing, span detection, and meaning identification, finding task-specific leaders rather than one consistently best model.
Problem
Bangla idiom research lacks large-scale annotated resources and systematic evaluation of LLMs across idiom-related tasks.
Method
The paper builds a Bangla idiom benchmark and synthetic MCQ dataset, then evaluates recent LLMs across paraphrasing, span detection, and meaning identification.
Results
Performance varies by task: Phi-4-mini-instruct excels in paraphrasing, Kimi-K2-32b-instruct in span detection, and Gemini-2.5-flash in meaning identification.
Takeaways & Limitations
The datasets and analyses provide resources for advancing idiomatic understanding in Bangla and other low-resource languages.
Takeaways & Limitations
Only 4,772 of 10,822 idiomatic expressions have contextual sentences, limiting applicability for context-dependent tasks.
Abstract
from arXiv · showhide
Idiomatic expressions are an integral part of natural language, reflecting cultural nuances and posing unique challenges for computational models, particularly in low-resource languages. In this paper, we present the first large-scale benchmark dataset of Bangla idioms, complemented by a synthetic multiple-choice question (MCQ) dataset for idiom meaning identification. We conduct a comprehensive evaluation of recent large language models (LLMs) across three idiom-related tasks: paraphrasing, idiom span detection, and meaning identification, leveraging zero-shot and few-shot prompting strategies. Our results reveal substantial variability in model performance, with no single LLM consistently outperforming others across all tasks. Notably, Phi-4-mini-instruct excels in paraphrasing, Kimi-K2-32b-instruct in span detection, and Gemini-2.5-flash in meaning identification. We believe that our datasets and analyses will provide valuable resources to guide future research in improving LLM comprehension of idiomatic expressions, particularly in Bangla and other low-resource languages.
1 Introduction
The paper addresses limited Bangla idiom resources by introducing benchmark and synthetic MCQ datasets, then evaluates LLMs across three idiom-related tasks. The models show task-specific strengths rather than one consistently best system.
- Bangla idioms lack annotated resources and a large-scale benchmark dataset, limiting development and evaluation of robust NLP models.
- The benchmark contains 10,822 idiom entries, including 4,772 with usage examples.
- The synthetic MCQ dataset includes 10,913 single-answer and 38,688 multiple-answer samples for meaning detection.
- The evaluation covers paraphrasing, idiom span detection, and MCQ-based meaning detection across recent LLMs.
- Phi-4-mini-instruct excels in paraphrasing, Kimi-K2-32b-instruct in span detection, and Gemini-2.5-flash in meaning identification.
2 Related Works
Prior work has developed idiom resources and studied idiomaticity across languages, but Bangla remains underrepresented and lacks systematic LLM evaluation for idiom-related tasks. This paper addresses that gap with datasets and benchmarking.
- Previous NLP research has produced idiom resources for Hindi, Marathi, English, multilingual settings, and Chinese.
- Bangla remains underrepresented in NLP despite datasets for summarization, question answering, paraphrasing, back-transliteration, and named-entity recognition.
- Prior Bangla figurative-understanding efforts were small-scale, and no prior work systematically evaluated LLMs on Bangla idiom-related tasks.
- The paper introduces a large-scale Bangla idiom dataset, a synthetic MCQ corpus, and benchmarks recent LLMs on three idiom-related tasks.
3 Bangla Idiom Dataset
The paper presents a curated Bangla idiom benchmark with meanings and contextual examples, then derives a balanced MCQ dataset supporting single- and multi-answer meaning detection.
- The benchmark was manually collected from open-access and permissive sources, cleaned for duplicates and spacing, and verified by native Bangla-speaking authors.
- Each dataset entry contains an idiom surface form, Bangla meaning glosses, and a sentence illustrating natural usage.
- The final dataset contains 10,822 unique idiom forms, including 2,624 with multiple meanings and 4,772 with contextual sentences.
- The MCQ dataset samples meanings from other idioms as distractors, shuffles options, and supports multiple correct meanings.
- The resulting MCQ dataset contains 10,913 single-answer instances and 38,688 multi-answer instances.
4 Experimental Setup
The experimental setup evaluates LLM understanding of Bangla idioms through three tasks using multiple models and zero-shot or five-shot prompting. Each task uses task-specific inputs, references, and metrics.
- The study defines paraphrasing, idiom span detection, and MCQ meaning detection as its three evaluation tasks.
- Eight LLMs were evaluated through the Groq AI4 API on a 1,000-sample subset.
- Paraphrasing: Paraphrasing asks models to replace an idiom while preserving meaning and evaluates outputs with ROUGE, BERTScore, and cosine similarity.
- Idiom Span Detection: Idiom span detection asks models to identify idiomatic portions and uses n-gram overlap percentage and Levenshtein distance.
- MCQ Meaning Detection: MCQ meaning detection tests single- and multiple-correct-answer settings and evaluates recognition of valid interpretations using accuracy.
5 Results and Analysis
Across Bangla idiom tasks, model performance depended strongly on the task, prompting setup, and evaluation metric. Phi-4-mini-instruct led paraphrasing, Kimi-K2-32b-instruct led span detection, and Gemini-2.5-flash led meaning identification, while embedding-based metrics aligned better with human judgments than lexical metrics.
- Automated Evaluation: Five-shot prompting generally improved performance, and Phi-4-mini-instruct achieved the highest paraphrasing average score of 0.73.Its five-shot ROUGE-1 and ROUGE-2 scores were 0.63 and 0.50, respectively.
- Automated Evaluation: Kimi-K2-32b-instruct achieved the best five-shot span detection, with 48.25% unigram overlap and a 6.27 Levenshtein distance.Most models benefited from in-context demonstrations in this task.
- Automated Evaluation: Gemini-2.5-flash achieved the highest meaning-detection accuracy in both settings, scoring 0.76 for single-answer and 0.55 for multi-answer questions.The MCQ evaluation tested recognition of one or multiple valid idiom meanings.
- Overall Findings: Overall, no single model consistently led all three tasks, demonstrating that Bangla idiom understanding varies substantially across subtasks.The strongest model differed for paraphrasing, span detection, and meaning identification.
- Human Evaluation: Human evaluation found Kimi-K2-instruct highest overall at 1.438, followed by Gemini-2.5-flash at 1.375, while Qwen3-32B scored lowest at 0.583.Scores were averaged on a 0–2 scale across manually assessed paraphrases.
- Metric Validation: Embedding-based metrics aligned more closely with human judgments than lexical metrics, with BERTScore Precision reaching r = 0.80 while ROUGE-1 reached only r = 0.15.The analysis therefore identifies semantic metrics as stronger proxies for Bengali idiomatic paraphrase quality.
6 Error Analysis and Limitations of Open-Source LLMs for Bangla Idiom Understanding
Open-source Bangla LLMs show weaknesses in idiom understanding, including incoherent responses, span over-extraction, and incomplete multi-label meaning identification. Few-shot prompting can sometimes worsen span detection, while Gemini-2.5-Flash more precisely identifies idiom boundaries.
- General Model Limitations: Existing Bangla pre-trained LLMs struggled to maintain semantic coherence even in simple conversational settings.This limited their applicability to downstream idiom understanding tasks.
- Prompting Effects: Few-shot prompting sometimes degraded Bangla idiom span detection by introducing hallucinated or semantically irrelevant spans.In one example, zero-shot prompting overextended the span but retained the idiom, whereas five-shot prompting omitted the core idiomatic expression.
- Span Detection Errors: Open-source LLMs frequently over-extracted idiomatic spans by including surrounding non-idiomatic words.This issue was especially prominent when idioms appeared within broader noun phrases or figurative constructions.
- Span Detection Errors: Gemini-2.5-Flash identified precise idiomatic boundaries more successfully than the evaluated open-source models.The comparison indicates stronger fine-grained semantic understanding for idiomatic spans in Bangla.
- Meaning Identification: Open-source LLMs failed to identify the complete set of correct answers in Bangla idiom MCQs requiring multiple valid meanings.The task requires selecting all correct meanings from four options, and culturally grounded meanings were especially difficult to identify simultaneously.
7 Conclusion
The paper introduces large-scale Bangla idiom resources and evaluates recent LLMs across paraphrasing, span detection, and meaning identification. Results vary substantially across tasks, with no single model performing optimally throughout; the resources support further study of idiomatic understanding in low-resource languages.
- Conclusion: The paper introduces the first large-scale Bangla idiom benchmark and a synthetic MCQ dataset for meaning detection.The benchmark contains idioms with meanings and usage examples, while the MCQ dataset targets idiom meaning identification.
- Conclusion: LLM experiments cover paraphrasing, idiom span detection, and meaning identification using varied prompting strategies.The evaluation examines recent models across multiple idiom-related tasks.
- Conclusion: Model performance varies notably, and no single LLM performs optimally across all evaluated tasks.This is the paper’s overall conclusion from the multi-task evaluation.
- Conclusion: The datasets and analyses provide a foundation for advancing idiomatic understanding in LLMs, particularly for low-resource languages such as Bangla.Further linguistic analyses, including translation challenges and vocabulary complexity, remain outside this work’s scope.
Limitations
The benchmark and evaluation have several scope constraints, including incomplete idiom coverage, limited contextual examples, sampled evaluation settings, simple distractors, and restricted model inclusion.
- Dataset Scope: Some idioms may have been missed unintentionally, leaving room for future dataset expansion.The authors describe the dataset as broad and carefully curated but not necessarily exhaustive.
- Context Coverage: Only 4,772 of 10,822 idiomatic expressions, or 44.1%, include contextual usage sentences.This limits applicability to context-dependent tasks, although the available sentences were collected from original sources and are human-written.
- Evaluation Scope: Experiments used 1,000 samples per setting because of computational constraints, which may reduce evaluation granularity.The authors state that this still captures overall performance trends.
- MCQ Design: Randomly sampled MCQ distractors may allow models to solve questions through elimination rather than meaning understanding.The authors suggest developing more challenging distractor-construction strategies.
- Model Coverage: Budget constraints limited the inclusion of some commercially available or high-cost LLMs.This narrows the range of systems represented in the evaluation.
A.1 Data Licensing and Source Redistribution Permissions
The benchmark sources are distributed under open, public, or Creative Commons frameworks that permit specified forms of reuse, adaptation, and redistribution for educational and non-commercial research.
- Source Permissions: The benchmark’s text sources use open, public, or Creative Commons frameworks permitting educational and non-commercial research reuse.The paper states that source annotations, vocabulary phrases, and contextual sentences follow permissive reuse guidelines.
- Source Permissions: Bangla Wikipedia entries are distributed under CC BY-SA 4.0, which permits copying, remixing, and redistribution with attribution under the same terms.The licensing framework for each source category is summarized in Table 9.
A.2 Data Collection Strategy
The dataset was built through a two-stage digitization and curation framework combining digital-source collation with print-resource digitization. Native-speaker auditing, annotation, deduplication, and standardized JSON structuring prepared the verified idiom entries for benchmarking.
- Two-stage sourcing: The authors combined manual collation of natively digital Bangla sources with digitization of physical grammar books and newspapers.Digital sources included Bangla Wikipedia and established online lexicons; print sources included NCTB grammar books and contemporary newspapers.
- Two-stage sourcing: High-resolution images of print resources were converted into editable Bengali text through an OCR pipeline optimized for the Bengali script.
- Curation and structuring: Bangla native speakers audited entries for spelling inconsistencies and duplicates, while annotators identified idioms with multiple senses.Multiple senses were stored as a single list under each corresponding idiom entry.
- Curation and structuring: The verified artifacts were organized in a standardized JSON schema to support future benchmarking.