Source-linked AI summary
BengaliMCQ: Automatic Generation and Answer Prediction of Academic Multiple-Choice Questions in a Low-Resource Language
Abu Tarabin Surzo, A. K. M. Nihalul Kabir, Sm Azmain Faysal, Ariana Haque Ami, Lawrence Amlan Gomes, Farig Sadeque
TL;DR
BengaliMCQ addresses structure-blind retrieval and context limits in Bengali textbook-based MCQ generation by modeling textbooks as hierarchical graphs and retrieving focused passages. It outperforms baselines across retrieval, question quality, and answer accuracy, reaching 91.41% answer prediction accuracy.
Problem
Standard dense retrieval often ignores textbook hierarchy, while feeding entire chapters to LLMs is impractical for topic-specific MCQ generation.
Method
BengaliMCQ models textbooks as hierarchical graphs and uses a GNN-based retriever to rank relevant passages while reducing context length without sacrificing coverage.
Results
BengaliMCQ outperforms baselines in retrieval quality, question quality, and answer accuracy, reaching 91.41% answer prediction accuracy.
Takeaways & Limitations
The framework reduces LLM input length while maintaining high coverage for topic-relevant Bengali educational question generation and answer prediction.
Takeaways & Limitations
GNN training relies on synthetic supervision, which may introduce hallucination noise into the training signal.
Abstract
from arXiv · showhide
Traditional retrieval-augmented generation (RAG) frameworks process documents without attending to their hierarchical structure, leading to poor performance, especially in low-resource languages such as Bengali. To address this, we propose a structure-aware RAG framework that models Bengali textbooks as hierarchical graphs and uses a contrastively trained graph neural network to retrieve a small set of relevant passages. These passages provide focused context for a large language model, enabling topic-specific multiple-choice question (MCQ) generation and in-domain answer prediction. Experimental results demonstrate that our framework outperforms strong dense retrieval baselines across retrieval metrics, produces more relevant MCQs, and achieves superior answer prediction accuracy.
1. Introduction
BengaliMCQ addresses the challenge of topic-specific academic MCQ generation in Bengali by using document graphs and GNN-based retrieval to provide focused, relevant context. The framework is evaluated for retrieval, question quality, answer accuracy, and the contribution of its design choices.
- Motivation: LLM-based MCQ generation depends heavily on relevant input context, while entire textbook chapters are impractical because of context-window limits.Even full chapters may not consistently receive adequate model attention.
- BengaliMCQ: BengaliMCQ uses document graphs to generate topic-relevant academic MCQs in Bengali and predict answers to in-domain MCQs.The approach addresses a gap in graph-based document modeling, which has largely focused on high-resource languages.
- Method: A GNN-based retrieval framework models textbook hierarchical structure to rank query-relevant passages and reduce context length without sacrificing coverage.The retrieved passages support focused downstream processing by the language model.
- Evaluation: Automatic metrics and expert validation evaluate retrieval quality, question quality, and answer accuracy against baselines.The framework is reported to outperform baselines across all three evaluation areas.
- Evaluation: An ablation study analyzes the contribution and value of the framework’s design choices.The study is included to assess how individual choices affect the approach.
2. Related work
Prior work on automatic MCQ generation has progressed from probabilistic and rule-based methods to neural and transformer-based frameworks. Recent studies also combine graph neural networks with retrieval-augmented generation to model document structure and retrieve salient content.
- Neural and transformer-based methods: Automatic MCQ generation evolved from TF-IDF, n-grams, and part-of-speech tagging to neural-network and transformer-based approaches.T5- and BERT-based fine-tuning became a standard approach for downstream tasks including question generation, named-entity recognition, and question answering.
- Neural and transformer-based methods: T5- and BERT-based models have become the de facto standard for downstream NLP tasks such as question generation, named-entity recognition, and question answering.
- Graph-based retrieval: Recent graph-RAG research models structural relationships with GNNs, including question–answer sentence graphs for answer sentence selection and hierarchical research-paper documents.Iyer et al. trained a GNN on an offline temporary graph, while AutoRev models research papers as hierarchical documents and retrieves salient content.
3. Methodology
BengaliMCQ represents Bengali textbooks as hierarchical, semantically enriched graphs and trains a contrastive graph attention network to rank relevant passages. At inference, GAT retrieval and cross-encoder reranking provide a compact grounded context for MCQ generation and answer prediction.
- Textbook preparation: The study processes four secondary-school Bengali textbooks across literature, STEM, and social science domains using OCR to identify hierarchical textual units.The books are Bangla Shahitto, Shohopath, Biology, and Bangladesh and Global Studies; Tesseract-4 and Google Lens extract chapters, headings, subheadings, passages, and sentences.
- Graph construction: The method models each book as a hierarchical directed multigraph whose nodes are chapters, headings, subheadings, passages, and sentences connected by hierarchical, sequential, and weighted semantic edges.Semantic edges link each sentence to its top-K_s similar neighbors, with K_s∈{3, 5} and cosine-similarity thresholds T∈{0.5, 0.6, 0.7}.
- Graph attention network: A custom multi-head graph attention network integrates semantic edge weights and edge-type biases, stacks three attention layers, and uses residual fusion to preserve initial node information.Node features use 1024-dimensional fine-tuned bi-encoder embeddings, while passage nodes are also evaluated with zero-vector and mean-pooled child-node initialization.
- Contrastive training: The GAT is trained with multi-positive InfoNCE to score all relevant passages highly while suppressing other passages from the same book.Training varies K_s and T, producing models denoted GAT(K_s, T).
- Inference pipeline: At inference, GAT ranking retrieves 10 candidate passages, a BGE-Reranker-V2-M3 cross-encoder reranks them, and the top 5 passages form the grounded context.The pipeline accepts either a topic for MCQ generation or a question for answer prediction.
4. Results and analysis
The proposed framework outperforms retrieval baselines, achieves the highest answer-prediction accuracy, and benefits from hierarchical graph structure and learned temperature. Expert evaluation further finds most generated MCQs relevant, with moderate annotator agreement.
- Answer prediction: 91.41% average accuracy is the highest among all baselines on 1,000 held-out gold-standard MCQs.The dataset was manually curated from secondary school public exam archives to match the target textbooks.
- Ablation studies: Removing hierarchical edges causes retrieval performance to collapse, identifying hierarchical structure as the most important learned graph signal.The ablation initializes passage nodes with zero vectors so ranking depends on graph message passing.
- Ablation studies: Direct BGE-M3 passage-node encoding achieves the best overall performance, while zero initialization outperforms mean-pooling of child sentence embeddings.The comparison covers mean-pooled child sentence embeddings, direct fine-tuned BGE-M3 encoding, and zero vectors.
- Expert validation: 82.1% of 140 expert-evaluated MCQs receive relevance scores of at least 3, while annotator agreement is moderate at Cohen’s κ=0.55.Two domain experts evaluated samples from literature, biology, and social science textbooks.
5. Conclusion
BengaliMCQ uses hierarchical textbook graphs and a GNN to retrieve topic-relevant content for low-resource Bengali MCQ generation and answer prediction, reducing context length while maintaining high coverage. The authors note limitations from synthetic supervision and approximate, limited evaluation.
- Contributions: BengaliMCQ models textbooks as hierarchical document graphs and uses a GNN to retrieve topic-relevant content for automated MCQ generation and answer prediction.The system targets a low-resource language and reduces LLM input length while maintaining high coverage.
- Results: 43,591 to 1,651 tokens: BengaliMCQ reduces average context length while maintaining high coverage.Experiments also show strong empirical performance through automatic metrics and expert validation.
- Limitations: Synthetic supervision may introduce hallucination-related noise into GNN training, whereas a completely human-curated dataset could provide a more reliable and robust signal.This is identified as a central limitation of the work.
- Limitations: LLM- and embedding-based evaluation scores are approximations, and human evaluation covered only a small stratified subset of topics and items across two domains.The authors characterize automatic evaluation as useful but limited in scope and reliability.
CRediT authorship contribution statement
The authors contributed across conceptualization, methodology, drafting and revision, analysis, visualization, data curation, investigation, and supervision. Contributions were distributed among six authors with overlapping methodological and analytical roles.
- Writing and methodology: Abu Tarabin Surzo led conceptualization, methodology, and writing the original draft, while A.K.M. Nihalul Kabir contributed to methodology and drafted, reviewed, and edited the manuscript.Both authors contributed to the original draft; Kabir additionally handled review and editing.
- Analysis and visualization: Sm Azmain Faysal contributed methodology, formal analysis, and visualization, while Ariana Haque Ami handled data curation and formal analysis.Formal analysis was shared by Faysal and Ami.
- Data and supervision: Lawrence Amlan Gomes contributed data curation and investigation, while Farig Sadeque contributed conceptualization and supervision.Sadeque’s role included supervision in addition to conceptualization.