PaperlayerView exact revision on arXiv

Source-linked AI summary

From Promises to Accountability: Comparative AI Modeling and Visualization of Electoral Commitments in Nepali Political Texts

Prakash Poudyal, Nischal Karki, Amrit Dahal, Rajani Chulyadyo, Sameer Tamrakar, Bal Krishna Bal

arXiv:2610.32150v1cs.CL

TL;DR

Campaign promises in Nepal are often not systematically archived, making it difficult for citizens to assess their fulfillment. The paper builds a gold-standard dataset and compares ML, DL, and LLM approaches for extracting commitments; fastText-LSTM and LaBSE-LR lead their respective model groups, while the mobile app presents commitments with GAI-supported visuals.

  • Problem

    Nepali electoral commitments are often not systematically archived, limiting citizens’ ability to evaluate whether elected representatives fulfill them.

    Source: §1 · p. 2

  • Method

    The paper builds a gold-standard dataset of Nepali political texts, compares ML, DL, and LLM commitment classifiers, and integrates the system into a mobile application with GAI-supported visuals.

    Source: §1 · p. 3, §3.3 · p. 8

  • Results

    fastText-LSTM achieved 0.913 macro F1-score and LaBSE-LR achieved 0.896 macro F1-score, with no statistically significant difference between the two; lightweight models outperformed the evaluated LLMs.

    Source: §5 · pp. 23, 25

  • Takeaways & Limitations

    The mobile application presents candidate commitment summaries and GAI images as a tool for citizens to evaluate whether promises are honored and promote transparency and accountability.

    Source: §8 · p. 27

    The study did not establish a human performance baseline for commitment classification, so it lacks a direct human benchmark for model performance.

    Source: §6 · p. 26

Abstract

from arXiv · show

Election Commitment Mining refers to the analysis of election commitments made by electoral candidates or political parties during election campaigns. However, many elected candidates and political parties, especially in developing South Asian countries, do not fully deliver on the promises they make before elections. Even after the elections, commitments presented through election manifestos, posters, pamphlets, videos, and other campaign materials are often not systematically or transparently archived. As a result, these promises tend to fade away over time, making it difficult to hold the elected candidates accountable for their commitments. To address this persistent issue, this research aims to develop a system capable of structuring the commitments made by candidates during election periods. In this work, we created the first gold-standard dataset of Nepali political texts (Cohen's kappa = 0.777) and evaluated machine learning, deep learning and large language models to automatically extract and analyze election commitments. LaBSE-LR achieved the best results among machine learning models (0.897 accuracy, 0.896 macro F1-score, 0.947 AUROC), while fastText-LSTM achieved the highest performance among deep learning models (0.914 accuracy, 0.913 macro F1-score, 0.961 AUROC). Among the three evaluated large language models, Qwen3-8B performed best in the few-shot setting (0.848 macro F1-score). The outcome of this work is integrated into a mobile application, where commitments can be visualized with the support of GAI, enhancing transparency, civic engagement, and electoral accountability.

1. Introduction

Nepali electoral commitments are difficult to track because campaign materials are rarely archived systematically, limiting citizens’ ability to assess whether promises are fulfilled. The paper addresses this gap with a gold-standard dataset, comparative commitment-classification models, and a mobile application that presents commitments with GAI-supported visualizations.

1. Introduction

  • Campaign commitments are rarely preserved or systematically archived in Nepal, making it difficult for citizens to evaluate fulfillment and hold elected representatives accountable.Commitments appear in materials including manifestos, posters, pamphlets, and videos.Source: §1 · pp. 1–2
  • Here, “low-resource” refers mainly to limited computational and annotated resources, not to the number of Nepali speakers.The paper notes fewer large, diverse corpora, annotated datasets, translation resources, evaluation benchmarks, and pretrained language models than for high-resource languages.Source: §1 · p. 2
  • The paper creates an annotated gold-standard dataset and benchmarks ML, DL, and LLM models to extract and analyze commitments in candidates’ publicity materials.It also uses GAI to create visual representations and delivers them with commitment summaries through a mobile application.Source: §1 · pp. 2–3

2. Literature Reviews

The literature spans political communication, electoral accountability, speech-act theory, and computational political-text analysis. It identifies limited computational research on Nepali political texts and motivates an operational definition of commitments that centers on specific, verifiable future actions.

2. Literature Reviews

  • Nepal-focused studies examine election communication and accountability, including Facebook use in the 2022 local elections and citizen-representative assessments in two districts.Paudel and Pahari (2020) report that accountability remains a work-in-progress and that citizens were neither clearly satisfied nor dissatisfied overall.Source: §2 · p. 3
  • Prior political-science work treats manifestos as tools for accountability and examines localized pledges and how commitments relate to governance and public expenditure.Dahal (2001) argues for civic education and support mechanisms to help voters make informed choices.Source: §2 · pp. 3–4
  • The study defines a commitment narrowly as a specific future action or outcome that can be evaluated as fulfilled or unfulfilled, adding specificity and verifiability criteria to commissive speech acts (Searle 1969, 1976).It distinguishes a statement’s commitment at the time it is made from whether the candidate later fulfills it.Source: §2 · p. 4
  • Computational political-text research covers tasks including election prediction, ideology classification, stance detection, and topic modeling, while studies also compare traditional ML and DL approaches (O’Connor et al. 2010; Hirst, Riabinin, and Graham 2010; Thomas, Pang, and Lee 2012; Mohd Bahrin and Jantan 2024).Fagni and Cresci (2022) describe an unsupervised representation-learning and clustering approach to political-leaning inference.Source: §2 · pp. 4–5
  • Despite research on Nepali NLP tasks such as translation, speech recognition, and classification, computational mining and classification of Nepali political texts remain little studied.The review identifies computational analysis of electoral commitments in Nepal as a research gap.Source: §2 · p. 5

3. Methods and Materials

The system structures Nepali electoral commitments through data collection, preprocessing, expert-checked annotation, and comparative model evaluation, then turns extracted commitments into visuals and a mobile application. The methods span collection from multiple sources, construction of a 2,430-instance labeled dataset, and tools intended to present candidate commitments to the public.

3.1 Data Collection

  • Data collection drew on candidate manifestos, pamphlets, videos, and social media across five constituent bodies; party offices lacked systems to archive or manage candidates’ manifestos.The study collected 119 images and 29 videos and searched 37 Nagarpalikas, covering 45 political individuals in the 2022 local elections; only two of five visited party offices supplied party manifestos, not individual candidates’ manifestos.Source: §3.1 · pp. 5–6

3.2 Data Preprocessing

  • Preprocessing converted pamphlet images to text, removed OCR artifacts and non-Devanagari content, segmented Nepali sentences, and standardized the resulting text.Audio and video were collected but not processed; data from only 28 of 45 candidates were processed.Source: §3.2 · p. 7

3.3 Annotation and Gold Standard Dataset Construction

  • Annotators labeled statements for specificity, actionability, and verifiability; an expert adjudicated final labels, with substantial inter-annotator agreement (κ = 0.777).The expert reclassified 82 vague commitment statements as non-commitments.Source: §3.3 · pp. 8–9

3.4 Data Distribution and Analysis

  • The final dataset contains 2,430 statements: 1,321 commitments and 1,109 non-commitments; no instances came from Provinces 4 or 6, while Province 3 contributed the most.The data were slightly skewed toward commitments.Source: §3.4 · p. 10

3.5.1 Machine Learning Models

  • The study benchmarked six classical classifiers using TF-IDF and LaBSE representations, recurrent and convolutional networks with fastText or GloVe embeddings, and fine-tuned mBERT and XLM-R.Three general-purpose LLMs were evaluated without task-specific fine-tuning in zero-shot and few-shot settings, with five commitment and five non-commitment examples in the few-shot prompts.Source: §3.5.1 · pp. 10–11, §3.5.2 · p. 11, §3.5.3 · p. 11

3.6 Generative AI for Visual Representation of Classified Commitments

  • For visual representation, the team translated selected Nepali commitments into English prompts for Stable Diffusion and iteratively revised prompts to improve semantic and visual alignment.Reviewers assessed 39 images for 11 candidates across all five provinces; adding constituency metadata improved relevance when a commitment lacked geographic context. The internal review was not a formal user study or expert evaluation.Source: §3.6 · p. 12

3.7 Mobile Application Development

  • The mobile prototype combines a map organized by province and district with candidate profiles, summarized commitments, and generated images, enabling users to explore candidate information by district.The app is presented as a platform for exploring commitments and performance and supporting civic engagement and accountability.Source: §3.7 · p. 13

4. Experiments and Results

The experiments compare machine-learning, deep-learning, and large-language models for political-text classification, using an annotated dataset and multiple evaluation metrics.

4. Experiments and Results

  • The experiments compare machine-learning, deep-learning, and large-language models on the annotated political-text classification task.The section reports experimental results, ablations, and error analysis.Source: §4 · p. 13

4.1 Experimental Setup

  • The 80% training split was stratified by province, district/gaupalika, sentence length, and final label; 155 reviewed non-commitment instances were added to address class imbalance.The remaining 486 instances were held out for final evaluation, and machine-learning and deep-learning models used five-fold cross-validation on the training split.Source: §4.1 · p. 15

4.2 Evaluation Metrics

  • Models were assessed using accuracy, macro recall, macro precision, macro F1-score, and AUROC.These measures cover different aspects of prediction performance.Source: §4.2 · pp. 15–16

4.3 Statistical Significance Testing

The study uses McNemar’s mid-p test to assess whether paired model-performance differences on the same test instances are statistically significant.

4.3 Statistical Significance Testing

  • McNemar’s mid-P test was selected for paired binary-label predictions and small or moderate numbers of discordant pairs, using α = 0.05.The test compares paired predictions on the same 486 test instances.Source: §4.3 · p. 16
  • The two-sided hypotheses test equal marginal performance (π = 0.5) against unequal performance, with π defined as the probability a discordant pair favors Model A.Under the null, the count of discordant pairs favoring Model A follows a binomial distribution with probability 0.5.Source: §4.3 · p. 16
  • If the mid-P value is at least 0.05, the test fails to reject the null and finds insufficient evidence that either model outperforms the other.For a significant result, the direction of superiority is determined by the discordant-pair counts.Source: §4.3 · p. 17

4.4 Results

Across the evaluated models, fastText-LSTM leads the deep-learning results, while Qwen3-8B is the strongest evaluated LLM; significance tests qualify several numerical rankings.

4.4 Results

  • 0.913 macro F1-score and 0.961 AUROC made fastText-LSTM the strongest evaluated model, exceeding the best LLM, Qwen3-8B, by +6.50% macro F1-score.LaBSE-LR led the machine-learning models at 0.896 macro F1-score; Qwen3-8B led LLMs at 0.848 macro F1-score in few-shot prompting.Source: §4.4 · p. 17, Table 5 · p. 17, Table 6 · p. 18
  • FastText-LSTM significantly outperformed fastText-CNN, XLM-R, TF-IDF-LR, and TF-IDF-SVM, but not fastText-SRN, fastText-GRU, LaBSE-LR, or LaBSE-SVM.The comparisons use McNemar’s mid-P test; Table 8 reports pairwise results.Source: §4.4 · p. 18, Table 8 · p. 19
  • Few-shot prompting significantly improved Qwen3-8B over zero-shot, while few-shot Qwen3-8B and Llama-3.1-8B-Instruct performed comparably.The cited tests report p = 0.0113 for Qwen3-8B’s prompting comparison and p = 0.683 for the two few-shot models.Source: §4.4 · p. 18

4.5 Ablation Studies

The ablations find that augmentation gains vary by embedding type, while corpus fine-tuning benefits fastText models more than LaBSE models.

4.5.1 Impact of Data Augmentation

  • +2.30% macro F1-score for TF-IDF-LR and +2.40% for fastText-LSTM were the largest augmentation gains; LaBSE-LR showed no improvement.The study treats gains of at least 1.00% macro F1-score as meaningful, roughly five correctly classified test instances.Source: §4.5.1 · p. 19, §4.5 · p. 19, Table 9 · p. 20
  • Data augmentation benefits varied more by embedding type than architecture: sparse embeddings gained, while LaBSE showed little sensitivity to augmentation for class-imbalance correction.LaBSE-SVM gained only +0.60% macro F1, below the stated meaningful-improvement threshold.Source: §4.5.1 · pp. 19–20

4.5.2 Impact of Fine‑tuning Pretrained Embeddings

  • Fine-tuning improved fastText-LSTM by 6.40% macro F1-score over its pretrained baseline, whereas LaBSE-based models showed no meaningful improvement.LaBSE-LR dropped 0.20% and LaBSE-SVM did not change; the table defines fine-tuned embeddings as further trained on the electoral commitment training corpus.Source: §4.5.2 · p. 20, Table 10 · p. 21

4.6 Error Analysis

Across four models, false negatives outnumber false positives, and the imbalance increases from classical embedding-based classifiers to larger multilingual models. Errors cluster around lexical cues, syntactic completeness, and information density.

4.6 Error Analysis

  • Qwen3-8B produced the fewest false positives but more than twice as many false negatives as LaBSE-LR; false negatives outnumbered false positives in every model.The imbalance grew from classical embedding-based classifiers to larger multilingual models.Source: §4.6 · p. 21, Figure 5 · p. 22
  • Lexical cues such as “encouragement” can trigger false positives when a statement sounds concrete but its main predicate lacks a measurable, deliverable commitment.All six test cases containing the cue “encouragement” were predicted as commitments, though the true label was non-commitment.Source: §4.6 · pp. 21–22
  • Telegraphic statements specifying concrete commitments can be misclassified as non-commitments when they lack a canonical finite future-tense form.This pattern appeared in 24 cases where at least two models misclassified the statement.Source: §4.6 · p. 22
  • Statements of 30 or more words can be misclassified when intervening clauses dilute the main predicate signal.The analysis reports six such long-sentence errors.Source: §4.6 · pp. 22–23

5. Discussion

The discussion finds that fastText-LSTM performs strongly on this Nepali political-text task and is substantially lighter to deploy than LaBSE-LR. It also relates model performance to representation choices, data adaptation, and recurring linguistic error patterns.

5. Discussion

  • 0.913 macro F1-score for fastText-LSTM led all configurations, statistically equivalent to LaBSE-LR’s 0.896 while requiring 19.54 MB versus 1,798.98 MB for inference.The authors identify fastText-LSTM as the more practical choice for resource-constrained deployment, including mobile or edge applications.Source: §5 · p. 23
  • LaBSE embeddings exceeded TF-IDF across ML model families, but the difference for the best LR and SVM models was not statistically significant.The discussion suggests that explicit lexical cues carry much of the discriminative signal in Nepali commitment mining.Source: §5 · p. 23
  • FastText-LSTM significantly outperformed both transformer models; the authors relate this to dataset size and Nepali morphology, which fastText character n-grams can capture.XLM-R slightly exceeded mBERT, but their difference was not statistically significant on this task and dataset.Source: §5 · p. 23
  • Augmentation benefited TF-IDF and domain-fine-tuned fastText but added little for LaBSE; fastText domain adaptation improved all its architectures, whereas LaBSE fine-tuning did not meaningfully help.The authors attribute the fastText gains to adaptation to electoral-manifesto vocabulary and phrasing.Source: §5 · pp. 24–25
  • Qwen3-8B improved significantly with few-shot prompting over zero-shot, yet neither it nor Llama-3.1-8B-Instruct matched lightweight fine-tuned models.The authors conclude that a few in-context examples do not capture the lexical range available through labeled, domain-specific fine-tuning.Source: §5 · p. 25
  • Errors across model families cluster around lexical cues, syntactic completeness, and information density, including cases that challenged human annotators.The accompanying table caption identifies representative error cases grouped by linguistic factor.Source: Table 11 · p. 24, §5 · p. 25

6. Limitations

The study’s limitations concern dataset size, evaluation scope, the prototype’s readiness, and the coverage of collected political materials. These boundaries should inform interpretation of its model and application results.

6. Limitations

  • 2,585 training instances may have limited performance, especially for deep learning and transformer-based models.The study notes these models usually need more data to perform well.Source: §6 · p. 25
  • Generated images received internal evaluation only, and the small subset precluded systematic prompt-engineering ablation.The authors instead informally compared prompt variants.Source: §6 · pp. 25–26
  • The study did not establish a human performance baseline for commitment classification.Annotator agreement of κ = 0.777 is an indirect measure; independently classifying held-out test instances would provide a more direct upper bound.Source: §6 · p. 26
  • The mobile application remains a working prototype without formal usability testing among voters or users.Source: §6 · p. 26
  • Ablation and error analyses covered representative embedding families and architectures, not every evaluated model.Consequently, some models’ behavior remains unexplored.Source: §6 · p. 26
  • Online sources and manual field visits may underrepresent informal or grassroots commitments, while audio and video were not processed.Processing audio and video was outside the study’s scope.Source: §6 · p. 26

7. Ethical Considerations

The authors describe potential subjectivity in human annotation and limits in the dataset’s coverage. They also caution that generated images and imperfect classification can be misread, and that app visibility reflects data availability rather than candidate performance.

7. Ethical Considerations

  • Blind adjudication and two independent annotators helped make labels content-based, but subjective bias cannot be fully ruled out.The authors note possible differences in interpretation, prior beliefs, or perspectives.Source: §7 · p. 26
  • The dataset is unbalanced across parties and provinces, so results should not be read as direct evaluations of candidates or parties.Its composition depended on publicly accessible campaign materials.Source: §7 · p. 26
  • The authors did not explicitly account for source-platform terms; future large-scale collection should check policies or obtain permission.The data were collected manually from publicly accessible profiles for research purposes.Source: §7 · p. 26
  • GAI images are illustrative, and classifier false negatives or false positives mean prototype outputs are not final or verified judgments about candidates.Source: §7 · p. 26
  • Because only some candidates and provinces are covered, app visibility reflects data availability, not actual performance.The authors aim to address coverage through wider future data collection.Source: §7 · p. 27

8. Conclusion and Future Work

The paper structures Nepali candidates’ electoral commitments using a gold-standard dataset and comparative model analysis, with fastText-LSTM reported as the strongest model for performance and deployment. A mobile app presents commitment summaries with GAI images, while future work targets broader data coverage and application testing.

8. Conclusion and Future Work

  • Comparative analysis identifies fastText-LSTM as the best model in both performance and deployment aspects.Source: §8 · p. 27
  • The study contributes a gold-standard dataset and a system to identify, structure, and analyze candidates’ electoral commitments.Source: §8 · p. 27
  • The mobile application provides concise candidate-commitment summaries supported by GAI images for interpretability.The authors present it as a demonstration of practical implications.Source: §8 · p. 27
  • Future work plans broader, context-rich multimedia collection, local field documentation across candidates, and robust application testing with voters and users.The authors aim to improve dataset coverage and reliability.Source: §8 · p. 27

Appendix 1. Model Configuration and Hyperparameter Details

The appendix reports optimal configurations for the study’s machine-learning and deep-learning models. It also specifies shared training choices, fine-tuning of pretrained representations, and text tokenization.

Appendix 1. Model Configuration and Hyperparameter Details

Related papers

Loading 2610.32150v1…