PaperlayerView exact revision on arXiv

Source-linked AI summary

PolERo: Studying Political Evasion in Romanian

Gabriel Stefan, Sergiu Nisioi

arXiv:2609.02391v1cs.CLcs.AI

TL;DR

Response-clarity and evasion classification lack resources beyond English, leaving cross-lingual and cross-context transfer uncertain. The paper introduces PolERo and evaluates matched classifier, encoder, multi-head, and prompting approaches across Romanian and English. Fine-tuned encoders are competitive, but transfer is asymmetric and pragmatic ambivalent categories remain difficult; the study covers only one non-English language.

  • Problem

    Response-clarity and evasion classification are largely limited to English political interviews, leaving transfer across languages and political contexts underexamined.

  • Method

    The paper introduces 3,574 annotated Romanian question-answer pairs and compares multiple model families under matched English–Romanian conditions, including a chunked multi-head encoder.

  • Results

    Fine-tuned encoders are competitive, with multi-head gains of +0.097 macro-F1 on English clarity, +0.049 on English evasion, +0.043 on Romanian clarity, and +0.026 on Romanian evasion.

  • Takeaways & Limitations

    Ambivalent evasion categories involving pragmatic cues remain the main challenge across model families, while cross-lingual transfer is asymmetric.

  • Takeaways & Limitations

    PolERo covers a single non-English language, so generalization to other languages and political contexts remains an open question.

Abstract

from arXiv · show

Political evasion refers to responses that engage with a question while withholding the requested information. Recent NLP work frames political evasion as a classification task using a two-level taxonomy of response clarity and fine-grained evasion strategies. Existing work on response clarity and evasion classification is limited to English, leaving open whether the taxonomy and model behavior transfer across languages and political contexts. We introduce PolERo, a dataset of 3,574 human-annotated question-answer pairs extracted from official transcripts of five Romanian presidents. We evaluate multiple classification approaches on both datasets under matched conditions, including TF-IDF baselines, fine-tuned encoder models, a proposed sliding-window encoder, and zero/few-shot LLM prompting. We study cross-lingual transfer through joint bilingual training and machine-translation-based data augmentation. Our results indicate that fine-tuned encoders are competitive, cross-lingual transfer is asymmetric, and ambivalent evasion categories involving pragmatic cues remain the main challenge across all model families.

1 Introduction

Response clarity and political evasion remain underexplored NLP problems, with existing resources concentrated in U.S. English despite their dependence on political and institutional context. PolERo addresses this gap with a Romanian dataset and matched multilingual evaluation.

  • Motivation: Response clarity asks whether an answer adequately addresses a question, extending beyond question answerability, responder intent, and broader political discourse analysis.The problem is relevant to dialogue understanding, question answering, and user-facing technologies.
  • Research gap: Existing response-clarity evidence is restricted largely to U.S. English political interviews, with no Romanian resource targeting clarity or evasion classification.The taxonomy’s transfer across languages and political contexts is therefore left open.
  • Dataset contribution: PolERo contains 3,574 human-annotated question-answer pairs from official transcripts covering five Romanian presidents.It is presented as the first such resource for a non-English language.
  • Dataset contribution: The dataset adopts the exact annotation protocol and two-level taxonomy used for the English benchmark, enabling controlled English–Romanian comparison.This supports comparison of response clarity and fine-grained evasion classification under matched conditions.
  • Evaluation scope: The study compares traditional classifiers, encoder models, a multi-head long-text encoder, and zero- and few-shot LLM prompting across both datasets.It also examines cross-lingual transfer in low-data settings and machine-translation-based data augmentation.

2 Dataset and Annotation

PolERo is built from Romanian presidential transcripts, filtered to one question-response pair per instance and annotated with a translated two-level evasion taxonomy. The resulting corpus is predominantly Clear Reply, while evasion labeling is substantially harder than clarity labeling.

  • Data collection: The corpus spans official press conferences, interviews, and declarations from five Romanian presidents over 25 years.The transcripts cover 2001–2026 and are collected from presidential websites.
  • Data preprocessing: 2,053 multi-part entries and 607 entries without an extractable question were removed to enforce one-to-one question-response mapping.The retained corpus contained no remaining multi-part questions according to annotator checks.
  • Taxonomy: The taxonomy assigns three clarity labels and nine fine-grained evasion categories, with fine-grained annotations mapped upward to clarity labels.Ambivalent includes implicit, general, partial, dodging, and deflection responses; Clear Non-Reply includes explicit refusal, ignorance, and clarification-related categories.
  • Annotation: An expert translator adapted the English taxonomy for Romanian, while annotators trained with worked examples, a 15-item screening quiz, and group reviews of boundary cases.Each instance received one evasion label, and the three-way clarity label was inferred through the taxonomy.
  • Dataset statistics: 55.09% of PolERo responses are Clear Reply, whereas 59.80% of the English corpus is Ambivalent.Romanian and English therefore differ in their overall clarity-label distributions.
  • Dataset statistics: Romanian data discard multi-part questions, unlike the English data, which decompose interviews into separate sub-questions using ChatGPT.This preprocessing difference is an important qualification for cross-dataset comparisons.
  • Annotation quality: 95.3%, 95.3%, and 94.1% are the annotators’ clarity accuracies against expert labels, compared with 89.4%, 88.2%, and 84.7% for evasion.Test-set agreement is κ = 0.843 for clarity and κ = 0.678 for evasion, with the lowest category agreement for Deflection and General.

3 Experimental Setup

The experiments evaluate clarity and evasion separately and jointly across matched English and Romanian settings. They compare non-neural, fine-tuned encoder, multi-head, and prompted LLM approaches, including a chunked architecture for long question-answer pairs.

  • Task formulation: Macro-averaged F1 on each test split is the primary metric for three-class clarity and nine-class evasion classification.Mapped clarity is also computed by projecting predicted evasion labels to their parent clarity classes.
  • Model families: The comparison includes TF-IDF logistic regression, single-head encoder fine-tuning, a custom multi-head encoder, and zero- and few-shot LLM prompting.Models are trained and evaluated independently for English and Romanian under matched input formats, hyperparameters, and prompt templates.
  • Encoder models: Standard encoder experiments use RoBERTa-large, ModernBERT-large, ELECTRA-large, BERT-large-cased, and XLM-RoBERTa-large, with additional Romanian and multilingual models for PolERo.Romanian-specific additions include RoBERT-large and Romanian BERT, while mBERT-base supports multilingual comparison.
  • Multi-head architecture: The multi-head model shares an encoder, splits concatenated question-answer inputs into overlapping chunks, max-pools chunk representations, and predicts clarity and evasion jointly.Its two heads predict 3 clarity classes and 9 evasion classes using the unweighted sum of two cross-entropy losses.

4 Results and Discussion

Across matched English and Romanian evaluations, fine-tuned encoders perform competitively, with multi-head training improving both tasks and mapped clarity often outperforming direct prediction. LLMs are strong on fine-grained evasion, but performance varies, multilingual models trail language-specific backbones, and rare-category bias remains a limitation.

  • Baseline and encoder results: TF-IDF logistic regression reaches macro-F1 .422/0.453 on English/Romanian 3-class clarity, establishing a weak lexical baseline.The result supports the insufficiency of surface-level lexical features alone.
  • Baseline and encoder results: RoBERTa-large reaches .580 clarity and .481 evasion on English, while RoBERT-large reaches .504 evasion and .701 mapped clarity on Romanian.The Romanian-specific backbone is strongest on PolERo, whereas RoBERTa-large is strongest among English single-head encoders.
  • Cross-lingual comparison: Language-specific backbones consistently outperform XLM-RoBERTa-large on their respective languages, indicating a trade-off between within-language performance and cross-language coverage.RoBERT-large exceeds XLM-RoBERTa-large on Romanian metrics, while RoBERTa-large does so on English benchmarks.
  • Multi-head architecture: +0.097 English clarity, +0.049 English evasion, +0.043 Romanian clarity and +0.026 Romanian evasion separate the complete multi-head architecture from single-head baselines.The joint objective provides the largest individual clarity gains, while cross-validation and ensembling account for most evasion improvement.
  • Mapped versus direct clarity: Mapped clarity generally outperforms direct clarity prediction when fine-grained evasion categories are predicted first and then projected through the taxonomy.This pattern holds for the main monolingual encoders and nearly every LLM configuration.
  • Limitations and error patterns: The mapping strategy fails for four encoders when evasion predictions collapse onto majority categories, amplifying bias against rare classes.ELECTRA-large predicts Explicit for every Romanian instance, producing macro-F1 .084 and mapped clarity .24.
  • LLM prompting: LLMs obtain some of the best results across metrics, but their task-specific rankings vary and they can be substantially weaker than encoders.Gemma-4-31B-it leads English few-shot direct clarity, Qwen3.6-35B-A3B leads Romanian few-shot evasion, and GPT-5.4 is strong on mapped clarity.
  • LLM prompting: Few-shot examples usually improve mapped clarity by clarifying similar category boundaries, but Llama-3.3-70B instead drops from .624 to .564 on English.The model becomes heavily biased toward Deflection, predicting it for 79.5% of English test instances and 44% of Romanian predictions.

5 Cross-Lingual Transfer

Cross-lingual transfer is asymmetric: English-trained models generalize better to Romanian than Romanian-trained models generalize to English, while joint multilingual training improves Romanian performance. Machine translation partially recovers clarity but offers limited evasion gains, especially for pragmatically difficult categories.

  • Cross-lingual transfer: EN →RO causes a 6-to-7-point performance drop, whereas RO →EN causes an 11-to-14-point degradation for XLM-RoBERTa-large and RoBERTa-large.The mapped score falls from .694 to .635 in the EN →RO direction.
  • Cross-lingual transfer: Romanian-trained models overpredict Clear Reply on English Ambivalent answers and map many Implicit and Deflection responses to Explicit.The Romanian-trained model misclassifies 54% of English Ambivalent answers as Clear Reply.
  • Multilingual augmentation: Joint EN+RO training with XLM-RoBERTa-large achieves the highest Romanian fine-tuned evasion score, 0.570.Joint training may reinforce rare Romanian categories through similar English examples; Declining to answer reaches perfect recall on Romanian.
  • Machine translation: Machine-translating English data into Romanian partially recovers clarity performance but provides limited gains for evasion detection.Deflection and Dodging degrade most under machine-translation augmentation, while Clarification and Declining to answer translate more reliably.
  • Experimental comparison: Table 3 compares training-testing directions and translated-data configurations, with joint multilingual training outperforming monolingual training on translated data when data are available.Its rows identify best models by language and task, while EN2RO and RO2EN denote translated training sets.

6 Remarks

The authors recommend benchmarking few-shot LLM prompting in low-resource settings, using translated data and cross-lingual encoders when needed, and exploiting sufficiently large target-language datasets with monolingual encoders.

  • Recommendations: For a new low-resource language, first benchmark few-shot prompting with available LLMs, preferably open-weight models.This recommendation is framed for dataset creation in languages such as Romanian.
  • Recommendations: If LLMs perform poorly, translated datasets and cross-lingual encoders may provide a viable alternative.The authors caution that representativeness, geopolitical factors, and local histories require careful consideration.
  • Recommendations: With at least 3,000 target-language samples and available monolingual pretrained encoders, a strong strategy is to use them for classification.The supplied passage introduces this threshold-based recommendation but does not include its full continuation.

7 Conclusions

PolERo contributes a Romanian benchmark aligned with an existing English taxonomy and compares diverse classifiers under matched conditions. Fine-tuned encoders can be competitive with LLMs for clarity, but larger models perform better on fine-grained evasion, and cross-lingual transfer remains asymmetric.

  • Contributions: PolERo contains 3,574 Romanian presidential-transcript question-answer pairs annotated with a two-level evasion taxonomy.The study compares PolERo with an existing English dataset under matched conditions.
  • Model findings: Sliding-window multi-head encoders make fine-tuned encoders competitive with LLMs on three-class clarity classification while using less memory.The architecture jointly predicts clarity and evasion.
  • Model findings: Larger models achieve better predictive performance on the nine-class fine-grained evasion task.This contrasts with the stronger encoder competitiveness reported for clarity classification.
  • Cross-lingual findings: English-trained models generalize better to Romanian than Romanian-trained models generalize to English, while joint multilingual training performs well for Romanian.Ambivalent categories remain difficult because they depend on pragmatic intent rather than surface form.

8 Limitations

The study’s evidence comes from one non-English language, and annotation procedures leave uncertainty because training instances usually receive only one annotator and communicative intent can vary across annotators.

  • Scope: PolERo covers only one non-English language, leaving generalization to other languages and political contexts unresolved.The authors identify transfer of the taxonomy, annotation difficulty, and model behavior as open questions.
  • Annotation: Training instances receive single annotation while triple annotation is reserved for the test split.The authors note that full triple annotation by domain experts could reduce uncertainty but would cost substantially more.
  • Annotation: Interpretation of evasion may vary because annotation requires communicative-intent judgments and annotators have psychology rather than political-science or linguistics training.This may affect borderline cases.

9 Ethical Considerations

The annotation process used official public presidential transcripts, structured review procedures, and safeguards for difficult cases. The paper also cautions against treating automated labels as definitive judgments about speakers.

  • The corpus uses official statements from Romanian presidents and excludes private communications, personal data, and nonpublic records.
  • The authors discourage using PolERo as a standalone accountability instrument because human annotators disagree on some instances and model predictions are not ground truth.
  • Annotators classified evasion from original texts while separately verifying GPT-5.4-generated summaries, including 60 intentionally incorrect summaries as attention checks.
  • Annotators received task training, consulted taxonomy examples, and participated in expert-led review sessions for disagreements and boundary cases.
  • 87.2% of clarity items received unanimous agreement, compared with 69.6% for evasion; complete disagreement occurred in 2.4% of evasion items.
  • Agreement was weakest for Deflection, including Deflection–Dodging κ = 0.26, reflecting closely related evasion categories.

B.3 Expert Benchmark Evaluation

Expert benchmarking shows that annotators were more accurate on the three-class clarity task than on the nine-class evasion task. The appendix also describes substantial label imbalance and distributional differences between PolERo and CLARITY.

  • Annotator accuracy against an 85-instance expert benchmark was 94.1–95.3% for clarity and 84.7–89.4% for evasion.
  • Clarity accuracy exceeded evasion accuracy by 5–10 percentage points for every annotator, consistent with the greater difficulty of the nine-class task.
  • Explicit answers comprise 55.09% of PolERo, while Clarification accounts for only 1.09% of question-answer pairs.
  • Restricting CLARITY to single-question instances reduces Ambivalent from 59.8% to 51.7% and increases Clear Reply from 30.1% to 36.1%.
  • Speaker coverage is uneven: Băsescu and Iohannis account for 64.8% of the corpus, while Bolojan contributes 56 pairs.

C.3 Distribution by Communicative Context

PolERo’s response distributions vary across communicative contexts, topics, speakers, and years. Interviews contain more direct answers, while formal contexts show more deflection and declining-to-answer responses, with topic composition contributing to these differences.

  • Communicative context: Interviews contribute 1,904 pairs, declarations 1,213, and press conferences 457.
  • Communicative context: Interviews have the highest Clear Reply rate at 63.03% and the lowest Clear Non-Reply rate at 3.62%.
  • Topics: Topic composition partly explains interview clarity: Personal & Others forms 38.39% of interview questions and has a 65.62% Clear Reply rate.
  • Evasion by context: Interviews contain more Dodging at 10.45%, whereas declarations and press conferences show higher Declining to answer and Deflection rates.
  • Evasion by context: Among declarations and press conferences, Explicit responses fall to 46.05%, while Deflection reaches 8.38% and Declining to answer 7.79%.
  • Topics: Justice & Anti-corruption has the highest Clear Non-Reply rate at 15.05%, despite comprising only 7.81% of the dataset.
  • Temporal trends: Clear Non-Reply rates stay below 6% in 2001–2007 but rise to approximately 10–20% in 2016–2026, alongside increasing Ambivalent responses.
  • Length and verbosity: Answer length does not align with evasion: Iliescu is verbose yet predominantly Explicit, while Dan is concise and also predominantly Explicit.

E.2 Component Ablations

Ablations show that chunk aggregation, joint training, and larger ensembles improve or preserve performance, while alternative losses do not. Seed variation is smaller than reported differences between model families, and lexical baselines struggle particularly with minority and ambiguous classes.

  • Chunk aggregation: Max-pooling outperforms mean-pooling and first-chunk aggregation, whose truncation misses information beyond 512 tokens for 28.8% of English responses.
  • Multi-task versus single-task: Joint training matches single-task clarity performance and improves single-task evasion by +0.03 macro-F1.
  • Ensemble size: Increasing the ensemble from 3 to 7 folds consistently improves both subtasks, with larger gains on the more difficult evasion task.
  • Loss function: Neither class-weighted cross-entropy nor focal loss improves macro-F1, although class weighting raises minority-class recall for selected labels.
  • Seed variation: Test-set variation across three seeds is smaller than differences between model families, with clarity scores of .676±.019 on English and .743±.023 on Romanian.
  • TF-IDF baseline: The TF-IDF baseline achieves macro-F1 of 0.42 on English and 0.45 on Romanian but poorly identifies Clear Non-Reply, with F1 values of 0.25 and 0.26.
  • Error patterns: Ambivalent responses are frequently confused with Clear Reply, while brief minority responses produce weak term-frequency signals and are underpredicted.

F.2 Encoder Models

Multi-head and in-language encoders perform strongly, but fine-grained evasion remains difficult where categories depend on pragmatic or relational cues. Cross-lingual transfer and translation-based augmentation help unevenly, while ambivalent categories remain the main challenge.

  • Encoder Models: RoBERT-large is strongest on PolERo, while XLM-RoBERTa-large is the strongest multilingual model but underperforms on fine-grained Romanian evasion.The Romanian model benefits from Romanian pretraining and tokenization.
  • Multi-head architecture: Multi-head models consistently outperform single-head models across all reported metrics, with the strongest results from in-language RoBERTa-large and RoBERT-large.XLM-RoBERTa-large gains relatively more from the architecture but still underperforms monolingual encoders.
  • Error Analysis: 285 English and 215 Romanian errors occur along the Implicit–Explicit axis, where models must distinguish directly stated information from contextually inferred information.General-Explicit confusion rises from 146 English errors to 315 Romanian errors, becoming Romania’s largest error axis.
  • Error Analysis: Partial/half-answer is the hardest encoder category, with F1 values of 0.066 on English and 0.024 on Romanian.The category requires judging whether one part of a multi-part question is answered while another is ignored.
  • LLM Results: GPT-5.4 achieves the best mapped scores across both languages, while Qwen3.6-35B-A3B leads Romanian evasion under few-shot prompting.LLMs outperform encoder ensembles by a larger margin on evasion than clarity, although multi-head fine-tuning reduces that difference.
  • Cross-lingual Transfer: Translation recovers part of the mapped clarity gap but produces only small evasion gains because pragmatic cues separating ambivalent categories are distorted.On Romanian, Deflection recall falls from 41% to 10% and Dodging recall from 50% to 30%.
Loading 2609.02391v1…