Source-linked AI summary
SemEval-2026 Task 6: CLARITY -- Unmasking Political Question Evasions
Konstantinos Thomas, Giorgos Filandrianos, Maria Lymperaiou, Chrysoula Zerva, Giorgos Stamou
TL;DR
Political speakers can preserve the appearance of responsiveness while evading direct answers, yet this strategic behavior remains underexplored computationally. CLARITY introduces a taxonomy-based shared task with coarse clarity and fine-grained evasion classification, finding that LLM prompting and hierarchical reasoning are especially effective while fine-grained evasion remains difficult.
Problem
Political question evasion is underexplored in NLP despite its importance for understanding strategic ambiguity in political discourse.
Method
CLARITY evaluates clarity-level classification and nine-category evasion-level classification on question–answer pairs from U.S. presidential interviews using an expert-grounded taxonomy.
Results
Clarity classification reached 0.89 Macro-F1, while evasion classification reached 0.68 Macro-F1 and remained harder for systems to solve.
Takeaways & Limitations
LLM prompting and hierarchical decomposition were the most consistent characteristics of strong systems across both subtasks.
Takeaways & Limitations
The dataset covers English-language U.S. presidential interviews, limiting direct generalization to other languages, political systems, and communicative settings.
Abstract
from arXiv · showhide
Political speakers often avoid answering questions directly while maintaining the appearance of responsiveness. Despite its importance for public discourse, such strategic evasion remains underexplored in Natural Language Processing. We introduce SemEval-2026 Task 6, CLARITY, a shared task on political question evasion consisting of two subtasks: (i) clarity-level classification into Clear Reply, Ambivalent, and Clear Non-Reply, and (ii) evasion-level classification into nine fine-grained evasion strategies. The benchmark is constructed from U.S. presidential interviews and follows an expert-grounded taxonomy of response clarity and evasion. The task attracted 124 registered teams, who submitted 946 valid runs for clarity-level classification and 539 for evasion-level classification. Results show a substantial gap in difficulty between the two subtasks: the best system achieved 0.89 macro-F1 on clarity classification, surpassing the strongest baseline by a large margin, while the top evasion-level system reached 0.68 macro-F1, matching the best baseline. Overall, large language model prompting and hierarchical exploitation of the taxonomy emerged as the most effective strategies, with top systems consistently outperforming those that treated the two subtasks independently. CLARITY establishes political response evasion as a challenging benchmark for computational discourse analysis and highlights the difficulty of modeling strategic ambiguity in political language.
1 Introduction
CLARITY addresses the underexplored problem of political question evasion by organizing it into coarse response-clarity and fine-grained evasion classification. The shared task attracted broad participation, and leading systems showed strong gains on clarity classification while relying heavily on LLM prompting.
- Political evasion avoids direct answers while preserving conversational legitimacy and limiting accountability or commitment.
- 39–46% of questions receive explicit answers from politicians in televised interviews, compared with 70–89% for nonpoliticians.
- CLARITY enables systematic comparison of computational methods for question-evasion classification, building on an existing evasion taxonomy.
- CLARITY evaluates three clarity labels and nine fine-grained evasion techniques using Macro-F1 across both subtasks.
- 124 teams registered and submitted 946 valid clarity-level prediction sets and 539 evasion-level prediction sets.
- The winning clarity system improved Macro-F1 from 0.82 to 0.89, while seven of 33 evasion submissions exceeded the 0.57 baseline.
2 Related work
Related work frames political evasion as strategic ambiguity and distinguishes it from ordinary answerability or argument-fallacy analysis. Prior research has examined political response intent, quality, ambiguity, and evasion patterns across several datasets and settings.
- Political communication research treats under-specification, ambiguity, verbosity, and evasion as strategic responses to accountability and public pressure.
- Political QA research has studied response intent, answer quality, informativeness, and large-scale collections of congressional and other political exchanges.
- Political discourse analysis increasingly evaluates whether answers provide requested information rather than only modeling uncertainty in the questions.
- Political settings exhibit deliberate non-answerability and strategic ambiguity through evasion patterns that govern interviews.
3 Data
CLARITY is built from English-language U.S. presidential interviews, decomposed into question–answer pairs and annotated by trained annotators with expert oversight. Agreement is higher for coarse clarity labels than for several fine-grained evasion distinctions.
- Dataset construction: The dataset contains 287 interviews from four U.S. presidents, producing 3,756 question–response pairs across train and validation splits.
- Dataset construction: ChatGPT decomposed multi-barreled interview questions into sub-question and subresponse pairs, which annotators checked against the original interviews.
- Annotation: Three annotators and a political-science expert validated decompositions, assigned taxonomy labels, and added missing question–answer pairs.
- Annotation: Annotators received category training, quizzes, expert checks, and counterfactual samples to monitor comprehension and annotation drift.
- Annotation agreement: 317 validation pairs received overlapping annotations, with Fleiss κ used to measure agreement and majority voting used for gold labels.
- Annotation agreement: Clarity agreement ranged from κ=0.65 to 1.00, including near-perfect agreement of κ=0.97 between Clear Reply and Clear Non-Reply.
- Annotation agreement: Fine-grained evasion labels were harder to distinguish, with agreement as low as κ=0.43 for General versus Implicit.
- Evaluation data: The evaluation set contains 237 question–answer pairs from 43 interviews and was annotated by two annotators per pair.
4 Task Description
CLARITY evaluates response clarity and specific evasion mechanisms on the same question–answer pairs. The two subtasks use taxonomy-based classification and Macro-F1, with published fine-tuned Llama 70b results serving as baselines.
- Participants classify textual question–answer pairs in either or both subtasks using Codabench for development and evaluation.
- Subtask 1: Clarity-level Classification: Subtask 1 assigns each response one of three clarity categories describing how clearly it addresses the question.
- Subtask 2: Evasion-level Classification: Subtask 2 assigns each response one of nine fine-grained categories describing the mechanism used to avoid a direct answer.
- Evaluation: The subtasks use Macro-F1 over k categories to evaluate balanced performance across classes.
- Evaluation: A prediction is considered correct when it matches any annotator’s label.
- Baselines: Published baselines include prompting and tuned approaches, with the strongest variants summarized in Table 2.
5 Participating Systems and Results
Participating systems predominantly used LLM prompting, often combined with hierarchical task coupling, while encoder-only systems required substantial engineering to remain competitive. The strongest clarity-level systems reached 0.89 Macro-F1, and most top systems exploited the relationship between clarity and evasion classification.
- 29 teams submitted systems for clarity-level classification, while 18 also attempted the more challenging fine-grained evasion task.
- 0.89 Macro-F1 was achieved by the top clarity-level team, surpassing the 0.82 fine-tuned baseline.
- All five top-ranked clarity systems, scoring 0.82–0.89, primarily relied on LLMs with sophisticated prompting strategies.
- Few-shot prompting with Chain-of-Thought reasoning was nearly universal among top performers, whereas zero-shot and instruction-only prompting were weakest.
- Four of the top seven systems predicted evasion labels first and mapped them upward through the taxonomy to derive clarity predictions.
- Approximately 14 teams used encoder-only models, but the best purely encoder-based system reached 0.81 only after combining NLI reframing, multi-task learning, GNNs, and Multiple Instance Learning.
5.3 Subtask 2 - Evasion-level
Evasion-level classification was substantially harder than clarity-level classification because nine semantically overlapping categories had to be distinguished with limited data. The strongest systems addressed this difficulty through confidence-aware multi-stage pipelines, hybrid encoder–LLM designs, and hierarchical decomposition.
- 5.3.1 Overview and Participation: 0.68 Macro-F1 was the highest evasion-level score, compared with 0.89 for clarity-level classification, while the median evasion score was approximately 0.52.
- 5.3.2 Dominant Approaches: Three-Stage Pipeline & Hybrid Encoder-LLM: TeleAI achieved 0.68 by decomposing nine-class prediction into three sequential stages rather than classifying all categories in one pass.
- 5.3.2 Dominant Approaches: Three-Stage Pipeline & Hybrid Encoder-LLM: Confidence-conditional routing directed the most confusable cases to contextualised prompting and helped separate categories such as Dodging versus Deflection.
- 5.3.2 Dominant Approaches: Three-Stage Pipeline & Hybrid Encoder-LLM: The rank-two hybrid system used RoBERTalarge to shortlist five candidates before Kimi-K2 performed constrained few-shot inference, then inferred clarity hierarchically.
- 5.3.3 Hierarchical Decomposition & Encoder Limitations: Hierarchical decomposition consistently outperformed direct nine-class inference by reducing output spaces, routing uncertain cases, or assigning deterministic classes early.
- 5.3.3 Hierarchical Decomposition & Encoder Limitations: 0.50 Macro-F1 was the best purely encoder-based result, while heavily engineered encoder pipelines remained below 0.51 and were outperformed by simpler LLM-prompted approaches.
- 5.3.5 Score Compression and Task Difficulty: The 0.68 Macro-F1 ceiling reflects nine-class confusion, semantically proximal categories, modest training data, and lower fine-grained annotation agreement.
6 Discussion and Conclusion
CLARITY results show that hierarchical, LLM-based approaches are especially effective for clarity classification, while fine-grained evasion classification remains substantially harder. The task also motivates richer, multilingual, multimodal, and context-sensitive research on political evasion.
- Results: LLM-based approaches substantially outperformed fine-tuned encoder models on clarity-level classification.All top-ranked systems used LLMs with sophisticated prompting pipelines.
- Results: Hierarchical decomposition was the most consistent differentiator between strong and weak systems across both subtasks.The results treat the taxonomy as an active source of inductive bias rather than merely an organizational convenience.
- Secondary findings: Prompt design, including confidence-conditional routing and dynamic few-shot selection, mattered more than model size or model choice alone.CoT distillation also offered a compute-efficient alternative to proprietary API inference.
- Future directions: Future work could add intermediate taxonomy-level evaluation, richer discourse context, and prosodic or gestural cues from video interviews.The dataset and systems also provide a foundation for studying political communication across languages, political systems, and interview contexts.
Limitations
The benchmark’s conclusions are bounded by its English-language U.S. presidential-interview setting, subjective fine-grained annotations, and text-only representation of political evasion.
- Scope: The dataset covers English-language U.S. presidential interviews from the official White House archive, limiting transfer to other settings.The authors identify other languages, political systems, parliamentary debates, press briefings, and campaign rallies as boundaries.
- Annotation: Annotation necessarily involves subjective judgment, especially for fine-grained strategies where several label pairs show lower agreement.Training, expert supervision, overlapping annotation, and agreement analysis were used to mitigate this issue.
- Modality: The benchmark uses textual information alone, excluding prosody, hesitation, timing, facial expressions, gestures, and interactional dynamics.This captures only the textual component of evasiveness and may omit complementary or contradictory cues in natural interviews.
Ethical Considerations
CLARITY is intended to analyze response clarity and evasion as discourse phenomena, not to judge speakers’ honesty, intent, or moral character. Its labels reflect structured human judgments, and model outputs require contextualized human oversight.
- The benchmark analyzes political response clarity and evasion rather than making definitive judgments about speakers’ honesty, intent, or moral character.
- Labels should be treated as structured human judgments under a specific taxonomy, not as indisputable ground truth.Annotator training, expert supervision, and agreement analysis were used to mitigate subjectivity.
- Automatic evasion predictions should not serve as stand-alone evidence about political actors in public or monitoring contexts.Practical deployment should preserve human oversight, account for context, and avoid simplistic or partisan conclusions.
A.1 Fleiss κ
Fleiss’ kappa is computed from annotator assignments across 317 items, using per-item agreement, mean observed agreement, class proportions, and chance agreement. The configuration covers three clarity categories and nine evasion categories.
- 317 items were annotated by 3 annotators each across 3 clarity categories or 9 evasion categories.For each item and category, n_ij denotes the number of annotators assigning the item to category j.
- Per-item agreement is calculated from the category assignment counts for each item.
- Mean observed agreement, class proportions, and expected chance agreement are computed as intermediate quantities.These quantities are introduced before the final Fleiss’ kappa calculation.
- Fleiss’ kappa is then computed from the observed and expected agreement quantities to evaluate annotators’ responses.