Source-linked AI summary
SemEval-2015 Task 3: Answer Selection in Community Question Answering
Preslav Nakov, Lluís Màrquez, Walid Magdy, Alessandro Moschitti, James Glass, Bilal Randeree
TL;DR
Long cQA answer lists can make useful information time-consuming to find. SemEval-2015 Task 3 evaluates answer classification and English yes/no answer aggregation; top Arabic subtask A systems scored higher than English systems.
Problem
cQA questions may have hundreds of answers, many of which do not meet users' information needs, making useful information time-consuming to find.
Source: §1 · p. 1Method
The task evaluates three-way answer classification in English and Arabic, plus English-only yes/no/unsure aggregation based on good answers.
Source: §1 · p. 1Results
78.55 vs. 57.19 macro F1 for top Arabic and English subtask A systems, respectively, an absolute difference of over 21 points.
Source: §7 · p. 11Takeaways & Limitations
The organizers expect the task's lessons to aid cQA development and its released datasets to be useful beyond SemEval.
Source: §8 · p. 12Whether complex linguistically based representations and inference can work on informal, ungrammatical cQA forum text remains largely unanswered.
Source: §8 · p. 12
Abstract
from arXiv · showhide
Community Question Answering (cQA) provides new interesting research directions to the traditional Question Answering (QA) field, e.g., the exploitation of the interaction between users and the structure of related posts. In this context, we organized SemEval-2015 Task 3 on "Answer Selection in cQA", which included two subtasks: (a) classifying answers as "good", "bad", or "potentially relevant" with respect to the question, and (b) answering a YES/NO question with "yes", "no", or "unsure", based on the list of all answers. We set subtask A for Arabic and English on two relatively different cQA domains, i.e., the Qatar Living website for English, and a Quran-related website for Arabic. We used crowdsourcing on Amazon Mechanical Turk to label a large English training dataset, which we released to the research community. Thirteen teams participated in the challenge with a total of 61 submissions: 24 primary and 37 contrastive. The best systems achieved an official score (macro-averaged F1) of 57.19 and 63.7 for the English subtasks A and B, and 78.55 for the Arabic subtask A.
1 Introduction
Long cQA answer lists make useful information difficult to find, motivating a task that classifies answers and aggregates good answers for English yes/no questions. The challenge drew 13 teams and its best systems achieved the reported accuracy results across English, Arabic, and extreme summarization.
1 Introduction
- Answer lists can contain hundreds of responses, many loosely related or irrelevant, making it time-consuming to find information that meets a user's needs.Source: §1 · p. 1
- Subtask A classifies answers as good, potentially useful, or bad, while English-only subtask B aggregates good answers to produce yes, no, or unsure for yes/no questions.Subtask A is offered in both English and Arabic.Source: §1 · p. 1
- Prior answer-selection research included syntactic transformations, tree-edit methods, and CRFs, with several studies emphasizing syntactic structures (Severyn and Moschitti, 2012; Severyn and Moschitti, 2013).Source: §1 · p. 2
- The task used Qatar Living for English and the Fatwa website for Arabic; English data was manually annotated, Arabic data was built automatically, and all datasets were released publicly.For English, the task was decoupled from information retrieval to focus on learning relationships between text pairs.Source: §1 · pp. 1–2
- Almost all submissions outperformed baselines on the official F1-based score; best-system accuracy was about 73% for English, 83% for Arabic, and 72% for extreme summarization.The authors describe the Arabic task as easier.Source: §1 · p. 2
- Qatar Living planned to use evaluation experience and developed technology to improve products, including automatic search for useful comments.Source: §1 · p. 2
2 Task Definition
Subtask A labels each community answer by its relevance to the question. English-only subtask B uses the answers to assign an overall yes, no, or unsure response to yes/no questions.
2 Task Definition
- For subtask A, each answer is classified as good, potentially useful, or bad/irrelevant.Source: §2 · p. 2
- Subtask A takes a question and several community answers as input, then classifies each answer.Source: §2 · p. 2
- English-only subtask B decides whether the overall answer to a yes/no question is yes, no, or unsure, based on the individual good answers.Source: §2 · p. 2
3 Datasets
The datasets pair English Qatar Living forum threads with Arabic Fatwa questions, using different candidate-answer setups and annotation processes. They provide annotated training, development, and test data, with English raw text also released for research use.
3 Datasets
- The task uses Qatar Living threads for English and Fatwa questions for Arabic, with different candidate-answer setups across the two datasets.English questions have lists of community answers; Arabic questions are paired with possible answers.Source: §3 · p. 3
3.1 English Data: CQA-QL corpus
- The CQA-QL corpus samples Qatar Living questions and answer threads, which were manually filtered and annotated with the task categories.English examples include question and comment attributes, with gold labels hidden at test time for the task predictions.Source: §3.1 · pp. 3–4
3.2 Annotating the CQA-QL corpus
- English annotations used Mechanical Turk after trial labeling and guideline development, with 3–5 annotators per decision and majority voting to resolve disagreements.Ties led to removal of some comments and sometimes whole questions; YES/NO question labels were derived from good comments’ labels.Source: §3.2 · p. 4
- About 10% of English questions are YES/NO; there are about 6 comments per question on average, ranging from 1 to 143, and roughly half are Good.The released raw Qatar Living questions and comments contain more than 100 million word tokens.Source: §3.2 · p. 4, Table 1 · p. 5
3.3 Arabic Data: Fatwa corpus
- Arabic Fatwa examples provide five candidate answers per question, labeled direct, related, or irrelevant.The corpus records question and answer attributes, and answer bodies may contain tags such as named entities, Quran verses, or Hadeeth.Source: §3.3 · p. 5
3.4 Annotating the Fatwa corpus
- Each Arabic question has one direct answer; linked-question answers are related, and random answers fill the five-answer set as irrelevant candidates.Test30 uses information retrieval to select irrelevant answers with significant question-term overlap; the general test set combines those 30 questions with 170 using random irrelevant answers.Source: §3.4 · pp. 5–6
4 Scoring
Scoring uses macro-averaged F1 over the target categories for each subtask. Classification accuracy is also reported.
4 Scoring
- Macro-averaged F1 over the target categories is the official score for both subtasks.The target categories differ across English subtask A, Arabic subtask A, and English subtask B.Source: §4 · p. 6
- English subtask A is scored over Good, Potential, and Bad, while Arabic subtask A uses direct, related, and irrelevant.Source: §4 · p. 6
- English subtask B is scored over Yes, No, and Unsure, and classification accuracy is also reported.Source: §4 · p. 6
5 Participants and Results
The leading systems achieved macro F1 scores of 57.19 for English subtask A, 78.55 for Arabic subtask A, and 63.7 for English subtask B. Results also reveal distinct challenges across class schemes, test settings, and evaluation metrics.
5 Participants and Results
- English A drew 12 teams and 30 runs, Arabic A four teams and 11 runs, and English B eight teams and 20 runs.The English A and B result sets included both primary and contrastive submissions.Source: §5.1 · p. 6, §5.2 · p. 8, §5.3 · p. 9
5.1 Subtask A, English
- 57.19 macro F1 and 72.52 accuracy made JAIST the top English subtask A system.It used topic models, word vectors, and an SVM classifier.Source: §5.1 · pp. 6–7, Table 4 · p. 7
- In English subtask A, four-way scores were up to 10 points lower than three-way scores, and Potential was hardest, with every team below F1 20.Separating Dialog from Bad was difficult; merging those classes raised HITSZ-ICRC's Bad+Dialog F1 to 74.32.Source: §5.1 · pp. 7–8, Table 5 · p. 8
5.2 Subtask A, Arabic
- 78.55 macro F1 and 83.02 accuracy made QCRI the clear winner on Arabic subtask A.QCRI ranked answers using logistic regression and labeled the top answer direct, the next related, and the rest irrelevant.Source: Table 6 · p. 8, §5.2 · p. 8
- On the 30 manually annotated Arabic questions, results were much lower, with common question terms in irrelevant answers and class-distribution differences cited as possible explanations.QCRI remained best, while VectorSLU, which relied heavily on overlap and similarity, had a relatively larger performance drop.Source: Table 7 · p. 8, §5.2 · p. 9
5.3 Subtask B, English
- In English subtask B, every submission beat the majority baseline on macro F1, but only half did so on accuracy.The top systems' primary runs outperformed their contrastive runs, unlike the pattern reported for subtask A.Source: §5.3 · pp. 9–10
- 63.7 macro F1 and 72 accuracy made VectorSLU the top English subtask B system.ECNU followed at F1=55.8 and Acc=68; QCRI and HITSZ-ICRC tied at F1=53.6 and Acc=64.Source: Table 8 · p. 9, §5.3 · p. 10
- Most subtask B teams made comment-level Yes/No/Unsure predictions and used majority voting for a question-level label; direct question-level prediction yielded lower performance.Most teams used nearly the same features as in subtask A, with some adding positive, negative, or uncertainty-word features.Source: §5.3 · pp. 9–10
6 Features and Techniques
Participants largely engineered features for supervised classifiers, drawing on question, comment, question-comment pair, and metadata signals. Some systems also used richer modeling techniques and a broad range of preprocessing tools.
6 Features and Techniques
- Participants grouped features into question-specific, comment-specific, question-comment pair, and metadata categories.Metadata included asker and commenter identities, their relationship, user history, timestamps, and question category.Source: §6 · p. 10
- Question-comment pair features measured similarity and overlap using lexical, WordNet, language-model, topic-model, and semantic methods.Other examples included information gain from the comment with respect to the question.Source: §6 · p. 10
- Most systems paired engineered features with standard classifiers, while some used kernels, embeddings, neural models, dialog or graph modeling, alignment, time modeling, or sentiment analysis.For Arabic, some teams translated data into English and extracted features from both language versions.Source: §6 · p. 10
- Teams used varied tools and resources for preprocessing, feature extraction, and machine learning, including NLP toolkits, classifiers, translation, and word-vector resources.Examples included GATE, LibSVM, MADA, NLTK, word2vec, and GloVe.Source: §6 · pp. 10–11
- Preprocessing ranged from tokenization, stemming, parsing, and entity recognition to word alignment, spam filtering, and removing or normalizing noisy content.The passage also lists sentence splitting, lemmatization, morphological analysis, tagging, and embedding among the techniques used.Source: §6 · p. 11
7 Discussion
Submissions consistently beat the majority-class baseline on macro-F1, but many English systems fell below it on accuracy. Arabic subtask A had higher macro-F1 than English, while the Potential/related class remained hardest, especially in English.
7 Discussion
- The task drew 13 teams and 61 submissions, with 12 and 8 teams for English subtasks A and B versus 4 for Arabic.English data followed natural forum threads, while Arabic data was somewhat artificial; third place was closely tied across all three subtask-language combinations.Source: §7 · p. 11
- All submissions beat the majority-class baseline on official macro-F1 across both subtasks and languages, but many English systems fell below it on accuracy.The macro-F1 improvement was smaller for English and much larger for Arabic.Source: §7 · p. 11
- Arabic subtask A’s top systems scored 78.55 macro F1 versus 57.19 for English, a difference of over 21 points.The authors suggest Arabic was generally easier because its language was more formal and each question had at most one direct answer.Source: §7 · p. 11
- The top-system accuracy gap between Arabic and English was 10 points: 82.02 versus 72.52.Source: §7 · p. 11
- Arabic class F1s were 77.31/67.13/91.21 versus English’s 78.96/14.36/78.24 for direct/related/irrelevant and Good/Potential/Bad, respectively.Precision and recall were very close to F1 for both languages.Source: §7 · p. 11
- Potential/related was hardest: its P, R, and F1 were around 67 for Arabic and 14 for English, while the other classes were about 80–90.The small English Potential class could hurt accuracy slightly but still substantially affect macro-F1; this intermediate category was easily confused with the other two.Source: §7 · p. 11
8 Conclusion and Future Work
SemEval-2015 Task 3 attracted submissions and produced datasets released for use beyond the task. The authors identify substantial unanswered questions about linguistic approaches to informal cQA text and plan follow-up tasks.
8 Conclusion and Future Work
- The task received 61 submissions from 13 teams, and its released datasets are intended to be useful beyond SemEval.Source: §8 · p. 12
- Whether complex linguistic representations and inference can work effectively on informal, ungrammatical cQA forum text remains largely unanswered.The task encouraged approaches beyond keyword and bag-of-words matching, and participants tried semantic, syntactic, contextual, metadata, and external-resource features.Source: §8 · p. 12
- The authors planned a SemEval-2016 follow-up focused on new Qatar Living questions and a real cQA dataset for Arabic.Source: §8 · p. 12