Source-linked AI summary
SemEval-2016 Task 3: Community Question Answering
Preslav Nakov, Lluís Màrquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhakim Freihat, James Glass, Bilal Randeree
TL;DR
Community question answering requires ranking useful answers from noisy, loosely moderated forums, and existing systems needed broader semantic and thread-aware evaluation. SemEval-2016 Task 3 addresses this with four supervised ranking subtasks across English and Arabic, attracting 18 teams and 95 runs; best MAP scores ranged from 45.83 to 79.19 across subtasks and beat the provided baselines.
Problem
Community forums can contain hundreds of noisy, informal answers, while CQA systems need semantic and thread-level methods beyond simple word matching.
Method
The task defines four independent supervised ranking subtasks across English and Arabic, evaluates retrieved candidates, and summarizes participating approaches and features.
Results
Best MAP scores ranged from 45.83 to 79.19 across subtasks, and the best systems beat the provided baselines by sizeable margins.
Takeaways & Limitations
The task established a broader CQA benchmark whose top systems combined neural word embeddings with syntactic kernels, while releasing datasets for follow-up research.
Takeaways & Limitations
Training part2 was noisier because a confusing annotation setup led annotators to label the wrong relevance relation; development data required manual correction.
Abstract
from arXiv · showhide
This paper describes the SemEval--2016 Task 3 on Community Question Answering, which we offered in English and Arabic. For English, we had three subtasks: Question--Comment Similarity (subtask A), Question--Question Similarity (B), and Question--External Comment Similarity (C). For Arabic, we had another subtask: Rerank the correct answers for a new question (D). Eighteen teams participated in the task, submitting a total of 95 runs (38 primary and 57 contrastive) for the four subtasks. A variety of approaches and features were used by the participating systems to address the different subtasks, which are summarized in this paper. The best systems achieved an official score (MAP) of 79.19, 76.70, 55.41, and 45.83 in subtasks A, B, C, and D, respectively. These scores are significantly better than those for the baselines that we provided. For subtask A, the best system improved over the 2015 winner by 3 points absolute in terms of Accuracy.
1 Introduction
SemEval-2016 Task 3 extends community question answering toward application-oriented answer retrieval and reranking across English and Arabic. It targets semantic and thread-level phenomena while providing supervised data and optional subtasks for broader participation.
- Motivation: The task ranks useful comments from large collections of community-created question-comment threads for new questions.The collection may contain hundreds of noisy, informal, and error-filled answers, making manual inspection time-consuming.
- Objectives: The organizers pursued semantic solutions beyond word matching, studied relations within and across threads, and examined question-to-question similarity.They also aimed to support participation by non-IR/QA experts through supplied candidate answers and optional subtasks.
- Scope: The challenge covered English and Arabic, with English data from Qatar Living and Arabic data from medical forums.Arabic used only the main CQA ranking task, while English supported three subtasks.
- Resources: Training data for all languages and subtasks was manually labeled by community annotators through crowdsourcing.The datasets and annotation procedure are described as part of the task resources.
2 Related Work
Prior CQA systems combine local question-answer matching with thread-level information and increasingly structured or neural models. The strongest cited prior approach used globally normalized pairwise CRFs with an Ising-like edge potential.
- Passage reranking: Passage reranking applies automatic classifiers to question-answer passage pairs to derive a relative ordering.This frames CQA within established answer-selection research.
- Thread-level modeling: Thread-aware systems add features for comment position, user interaction, and dependencies among comments beyond local question-answer features.Later work modeled global thread structure using LSTMs, structured prediction, graph cuts, and ILP.
- Joint structured models: FCCRF significantly outperformed other approaches on SemEval-2015 Task 3.Its crucial elements were global normalization and an Ising-like edge potential.
3 Definition of the Subtasks
The challenge defined four independent ranking subtasks: three for English and one for Arabic. They rank comments, related questions, or question-answer pairs using retrieved candidate sets and top-10-focused evaluation.
- Overall structure: The challenge comprised four independent subtasks: English A, B, and C, plus Arabic D.The subtasks cover question-comment, question-question, question-external-comment, and correct-answer reranking.
- English subtask A: Subtask A ranks the first ten comments in a question thread by relevance to the initiating question.It is evaluated as ranking rather than classification, with Good comments above PotentiallyUseful and Bad comments.
- English subtask B: Subtask B ranks ten retrieved related questions by similarity to a new question, treating PerfectMatch and Relevant as good.Participants can use each related question’s associated thread, making the task effectively question-plus-thread similarity.
- English subtask C: Subtask C ranks 100 comments from ten retrieved question threads according to relevance to the original question.MAP considers only the top ten results because the evaluation assumes users concentrate on the first positions.
- Arabic subtask D: Subtask D ranks 30 retrieved question-answer pairs for Arabic, placing Direct and Relevant answers above Irrelevant ones.Its MAP evaluation is also restricted to the top ten results, reflecting lower retrieval recall and fewer correct answers.
4 Datasets
The datasets cover English community-question threads and Arabic medical question-answer pairs, with annotations supporting the task’s subtasks. English data was organized around original questions, related questions, and their first ten comments, while Arabic data used retrieved question-answer pairs.
- English Dataset: The English CQA-QL corpus contains original questions, ten related Qatar Living questions, and the first ten comments from each related thread.It includes annotations for related-question relevance and comment relevance to both related and original questions.
- English Dataset: English training and development data used SemEval-2015 questions, whereas testing used new Qatar Living questions.This selection was intended to approximate a setting where test questions had not previously been asked.
- English Dataset: English related threads were retrieved through Google using the original question’s subject, filtered, and reduced to the top ten surviving threads.Threads with fewer than ten comments or questions longer than 2,000 characters were removed before final selection.
5 Scoring
Systems were officially evaluated with Mean Average Precision over their top ten ranked comments, alongside additional ranking and classification measures. A specialized scorer reported all seven measures.
- Official Measure: Mean Average Precision (MAP) over the ten highest-ranked comments was the official evaluation measure.The measure evaluates the ranked results submitted by each participating system.
- Ranking Measures: Mean Reciprocal Rank (MRR) and Average Recall (AvgRec) were also reported for the top ten results.These were described as unofficial ranking measures.
- Classification Measures: Precision, Recall, F1, and Accuracy were calculated over the full result list as standard classification measures.F1 was calculated with respect to the Good/Relevant class.
- Scoring Tools: A specialized scorer calculated and reported all seven evaluation scores.The scorer supported both the ranking and classification measures described for the task.
6 Participants and Results
Eighteen teams submitted 95 runs across four subtasks, with performance and baseline difficulty varying substantially by task. KeLP won English subtask A, UH-PRHLT won B, SUper won C, and SLS won Arabic D.
- Overall participation: 18 teams submitted 95 runs—38 primary and 57 contrastive—across English subtasks A–C and Arabic subtask D.The English subtasks attracted 12, 11, and 10 systems, while Arabic subtask D attracted five.
- Subtask A: 79.19 MAP made KeLP the subtask A winner, ahead of ConvKN at 77.66 and SemanticZ at 77.58.KeLP also ranked first on AvgRec and MRR, while ConvKN ranked first on Accuracy.
- Subtask B: 76.70 MAP made UH-PRHLT the subtask B winner, only 2 MAP points above the IR baseline.Its contrastive2 run achieved MAP of 77.33.
- Subtask C: 55.41 MAP made SUper the subtask C winner, while severe class imbalance let the all-Bad baseline beat 9 of 10 teams on Accuracy.SUper also ranked first on AvgRec and MRR; KeLP’s contrastive run reached MAP of 55.58.
- Subtask D: 45.83 MAP made SLS the Arabic subtask D winner on every measure, narrowly ahead of ConvKN at 45.50.The all-Bad baseline outperformed every participating system on Accuracy, and the random baseline beat one system on MAP and AvgRec.
7 Features and Techniques
Participating systems combined lexical, semantic, structural, metadata, and thread-level signals with SVMs and neural networks. Tree kernels were prominent among top-ranked systems, while neural networks contributed to the best systems across all subtasks.
- Feature families: Common features included question–comment or question–question similarities, content signals, thread metadata, and syntactic tree-kernel features.Examples include cosine similarity, user ID, comment rank, and automatically generated syntactic features.
- Similarity features: Similarity features used word ngrams, character 3-grams, TF-IDF, word overlap, nouns, named entities, and distributed representations.Distributed similarities were derived with models such as word2vec and GloVe.
- Learning methods: Systems also used CNNs, LSTMs, FNNLMs, SVM classification, regression or ranking, and Random Forests.SVMs supported complex convolutional kernels, including tree kernels.
- Task-specific features: Top systems applied task-specific structures, including relational links, knowledge graphs, FrameNet frames, user statistics, readability, credibility, sentiment, trollness, and goodness polarity.For Arabic, SLS concatenated features comparing original–related questions and original question–related answer pairs.
- Arabic systems: ConvKN combined SLS features with syntactic tree kernels and achieved a result very close to the winning SLS system in Arabic.The Arabic systems also used word2vec, language models, and Arabic-Wikipedia information for medical terms.
- Comparing approaches: Tree-kernel systems ranked first or second in subtask A, second or third in B, and second in C and D.Neural-network systems did not win a subtask, but neural networks contributed to the best systems in all subtasks, including through word2vec.
8 Conclusion
SemEval-2016 Task 3 expanded the CQA challenge with new subtasks, metrics, datasets, and domains, attracting broader participation and producing reusable research resources.
- The task extended SemEval-2015 with Question–Question Similarity, Question–External Comment Similarity, answer reranking, ranking-based metrics, new datasets, and biomedical Arabic data.
- 18 teams submitted 95 runs, up from 13 teams and 61 runs in 2015.
- Top systems combined neural-network word embeddings with syntactic kernels, highlighting the use of both distributed representations and linguistic analysis.
- The released datasets contain over 7,000 questions and over 57,000 annotated comments for follow-up research beyond SemEval.
- The organizers planned a SemEval-2017 rerun using data from a new domain.
Team ID Team Affiliation
Table 7 presents the participating teams and their affiliations.
- Table 7 lists the teams participating in the task.
- The table records each participating team’s affiliation.
- Team names and affiliations are presented together in one table.