Source-linked AI summary
SemEval-2017 Task 3: Community Question Answering
Preslav Nakov, Doris Hoogeveen, Lluís Màrquez, Alessandro Moschitti, Hamdy Mubarak, Timothy Baldwin, Karin Verspoor
TL;DR
Community question answering requires systems to retrieve related questions and identify useful answers amid noisy forum content. SemEval-2017 Task 3 reran four multilingual subtasks and introduced large-scale duplicate-question detection, attracting 23 teams and 85 runs; Subtask E received no submissions.
Problem
CQA systems must retrieve similar questions and useful answers despite noisy, informal forum content and lexical gaps between related questions.
Method
The task reran four prior subtasks with multilingual annotated data and added StackExchange-based Subtask E for large-scale duplicate-question detection.
Results
23 teams submitted 85 runs for the task, while participating systems used neural networks, SVMs with syntactic kernels, and new features.
Takeaways & Limitations
The released datasets are intended to support follow-up CQA research beyond the SemEval task.
Takeaways & Limitations
Subtask E received no submissions, mainly because working with its large amount of data was demanding.
Abstract
from arXiv · showhide
We describe SemEval-2017 Task 3 on Community Question Answering. This year, we reran the four subtasks from SemEval-2016:(A) Question-Comment Similarity,(B) Question-Question Similarity,(C) Question-External Comment Similarity, and (D) Rerank the correct answers for a new question in Arabic, providing all the data from 2015 and 2016 for training, and fresh data for testing. Additionally, we added a new subtask E in order to enable experimentation with Multi-domain Question Duplicate Detection in a larger-scale scenario, using StackExchange subforums. A total of 23 teams participated in the task, and submitted a total of 85 runs (36 primary and 49 contrastive) for subtasks A-D. Unfortunately, no teams participated in subtask E. A variety of approaches and features were used by the participating systems to address the different subtasks. The best systems achieved an official score (MAP) of 88.43, 47.22, 15.46, and 61.16 in subtasks A, B, C, and D, respectively. These scores are better than the baselines, especially for subtasks A-C.
1 Introduction
Community question answering forums offer broad access to information but require users to navigate noisy, uneven-quality answers. SemEval-2017 Task 3 reran prior CQA subtasks, added large-scale duplicate-question detection, and supplied multilingual annotated data for evaluation.
- CQA forums let users ask and answer freely, but popular questions may have hundreds of answers that are time-consuming to inspect.
- Automated tools can help users find similar existing questions and identify useful answers for a new question.
- The task addressed question-comment, question-question, and external-comment similarity, with subtasks A and B supporting the main external-comment task C.
- The 2017 edition reran four earlier subtasks, preserved English and Arabic coverage, and introduced Subtask E for large-scale StackExchange duplicate-question detection.
- Subtask E asked participants to rerank 50 candidate questions for each query question according to relevance.
- Training data were manually annotated for both languages and all subtasks, while new test data were created for the evaluation.
2 Related Work
Related work treats CQA similarity as a set of interconnected problems involving question retrieval, answer matching, and thread-level context. Prior systems used syntactic, neural, translation-based, structured, and multitask approaches, especially to address the complex external-comment task.
- Question similarity is a difficult but important first step in CQA because lexical gaps can separate a new question from relevant archived questions.
- Question-question similarity supports question recommendation, duplicate detection, community question answering, and question answering more generally.
- Prior question-question and question-answer systems used syntactic structures, neural networks, attention mechanisms, and machine-translation features.
- Thread-aware methods modeled comment position, comment dependencies, global thread structure, and structured predictions to improve answer assessment.
- Subtask C is less studied and has often been approached by combining thread-level Subtask A predictions with the reciprocal rank of related questions.
- A multitask neural architecture trained the three CQA subtasks together and reportedly improved Subtask C.
3 Subtasks and Data Description
SemEval-2017 Task 3 organized Community Question Answering into four established subtasks and a new large-scale duplicate-question task. The task reused earlier data for subtasks A–D while introducing new test data and a distinct StackExchange-based Subtask E.
- Data: The organizers reused 2016 data for the first four subtasks, created new test sets, and built a completely new dataset for Subtask E.For A–D, new English test data were generated from retrieved Qatar Living question-comment threads, with metadata supplied for related questions and comments.
- Old Subtasks: Subtask A ranked ten comments by relevance to their question, while Subtask B ranked ten retrieved questions by similarity to a new question.Subtask B participants could access the threads associated with related questions.
- Old Subtasks: Subtask C ranked 100 comments from ten retrieved question threads according to their relevance to a new question.The evaluation emphasized the top ten ranked results because users typically do not inspect later search-result pages.
- Old Subtasks: Subtask D ranked 30 question–answer pairs from Arabic medical forums, placing Direct and Relevant answers above Irrelevant answers.The task was offered in Arabic, whereas subtasks A, B, C, and E were offered in English.
- The New Subtask E: Subtask E reranked 50 candidate questions and added StackExchange metadata, surprise subforums for cross-domain testing, and many queries with zero relevant results.Available metadata included tags, view counts, question and answer scores, and user statistics such as reputation and badges.
- The New Subtask E: Only a handful of people annotated the Subtask E test pairs, making those annotations unusable for the shared task.The intended test annotations were meant to come from the same StackExchange community that supplied training and development labels.
- Evaluation Measure for E: Queries without relevant documents make an empty-result baseline difficult to beat and invalidate standard MAP, so Subtask E used a modified MAP.The adjustment gives an empty ranking MAP 1.0 when no relevant documents exist.
4 Participants and Results
SemEval-2017 attracted broad participation across subtasks A–D, while Subtask E received no submissions. Performance varied substantially by task: systems strongly exceeded baselines on A–C, whereas only the best system narrowly beat the strong Arabic baseline in D.
- Participation: 85 runs were submitted by 22 teams across the challenge, including 36 primary and 49 contrastive runs; Subtask E had no participants.English subtasks A, B, and C attracted 14, 13, and 6 systems, while Arabic Subtask D attracted 3 systems.
- Subtask A: 88.43 MAP was the best result for Subtask A, achieved by KeLP, followed closely by Beihang-MSRA at 88.24.Five additional systems obtained MAP scores around 86.5.
- Subtask B: 10 of 13 Subtask B systems outperformed the IR baseline in MAP, AvgRec, and MRR, with top improvements exceeding 7 MAP points.The paper characterizes this as a substantial improvement over the previous edition.
- Subtask C: 15.46 MAP was the best result for Subtask C, while all primary systems beat every baseline on ranking measures.The 2017 test set contained only 2.8% positive instances, so its MAP results cannot be compared directly with 2016.
- Subtask D: 61.16 MAP was the best result for Arabic Subtask D, only marginally above the IR baseline of 60.55.Only the best system improved over the baseline, and most systems did not produce sensible classification predictions because optimization focused on ranking.
5 Discussion and Conclusions
The discussion reviews shared feature families and the varied learning approaches used across the subtasks, highlighting representative systems for subtasks A–D.
- Common Features: Participants used similarity, content, thread-level/meta, and syntactic tree-kernel features across the subtasks.Similarity features included lexical, syntactic, semantic, and distributed representations; content features captured signals of bad comments, while metadata included user and thread information.
- Common Features: Similarity features compared different question and comment components according to each subtask’s input structure.Subtask A compared question fields with comments, Subtask B compared original and related questions, and Subtask C additionally compared the original question with comments from related threads.
- Learning Approaches: 31 machine-learning approaches ranged from SVMs to deep learning.The task featured SVMs, logistic regression, gradient-boosted regression trees, random forests, and neural-network-based representations.
- Systems by Subtask: KeLP performed best on Subtask A with an SVM using syntactic tree kernels, relational links, and standard similarity measures.The components were linearly combined with the tree kernel.
- Systems by Subtask: IIT-UHH performed best on Subtask C using textual, domain-specific, word-embedding, and topic-modeling features.Its dialogue-chain identification method in comment threads yielded substantial improvements.
- Systems by Subtask: SimBow led Subtask B with logistic regression over unsupervised textual similarities derived from lexical and semantic relations.The system used a relation matrix based on cosine similarity between bag-of-words representations and other relations.
- Systems by Subtask: GW QA led Arabic Subtask D with weighted textual matrix factorization and lexical surface-matching features.The latent semantic model was intended to address short-text sparsity by modeling observed and missing words.
6 Conclusions
The task extended earlier CQA evaluation with multi-domain question duplicate detection, attracted substantial participation, and released datasets for follow-up research, although Subtask E received no submissions.
- Conclusions: 23 teams submitted 85 runs, comparable to 2016 participation, while building on prior lessons and testing new features and learning frameworks.The 2016 edition had 18 teams and 95 runs.
- Conclusions: The released task datasets are intended to support follow-up research beyond SemEval.The authors identify the datasets as an additional contribution of the task.
- Conclusions: Subtask E received no submissions, mainly because it required working with a large amount of data.The authors nevertheless describe multi-domain question duplicate detection as an important problem likely to attract research interest.
Submission MAP AvgRec MRR P R F1 Acc
The result tables organize submissions by official MAP ranking and report additional evaluation measures for the task’s subtasks and baselines.
- Subtask B: Table 6 reports Subtask B English question–question similarity results for all submissions.Rows distinguish primary and contrastive runs; columns report MAP, AvgRec, MRR, P, R, F1, and Acc, with subindices ranking primary runs by each measure.
- Table Organization: The result tables use the first column for primary-run rank and the second column for team and submission type.The following columns contain primary and other unofficial evaluation measures.
- Subtask C: Table 7 reports Subtask C English question–external comment similarity results for all submissions.The table ranks primary runs by MAP and reports additional unofficial measures; all results are percentages.
- Subtask E: Table 9 presents baseline results for Subtask E on the English multi-domain duplicate-detection test dataset.It includes an empty-result baseline and BM25 IR baselines with perfect truncation.