The CL-SciSumm Shared Task 2018: Results and Key Insights

• Citances provide community-created summaries of a cited paper’s key points, contributions, and importance, adding interpretative context to its text.They may complement the context of scholars and literature-review writers.Source: §1 · p. 1
• The task encourages the use of research facets, scholars’ mini-summaries, and computational-linguistics concept taxonomies in automatic scientific-paper summarization.Source: §1 · p. 1
• The organizers anticipate that the task will spur new resources, tools, methods, and evaluation frameworks.Source: §1 · p. 2
• CL-SciSumm began at TAC 2014 and was held in 2018 as part of the BIRNDL workshop at SIGIR in Ann Arbor, Michigan.Source: §1 · p. 2

• Each topic pairs a Reference Paper with at least ten Citing Papers containing identified citation spans, and provides three kinds of RP summaries.These are the author-written abstract, a community summary collated from citances, and an annotator-written summary.Source: §2 · p. 2
• Task 1A links each citance to the most relevant text span in the Reference Paper, ranging from a fragment to at most five consecutive sentences.Source: §2 · p. 2
• Task 1B assigns each cited text span a facet from a predefined set.Source: §2 · p. 2
• Optional Task 2 generates a structured Reference Paper summary of no more than 250 words from its cited text spans.Source: §2 · p. 2

3 Development

• The training corpus used ACL Anthology papers with at least ten citations and refined the 40-Reference-Paper CL-SciSumm 2017 corpus.Citing sentences were mapped to referenced RP sentences, and RP summaries were included.Source: §3 · pp. 2–3
• The test set added 20 Reference Papers from the ACL Anthology Network, with each paper and its citing papers annotated independently three times.Three sets of human summaries were also created; the test set could differ somewhat from training data.Source: §3 · p. 3

3.1 Annotation

• The unchanged annotation scheme identifies each RP citation’s citation text, marker, reference text, and discourse facet in its citing paper.Source: §3.1 · p. 3

• The Beijing team used WMD and improved LDA for citation linkage, rule-based and supervised methods for facets, and a WMD-informed DPP kernel for summarization.Source: §4 · p. 3
• Thomson Reuters framed linkage as binary classification and facets as multilabel classification, while NUDT combined random forests with BM25, VSM, embeddings, and voting.Source: §4 · pp. 3–4
• NJUST combined classifier voting for linkage, dictionary/topic-model/XGBOOST facet methods, and cluster-based summary extraction; IIIT-Hyderabad used CNN matching and SVM/naive Bayes facet features.Source: §4 · p. 4
• Klick Labs explored embedding-based linkage and sentence classification, while Houston used Siamese networks and a positional language model for linkage plus WordNet-augmented rules for facets.Source: §4 · p. 4
• LaSTUS/TALN+INCO used similarity and CNN approaches for Tasks 1A/1B and CNN-scored RP sentences for Task 2; NLP-NITMZ used cosine/Jaccard similarity, rule-based facets, and OpenNMT.Source: §4 · pp. 4–5
• Team Magma treated Task 1A as binary classification and found logistic regression with topic- and word-similarity content features performed best.Source: §4 · p. 5

• Task 1A used sentence-ID overlap precision, recall, and F1, with overall performance reported as micro-averages across blind-test topics.ROUGE-2 and ROUGE-SU4 lexical overlap scores were also calculated for Task 1A.Source: §5 · p. 5
• ROUGE-2 and ROUGE-SU4 measured lexical overlap, and Task 1 results averaged performance across three independent annotation sets.Human-summary performance likewise averaged results across three human summaries.Source: §5 · p. 5
• Task 1B measured correctly classified discourse facets conditional on the expected Task 1A response, using precision, recall, and F1 for its multi-label setting.Source: §5 · p. 5
• Task 2 was optional and compared system output with the abstract, community summary, and human summary using ROUGE-2 and ROUGE-SU4.Source: §5 · p. 5

• The comparison covers participating systems, with three of the ten Task 1 systems also entering optional Task 2.Source: §6 · p. 6
• For Task 1A sentence-overlap F1, NUDT led; for ROUGE-based F1, Klick Labs led, followed by NUDT and NLP-NITMZ.CIST followed NUDT on sentence-overlap F1, and UPF-TALN ranked third.Source: §6 · p. 6
• CIST had the best Task 1B performance, followed by NJUST, with Klick Labs the second runner-up.Source: §6 · p. 6
• For Task 2, TALN-UPF led against abstract and human summaries, while NLP-NITMZ led against community summaries.TALN-UPF ranked second against community summaries; NLP-NITMZ was second runner-up against human summaries, and CIST placed second runner-up against abstract and community summaries and second against human summaries.Source: §6 · p. 6

• Ten teams participated in the 2018 task, whose corpus was 33% larger than the 2017 corpus.Source: §7 · p. 6
• Planned follow-up work includes detailed annotation comparison, micro-level error analysis, expanding multiply annotated corpus portions, and releasing complementary resources.The anticipated complementary resources include semantic concepts from the ACL Anthology Network.Source: §7 · p. 6
• The authors suggest lexical methods may work with scientific documents' structural and semantic characteristics, possibly complemented by domain-specific embeddings in deep learning.Source: §7 · pp. 6–7
• The authors see the task as having potential for transfer learning and expect its methods to generalize to other scientific-summarization areas.Source: §7 · p. 7

• Table 1 covers system performance for Tasks 1A and 1B and is ordered by Task 1A sentence-overlap F1.Source: Table 1 · p. 10
• The tables provide performance comparisons for different tasks, with Task 1A sentence-overlap F1 and Task 2 ROUGE-2 F1 serving as their respective ordering criteria.Source: Table 1 · p. 10, Table 2 · p. 11
• Table 2 covers Task 2 system performance and is ordered by ROUGE-2 F1.Source: Table 2 · p. 11