PaperlayerView exact revision on arXiv

Source-linked AI summary

NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned

Sewon Min, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, Tom Kwiatkowski, Patrick Lewis, Yuxiang Wu, Heinrich Küttler, Linqing Liu, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel, Sohee Yang, Minjoon Seo, Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Edouard Grave, Ikuya Yamada, Sonse Shimaoka, Masatoshi Suzuki, Shumpei Miyawaki, Shun Sato, Ryo Takahashi, Jun Suzuki, Martin Fajcik, Martin Docekal, Karel Ondrej, Pavel Smrz, Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, Wen-tau Yih

arXiv:2101.00133v2cs.CLcs.AI

TL;DR

EfficientQA tested how open-domain QA systems could balance accuracy with strict on-disk memory budgets. The paper reviews competition systems and human evaluation, finding that human judgments recognize many correct answers missed by automatic matching while restricted systems remain behind unrestricted ones.

  • Problem

    EfficientQA addressed how open-domain QA systems can predict correct answers while meeting strict on-disk memory budgets.

    Source: §1 · p. 2

  • Method

    The paper reviews systems across four memory-and-accuracy tracks and analyzes their predictions using human evaluation of correctness and ambiguity.

    Source: §1 · p. 2, §1.1 · p. 3

  • Results

    Human evaluation found 30% of predictions unmatched to references were nonetheless correct, with relative accuracy rising 18–32% under definitely-correct ratings and 42–71% under possibly-correct ratings.

    Source: §7 · p. 17

  • Takeaways & Limitations

    Automatic evaluation alone is insufficient for thorough open-domain QA evaluation because it misses semantically equivalent answers, time-dependent answers, and ambiguity in questions.

    Source: §7 · p. 17

    The authors note that better non-binary definitions of correctness are still needed because correctness and plausibility can depend on pragmatic interpretations of questioner intent.

    Source: §5.2 · p. 13

Abstract

from arXiv · show

We review the EfficientQA competition from NeurIPS 2020. The competition focused on open-domain question answering (QA), where systems take natural language questions as input and return natural language answers. The aim of the competition was to build systems that can predict correct answers while also satisfying strict on-disk memory budgets. These memory budgets were designed to encourage contestants to explore the trade-off between storing retrieval corpora or the parameters of learned models. In this report, we describe the motivation and organization of the competition, review the best submissions, and analyze system predictions to inform a discussion of evaluation for open-domain QA.

1. Introduction

EfficientQA challenged teams to build open-domain QA systems that balance answer accuracy against strict on-disk memory budgets. Its submissions show that approaches vary across memory tracks, while human evaluation changes measured accuracy by recognizing answers that automatic matching misses.

1. Introduction

  • The competition set four tracks, from unrestricted accuracy to systems under 6GiB or 500MiB and the smallest system reaching 25% accuracy.The budgets were intended to prompt exploration of trade-offs among retrieval corpora, structured data stores, and learned-model parameters.Source: §1 · p. 2

1.1. Key takeaways

  • Top submissions across all four tracks outperformed the provided baselines, but differed substantially in their retrieval corpora and answering modules.Table 1 distinguishes retrieval targets, corpus pruning, answer type, and other system features.Source: §1.1 · p. 2, Table 1 · p. 3
  • Unrestricted and 6GiB systems exceeded April 2020 state-of-the-art baselines by nearly 20%, combining retrieval, generation, broader Wikipedia content, and extractive or generative prediction.The 6GiB systems also reduced indexed-corpus size and used compression with minimal impact on accuracy.Source: §1.1 · p. 3
  • Under 500MiB, one approach reused a Transformer for retrieval and reading, while another indexed generated question-answer pairs and retrieved similar questions.The question-answer-pair approach also competed in the smallest 25% track.Source: §1.1 · p. 3
  • Human evaluation allowed multiple definitions of correctness for ambiguous questions; accuracy rose 20%–25% under the strictest definition and doubled when any semantically valid interpretation was accepted.The paper discusses this evaluation and offers suggestions for future work.Source: §1.1 · pp. 3–4

2. Competition Overview

The competition used open-domain Natural Questions data and evaluated systems with both automatic and human judgments. Restricted-track system size was measured as the on-disk size of a self-contained Docker image.

2. Competition Overview

  • The competition adapted Natural Questions to open-domain QA and introduced hidden test and development sets labeled slightly later than the original data.The open-domain training set had 88k one-way annotated examples and development set had 4k five-way annotated examples.Source: §2 · p. 4
  • Restricted-track size meant the at-rest on-disk size of each submitted Docker image, including its system contents.The competition received 39 submissions from 18 teams.Source: §2 · pp. 4–5
  • Automatic accuracy used exact match after minor normalization against reference answers, which could omit correct predictions because five references were not exhaustive.Predictions were sent for human rating to obtain a better accuracy estimate.Source: §2 · pp. 4–5
  • The competition ran from its June 2020 announcement through November submissions, followed by human evaluation and presentations in December.The official restricted-track leaderboard opened September 14, with submissions due November 14.Source: §2 · p. 5

3. Systems

The top systems used retrieval and neural answering modules, but their designs varied by memory track. Approaches ranged from enhanced retriever-reader systems to question-answer-pair retrieval and a single Transformer serving retrieval and reading.

3.1. Unconstrained track

  • Unrestricted systems built on retrieval-reader methods, with leading submissions improving training, combining extractive and generative answers, or incorporating Wikipedia lists and tables.FB Hybrid combined DPR and GAR retrieval and used a Fusion-in-Decoder reader; its index included linearized tables and infoboxes.Source: §3.1 · pp. 5–6

3.2. 6GiB track

  • All top-three 6GiB submissions pruned Wikipedia and used quantization or compression, while enhancing earlier state-of-the-art approaches.A baseline also reduced the corpus from 21M to 1.6M passages by retaining pages paired with training questions.Source: §3.2 · p. 6
  • The 6GiB systems included retriever-reader designs; one system compressed data and removed low-view passages to retain 18M passages without dropping accuracy.Another combined dense and generative-query-augmented retrieval with a Fusion-in-Decoder reader.Source: §3.2 · p. 7

3.3. 500MiB track

  • The top 500MiB submissions used markedly different approaches, including a generated question-answer database and a single Transformer for retrieval and reading.The database system generated and filtered question-answer pairs, then used retrieval and reranking to select an answer.Source: §3.3 · p. 8

3.4. 25% smallest track

  • The smallest-track UCLNLP-FB variant reduced memory by using TF-IDF retrieval, an ALBERT-base reranker, and 40k generated question-answer pairs.The provided T5-XL+SSM baseline was 5.65GB and achieved 28% accuracy.Source: §3.4 · p. 9

4. Human Annotations of Correctness

The annotation procedure asked raters to interpret question intent, assess ambiguity, and distinguish definite from possible correctness. Aggregated judgments showed varying agreement across the three-way and binary labels.

4. Human Annotations of Correctness

  • Each prediction was independently rated by three annotators, who interpreted the question, assessed ambiguity, and assigned one of three correctness labels.Ambiguity could arise from time, place, unspoken intent, or opinion.Source: §4 · p. 9
  • For questions whose denotation might have changed since collection, annotators considered any time since the web existed and used “possibly correct” for answers that might have been right when asked.The procedure avoided imposing a single correct point in time for such questions.Source: §4 · p. 9
  • The final rating aggregated three annotators’ labels by majority thresholds for “definitely correct” and “possibly correct.”The table summary identifies “Definitely correct” human ratings as the primary metric.Source: §4 · p. 9, Table 2 · p. 10
  • Pairwise agreement was 69.2% for three-way ratings, 85.7% for definitely-correct decisions, and 76.7% for possibly-correct decisions.The corresponding Cohen’s κ values were 53.8, 71.4, and 53.4, respectively.Source: §4 · p. 10

5. Results & Analyses

Across the competition tracks, leading systems outperformed earlier baselines, while human evaluation found many predictions that exact-match scoring rejected to be correct. The analyses show that answer validity can depend on ambiguity and interpretation, although rankings remained unchanged on the unambiguous subset.

5.1. Results

  • All five unrestricted and 6GiB systems significantly outperformed DPR (36.6%) and REALM (35.9%), while both top 500MiB systems beat T5-small (17.6%).The 6GiB systems approached unrestricted-track accuracy.Source: §5.1 · p. 10
  • Human raters judged 13% of predictions that missed reference answers definitely correct and 17% possibly correct; human evaluation could change rankings when automatic-score gaps were around or below 1%.Most systems gained 17–25% in accuracy under the definitely-correct evaluation and 41–54% under the possibly-correct evaluation, relative to automatic evaluation; NAVER RDR gained 32% and 71%, respectively.Source: §5.1 · p. 10
  • The FB Hybrid–Ousia-Tohoku Soseki pair had the highest ensemble oracle accuracy, while FB Hybrid and FB system showed the greatest prediction agreement.The oracle counts a question correct if either system is correct; the FB systems’ agreement was attributed as likely to their shared DPR and Fusion-in-Decoder basis.Source: §5.1 · pp. 10–11, Figure 2 · p. 11

5.2. Analyses

  • Possibly correct but non-best predictions included ambiguous answer forms (20%) and time-dependent answers (18%); closer-to-invalid cases included conflicting Wikipedia information (4%), condition-dependent answers (4%), and question-intent mismatches (8%).The report also identified definitely incorrect predictions (2%).Source: §5.2 · p. 12
  • Among predictions judged definitely correct, 60% were semantically equivalent to the gold answer; other valid-answer cases included ambiguous references (20%), open-ended questions (6%), and differing granularity (14%).For possibly-correct predictions, the corresponding reported shares were 22%, 20%, 4%, and 6%; categories could overlap.Source: §5.2 · pp. 11–12
  • On the unambiguous subset, which comprised 51.5% of questions, most systems improved their accuracy, but the overall system rankings did not change.The authors note that correctness and plausibility can depend on pragmatic interpretations of questioner intent, and call for better non-binary definitions of correctness.Source: §5.2 · p. 13

6. Trivia Experts vs Computer Systems

The human–computer competition compared skilled trivia teams with strong QA systems in a staged format designed to give both sides fair opportunities. The final match exposed questions that stumped both humans and computers, while illustrating how question interpretation and answer scope can complicate comparison.

6.4. Reflection

  • The evaluation sought fair comparisons with skilled humans while recognizing that trivia games can illuminate QA strengths and weaknesses.The authors also describe human evaluation as useful for exposing dataset problems, ambiguity, and answer usefulness.Source: §6 · p. 14, §6.4 · p. 16

6.1. A Fair Comparison

  • The organizers recruited trivia enthusiasts and selected five teams for the preliminary competition.They excluded overly ambiguous questions and judged answers using an “actually correct” standard consistent with traditional trivia interpretations.Source: §6.1 · pp. 14–15
  • The final match gave participants progressively more resources, from instant recall to discussion and, finally, search snippets for both sides.The human teams could discuss answers for thirty seconds in later phases, while computer systems faced increasing memory allowances.Source: §6.1 · p. 14, Table 4 · p. 15

6.2. Preliminary Competition

  • Only two human teams outscored the computer baselines in the preliminary competition, with Team B clearly the strongest on this question set.Scores were normalized as average margin per question because teams could answer different numbers of questions.Source: §6.2 · p. 15, Table 5

6.3. Final Game

  • Four questions reached the final phase, and only the question about Ethiopia’s second-largest city was answered correctly.The other questions involved unavailable Wikipedia information, language demands beyond QA, or answers whose intended scope was disputed.Source: §6.3 · p. 16

7. Conclusions and Future Work

The competition shows that strong QA performance remains difficult under tight memory budgets, and that automatic evaluation misses some correct answers. The authors call for continued exploration of efficiency trade-offs and more careful human-informed evaluation.

Conclusions

  • Memory-restricted systems trail unrestricted systems by 20%, leaving substantial room to improve QA under tight memory budgets.Restricted-track submissions explored generated question-answer corpora or sharply filtered Wikipedia with a single Transformer for retrieval and answer extraction.Source: §7 · pp. 16–17
  • Human raters judged 30% of predictions that missed reference answers to be correct, while accuracy gains varied by correctness standard and system.The gains were 18–32% for “definitely correct” and 42–71% for “possibly correct”; missed matches included semantically equivalent answers, time-dependent answers, and ambiguous questions.Source: §7 · p. 17

Future Work

  • Future work should continue exploring the trade-off among system size, accuracy, and abstaining, while refining QA evaluation.The authors suggest human verification, removing ambiguity, adversarial examples, and assessing whether responses are useful to users.Source: §7 · p. 17

Appendix A. Analyses

Questions with explicit Wikipedia support and high lexical overlap are easy for all systems, while hard questions often involve tables or less explicit support. Error analysis of two 500MiB systems identifies wrong-question retrieval as a common UCLNLP-FB failure and missing gold passages as a frequent NAVER RDR failure.

Characteristics of easy and hard questions

  • Easy questions often have explicit Wikipedia support with high lexical overlap and require little paraphrasing, while hard questions often have answers in tables.The authors conjecture that other hard cases involve supporting text that is harder to retrieve or less explicit; some questions are also not well-formed.Source: Appendix A · p. 23, Table 8 · p. 25

Error Analysis

  • The error analysis compares UCLNLP-FB and NAVER RDR on 100 randomly sampled development questions: 40 were correct for both, 33 for one, and 27 for neither.The two systems were selected because their approaches differ significantly and the 500MiB track fits the competition’s main motivation.Source: Appendix A · pp. 23–24
  • 47 of UCLNLP-FB’s 50 sampled errors came from retrieving an incorrect question, including 25 on a different topic and 19 asking about different details.Some retrieved questions with a different meaning nevertheless yielded the correct answer when both questions shared an answer.Source: Appendix A · pp. 24–25, Table 9 · p. 26
  • NAVER RDR’s sampled failures included missing the gold passage during dense retrieval (12%) or re-ranking (15%), while 7% contained the answer but failed extraction.Pruning the gold passage accounted for only 3% of failures; the gold passages in extraction failures had valid but implicit support.Source: Appendix A · p. 25, Table 10 · p. 26
Loading 2101.00133v2…