Answering Complex Questions Using Open Information Extraction

1 Introduction

• Curated knowledge bases are expensive to build and typically domain-specific, while automatically constructed open-vocabulary tuples had been used only for questions answerable by a single tuple.Elementary-level science exams are presented as a resource-poor domain requiring complex reasoning.Source: §1 · p. 1
• TUPLEINF applies a new ILP-based inference model to reason over Open IE tuples for complex multiple-choice questions requiring facts from multiple sentences.The goal includes resource-poor domains where curated knowledge is unavailable.Source: §1 · pp. 1–2
• A sample science question illustrates the need to combine multiple facts about the Moon, including that it reflects light and orbits a planet.The question’s challenge is its complex structure and multi-fact reasoning requirement.Source: §1 · p. 1
• Unlike curated-table reasoning, Open IE tuple inference must prioritize conjunctive evidence and redundant evidence because tuples are short, noisy, and difficult to chain reliably.The introduction contrasts these needs with TABLEILP’s search for an optimal subgraph connecting question and answer terms through curated-table rows.Source: §1 · pp. 1–2
• 11.8%: TUPLEINF significantly outperforms TABLEILP across more than 1,300 science questions, without curated tables, and generalizes to higher grades.The gains persist when both solvers receive identical knowledge.Source: §1 · p. 2

2 Related Work

• The paper groups prior work into retrieval-based Web QA with large-scale knowledge and science QA with complex reasoning over smaller knowledge sources.These two categories frame distinct QA settings.Source: §2 · p. 2
• Retrieval-based Web QA uses structured knowledge bases, Open IE tuples, or extracted tables, but these approaches are limited to simple questions with a single query.Structured KB approaches are also limited by data coverage.Source: §2 · p. 2
• Science QA methods include probabilistic reasoning over small rule sets and simple information-retrieval techniques for science and Gaokao exams.The cited approaches include Markov Logic Networks, science tests, and Chinese Gaokao tests.Source: §2 · p. 2
• TABLEILP is TUPLEINF’s closest related solver, using manually defined join rules over curated tables whose rows may contain context that must be reconstructed from multiple Open IE tuples.Reliable join rules are difficult to define for open-vocabulary tuples.Source: §2 · p. 2

3.1 Tuple KB

• The tuple KB is built by retrieving domain-relevant corpus sentences with training-question and answer-choice terms, then extracting and aggregating Open IE tuples.For each test set, the system retrieves sentences using its corresponding training questions.Source: §3.1 · p. 2

3.2 Tuple Selection

• For each question, TUPLEINF ranks KB tuples by normalized TF-IDF after filtering question-only matches, and adds on-the-fly tuples from discriminative short, non-negated sentences using Jaccard.It retrieves 1,000 overlapping KB tuples before selecting the top 50; the on-the-fly path also selects 50 tuples.Source: §3.2 · pp. 2–3

3.3 Support Graph Search

• The support-graph search connects question terms to an answer choice through tuple fields, scoring structured tuples against potentially multi-sentence questions.Figure 1 illustrates a question, two KB tuples, and the answer option nitrogen.Source: §3.3 · p. 3, Figure 1 · p. 3

Objective Function

• Tuple-field edges use word overlap because WordNet path scores are unreliable for longer Open IE phrases; question-term weights also favor later, rarer terms.The model scales term coefficients by question position and inverse frequency among selected tuples.Source: Objective Function · p. 3

Constraints

• Because Open IE tuples lack schema and join rules, TUPLEINF uses a substantially simpler model than TABLEILP, eliminating reliance on hand-authored join rules and regular expressions.The simpler model reduces reasoning capability but can match TABLEILP on the Regents test and generalize better to different grade levels.Source: Constraints · p. 3
• ILP constraints require active edges to connect active vertices, constrain edge counts, and make each selected tuple connect question terms to an answer choice.Tuple-specific constraints also require the subject to be active and impose an ordering constraint to address reversed relations.Source: Constraints · pp. 3–4
• TUPLEINF combines parallel evidence only: each tuple must connect question words to the answer choice; reliable multi-hop reasoning would require inter-tuple predicate rules.Learning such rules for the Science domain is identified as future work.Source: Constraints · p. 4

4 Experiments

• The evaluation covers 4th-grade questions (1,220 train, 1,304 test) and 8th-grade questions (293 train, 282 test), comparing TUPLEINF with IR and TABLEILP.The 4th-grade set is a 10x larger superset of NY Regents questions; TABLEILP is the state-of-the-art structured inference baseline.Source: §4 · p. 4

4.1 Results

• TUPLEINF without curated knowledge outperforms TABLEILP by more than 11% on both question sets; with identical knowledge, its improvement is statistically significant.TUPLEINF(C + T) scores 61.4% versus TABLEILP’s reported 61.5% on the latter’s target Regents questions, without specialized rules.Source: Table 2 · p. 4, §4.1 · p. 4
• TUPLEINF and IR are complementary, and their ensemble improves on individual solvers; IR + TUPLEINF is consistently better than IR + TABLEILP.The two solvers differ on 47.3% of training questions.Source: Table 3 · p. 4, §4.1 · pp. 4–5
• TUPLEINF’s ILP is solved in half TABLEILP’s time, reducing overall question-answering time by 68.6%.On average, TABLEILP has 3,403 constraints and 982 variables, versus TUPLEINF’s 1,628 constraints and 588 variables.Source: §4.1 · p. 4
• On the 4th-grade set, PLEINF scores 58.2%, compared with TABLEILP’s ensemble score of 56.7%, a comparison the passage presents as further evidence of TUPLEINF’s strength.Source: §4.1 · p. 5

Missing Important Words

• A tuple supports water spreading out and filling a container but omits “completely,” a critical word in the question.The passage suggests detecting salient question words as a possible remedy.Source: §5 · p. 5

Lossy IE

• Open IE fails to extract tuples from an imperative sentence, and converting sentences into tuples may lose important information.In the salt-and-water example, the IR solver answers using a sentence that TUPLEINF cannot handle.Source: §5 · p. 5

Bad Alignment

• A high alignment score between “to breathe” and “breathe out” leads TUPLEINF to return carbon dioxide despite the phrases’ different meanings.The authors suggest improving phrase alignment to mitigate this issue.Source: §5 · p. 5

Out of scope

• TUPLEINF misses questions requiring it to model a state described in the question and reason over that state.The authors classify these questions as out of scope for the solver.Source: §5 · p. 5

6 Conclusion

• TUPLEINF reasons over a large, potentially noisy tuple knowledge base to answer complex questions.Source: §6 · p. 5
• The authors describe TUPLEINF as a new state-of-the-art structured solver for elementary-level science that does not rely on curated knowledge and generalizes to higher grades.Source: §6 · p. 5
• The authors identify lossy information extraction and misalignment errors as directions for future work on incorporating context and distributional measures.Source: §6 · p. 5

A Appendix: ILP Model Details

• Question terms (qterm) are extracted by chunking the question.Source: A Appendix: ILP Model Details · p. 7
• The chunker used to obtain question terms is an in-house tool.Source: A Appendix: ILP Model Details · p. 7
• The in-house chunker is based on the postagger from FACTORIE.Source: A Appendix: ILP Model Details · p. 7

Variables

• Each question term, tuple, tuple field, and answer choice has an active vertex variable in the ILP.Question-term coefficients start at 0.8 and are scaled by IDF, science-term, and question-position boosts.Source: Variables · p. 7
• Question-term coefficients use an IDF boost, a boost for terms on a 9K science-term list, and a position boost based on token index divided by question length.The IDF boost depends on the number of relevant tuples containing the term.Source: Variables · p. 7
• An edge’s coefficient is its word-overlap score, and qterm-to-field and field-to-choice edges are created only above thresholds of 0.1 and 0.2, respectively.Source: Variables · p. 7
• The model requires active variables to have active edges, active edges to have active endpoints, active fields to have active tuples, and exactly one active answer choice.Source: Variables · p. 7
• For tuples formed from table rows, both the subject and object must be active, equivalent to requiring at least two row cells to be active.The model also uses TABLEILP’s which-term boosting constraints.Source: Variables · p. 7

• SCIP is the optimization engine used to optimize the ILP model.Source: B Experiment Details · p. 7
• An answer choice is scored by forcing its active variable to one and using the resulting ILP objective value.Source: B Experiment Details · p. 7
• Evaluations use a 2-core 2.5 GHz Amazon EC2 Linux machine with 16 GB RAM.Source: B Experiment Details · p. 7

• For each question, the procedure selects the 7 best-matching tables by tf-idf score against the question tokens.Source: §B.1 · p. 7
• It selects the top 20 rows from each table using Jaccard similarity between the row and the question.This selection is described as the same as Khashabi et al. (2016).Source: §B.1 · p. 7
• Rows are converted using TABLEILP relations into tuples, with connected cells as subject and primary object and other cells as contextual objects.The procedure retains the top-scoring 50 tuples by Jaccard score.Source: §B.1 · p. 7

• An additional TABLEILP table is created containing all tuples in T.Source: §B.2 · p. 7
• Because TABLEILP uses fixed-length subject-predicate-object triples, each object Oi in a multi-object tuple becomes a separate triple (S; P; Oi).Source: §B.2 · p. 7