TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Bases

• Large KBs contain extensive schema spaces that require compositional generalization to novel item combinations and zero-shot generalization to unseen items or domains.Source: §1 · p. 1
• PLM-based KBQA must ground question and KB semantics and generate logical forms that are syntactically and semantically valid and executable.Source: §1 · p. 1
• Earlier retrieval approaches include similar annotated questions suited to i.i.d. settings, logical forms within two hops, and schema items retrieved without KB connectivity.TIARA also analyzes how context granularity relates to generalization levels and question types.Source: §1 · pp. 2–3
• TIARA retrieves entities, exemplary logical forms, and schema items as multi-grained contexts for PLM generation, with constrained decoding controlling the output space.Logical-form retrieval supplies semantic and structural context, while schema retrieval supplements it; constrained decoding uses KB-derived prefix trees to constrain generation.Source: §1 · p. 2
• TIARA outperforms prior state of the art by at least 4.1 F1 points on GrailQA and 1.1 on WebQuestionsSP, including systems with oracle entity annotations.On GrailQA, it also leads previous models in every generalization category, with a 4.7 F1-point improvement in zero-shot generalization.Source: §1 · p. 2

2.1 Knowledge Base Question Answering

• TIARA extends PLM-based sequence generation by improving zero-shot mention detection, matching entity-independent semantics through schema retrieval, and constraining decoding.Source: §2.1 · p. 3
• KBQA systems include graph-based information retrieval and semantic parsing; semantic parsing uses staged query-graph ranking or more flexible sequence-to-sequence generation.Source: §2.1 · p. 3

2.2 Neural Semantic Parsing with PLMs

• Prior PLM semantic-parsing studies use contextual retrieval or constrained decoding, and TIARA brings multi-grained retrieval together with constrained decoding for KBQA.Source: §2.2 · p. 3

3.1 Preliminaries

• The paper uses RDF triples and s-expressions as logical forms that can be converted to SPARQL and executed over knowledge bases.An exemplary logical form is a retrieved form used as PLM context; schema comprises classes and relations needed in logical forms.Source: §3.1 · p. 3

3.2 Multi-grained Retriever

• Entity retrieval detects question spans with a BERT-based classifier, generates candidates from FACC1 aliases, and disambiguates them using question and entity context.Span classification is used to better detect zero-shot mentions.Source: §3.2 · pp. 3–4
• Exemplary logical forms are enumerated from entity neighborhoods up to two hops, converted to s-expressions, and ranked against the question.Hop limits and function diversity mean enumeration may not cover the target; questions without entities yield no retrieved exemplary forms.Source: §3.2 · p. 4
• A cross-encoder scores question–schema matches to retrieve semantically relevant classes and relations beyond entity neighborhoods.Class and relation retrievers are trained separately.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Target Logical Form Generation

• TIARA combines question, retrieved entities, exemplary logical forms, and schema items as T5 input to generate the target logical form.Beam search retains top-k sequences, with constrained decoding applied during inference.Source: §3.3 · pp. 4–5

3.4 Constrained Decoding

• Constrained decoding limits operator tokens to those allowed by s-expression syntax and schema-item tokens to valid trie continuations.KB classes and relations are stored in separate tries.Source: §3.4 · p. 5, Figure 4 · p. 5

4.1 Setup

• Evaluation uses GrailQA and WebQSP, with EM and F1 for GrailQA and F1 and hits@1 for WebQSP.For WebQSP, hits@1 is averaged over 100 random answer selections because semantic-parsing systems produce unordered answers.Source: §4.1 · p. 5

4.2 Overall Performance

• TIARA leads all three GrailQA generalization levels on EM and F1; compositional and zero-shot F1 gains are 5.3 and 4.7 points, with 76.5 compositional F1.Source: §4.2 · p. 6
• On WebQSP, TIARA outperforms existing IR- and SP-based methods in F1 without fixed-hop assumptions or oracle entity annotations; with oracle annotations, its F1 rises to 78.9 points.Table 2 reports F1 and hits@1; TIARA’s hits@1 is obtained by randomly selecting one answer per question 100 times.Source: §4.2 · pp. 6–7, Table 2 · p. 6

5.1 Ablation Study

• 2.7 F1 points are gained compositionally by multi-grained contexts, while generation reaches 72.1 F1 points without schema retrieval or constrained decoding.Direct exemplary logical form prediction reaches 72.9 F1 points overall but only 60.7 F1 points in compositional generalization.Source: §5.1 · p. 7
• 26.5 F1 points are lost in zero-shot when exemplary logical form retrieval is removed, underscoring its critical role.The analysis attributes this role to connected structures in exemplary logical forms, which help address PLMs’ difficulty perceiving KB structure.Source: §5.1 · p. 7
• 25.2 F1 points are lost by removing schema retrieval when exemplary logical forms are absent; schema retrieval yields 50.6 F1 points versus 2.3 with entity-only context in zero-shot.Removing schema retrieval decreases overall F1 by 2.7 points.Source: §5.1 · p. 7
• 0.4 F1 points overall come from constrained decoding when exemplary logical forms and schema items provide effective context.Without those contexts, constrained decoding improves F1 by 5.2 points over the version without retrieval or constrained decoding.Source: §5.1 · p. 7
• 3.4 F1 points are gained in zero-shot with SpanMD replacing BERT-NER, and overall F1 improves by 2.2 points.Mention detection sets the entity retriever’s upper-bound recall.Source: §5.1 · p. 8

5.2 In-Depth Analysis

• 3x F1 improvement is reported for superlative questions; TIARA also performs well across relation counts except four, where only two questions were evaluated.For superlatives lacking entities and exemplary logical forms, schema items remain available as context.Source: §5.2 · p. 8

5.3 Error Analysis

• In 50 sampled GrailQA validation errors, entity retrieval (46%) and syntactic errors (26%) dominate; semantic errors (12%), false negatives (6%), and miscellaneous cases (10%) make up the remainder.Semantic errors mostly involve incorrect schema selection when correct items are provided, and 83.3% of those errors come from zero-shot instances.Source: §5.3 · p. 8

5.4 Case Study

• The case studies show complementary failure modes: exemplary logical forms support graph connectivity, schema retrieval supplements limited exemplars, and constrained decoding excludes nonexistent KB items.The examples include a relation disconnected from the entity, a literal-and-function structure missing from an exemplar, and a predicted schema phrase absent from the KB.Source: §5.4 · p. 8

6 Conclusion

• TIARA’s retrieved contexts provide both semantic and syntactic support for generating target logical forms.The conclusion presents this as a way to move beyond insufficient exemplary logical forms and isolated schema items.Source: §6 · pp. 8–9
• The stage-by-stage analysis finds that constrained decoding prunes invalid search and reduces generation errors, while improved entity linking addresses a key zero-shot bottleneck.Flexible-scope schema retrieval supplements semantics, and retrieval together with constrained decoding can further empower PLM generalization.Source: §6 · p. 9
• A remaining research direction is bridging natural-language PLM pre-training and structured-KB semantic parsing.The paper says this bridge could be explored during the pre-training phase.Source: §6 · p. 9

Limitations

• Training data requires expensive, specialized crowdsourcing to collect logical-form annotations.Source: Limitations · p. 9
• Without caching, logical-form enumeration takes more than 7 seconds per question, while schema retrieval takes 1.41 seconds per question.The paper says the enumeration time needs improvement for certain practical scenarios and identifies schema retrieval as the major additional time cost compared with RnG-KBQA.Source: Limitations · p. 9
• The semantic-parsing approach applies well to RDF graphs but is not fully applicable to tables or unstructured data.The paper identifies both as important sources of knowledge in the wild.Source: Limitations · p. 9

A.1 Entity Retrieval Performance

• SpanMD improves recall over BERT-NER by 2.2 points overall and 4.7 points in the zero-shot setting.Source: §A.1 · p. 13, Table 6 · p. 13
• The entity-retrieval comparison includes a popularity-based GrailQA baseline and RnG-KBQA, which uses linked-entity relation information.Source: §A.1 · p. 13
• 5.0 F1 points: TIARA’s entity retriever surpasses all baseline models by at least this amount on the GrailQA validation set.Compared with the strongest baseline, Ye et al. (2021), it improves both precision and recall.Source: §A.1 · p. 13, Table 7 · p. 13
• For WebQSP, TIARA uses the off-the-shelf ELQ linker, as prior SOTA systems do.ELQ jointly performs mention detection and entity disambiguation and is trained on Wikipedia and WebQSP data.Source: §A.1 · p. 13

A.2 Schema Retrieval Performance

• The reported schema-retrieval measure is recall of the top 10 retrieved items on GrailQA validation.Source: Table 8 · p. 13
• TIARA’s top-10 schema recall exceeds ReTraCk’s across all three GrailQA generalization levels.Source: §A.2 · p. 13, Table 8 · p. 13
• ReTraCk uses 100 classes and 150 relations per question, which does not result in higher QA performance.Source: §A.2 · p. 13

B Performance on the WebQSP Validation Set

• Table 9 reports F1 and hits@1 results in percentages on the WebQSP validation set.WebQSP does not contain an official validation set; the dataset split is described in Section 4.1.Source: Table 9 · p. 13, B Performance on the WebQSP Validation… · p. 13

C Settings and Hyperparameters

• Hyperparameters were manually and nonexhaustively tuned using validation-set F1, with generator batch sizes [2, 4, 6, 8] and learning rates [1e-5, 3e-5, 5e-5] searched.Source: C Settings and Hyperparameters · p. 14
• The reported model sizes are 110M parameters for BERT-base-uncased and 220M for T5-base.Source: C Settings and Hyperparameters · p. 14

• Official scripts are used to evaluate both datasets.Source: D Evaluation Details · p. 14
• WebQSP hits@1 is the exception to the use of official evaluation scripts.Source: D Evaluation Details · p. 14
• The evaluation details identify separate scripts for GrailQA and WebQSP.Source: D Evaluation Details · p. 14