Compositional Semantic Parsing with Large Language Models

• Compositionality lets people understand and produce potentially infinite novel combinations from known components.Source: §1 · p. 1
• Prior compositional-generalization methods often rely on task-specific architectures or training procedures, while data augmentation is limited in the data it supports.Source: §1 · p. 1
• SCAN results leave open whether least-to-most prompting transfers to realistic tasks, since SCAN uses a synthetic language with a tiny vocabulary and simple grammar.Source: §1 · p. 1
• Dynamic least-to-most prompting uses LM-predicted syntactic parsing to decompose inputs, dynamically selects exemplars, then sequentially generates subproblem answers.Source: §1 · p. 1
• 95% accuracy averaged across CFQ MCD splits sets a new state of the art, reduces prior-best error by about 45%, and uses about 1% of traditional training data.Source: §1 · p. 2

2.1 COMPOSITIONAL GENERALIZATION

• Compositional generalization is the ability to use and comprehend unseen combinations given conceptual primitives and a few combinations.Source: §2.1 · p. 2

2.2 LEAST-TO-MOST PROMPTING ENABLES COMPOSITIONAL GENERALIZATION

• Least-to-most prompting decomposes a complex problem into easier subproblems and solves them sequentially, appending earlier answers as context for later steps.Source: §2.2 · p. 2

2.3 LEAST-TO-MOST PROMPTING: LIMITATIONS AND CHALLENGES

• CFQ and COGS pose three transfer challenges: harder decomposition, knowledge exceeding a single prompt, and context-dependent constituent translations.Source: §2.3 · p. 3

2.1 COMPOSITIONAL GENERALIZATION

• CFQ and COGS require compositional generalization across systematic gaps, but their richer language makes decomposition harder than SCAN’s fixed-symbol commands.Source: §2.1 · p. 2, §2.3 · p. 3

2.3 LEAST-TO-MOST PROMPTING: LIMITATIONS AND CHALLENGES

• CFQ and COGS contain richer linguistic structures, while CFQ has over 50 Freebase types and relations and COGS uses hundreds of verbs whose translations depend on lexical details.Source: §2.3 · p. 3
• In CFQ, the same constituent can translate differently by context, so subproblems need enough context rather than isolated translation.Source: §2.3 · p. 3
• The proposed adaptation obtains problem reduction through multi-step syntactic parsing and dynamically selects exemplars to cover as many decomposition parts as possible.Source: Figure 2 · p. 4

3 DYNAMIC LEAST-TO-MOST PROMPTING

• The method combines LM-predicted syntactic decomposition, dynamic exemplar selection from a small pool, and sequential prediction of simpler subproblems before the final output.The sequence is generated from the tree structure, with each subproblem's solution available before proceeding to the next.Source: §3 · p. 4

3.1 DECOMPOSITION USING LM-BASED SYNTACTIC PARSING

• For realistic tasks, syntactic parsing guides a tree decomposition whose subproblems, like the input, are well-formed sentences.This tree structure differs from the linear decomposition of traditional least-to-most prompting.Source: §3.1 · p. 4, Figure 3 · p. 5
• The parsing prompts break decomposition into steps such as clause, noun-phrase, and verb-phrase identification, phrase annotation, and verb normalization.Each step is illustrated with exemplars; some prompts are applied iteratively to identify nested clauses.Source: §3.1 · p. 5, §3.2 · p. 5

3.2 DYNAMIC EXEMPLAR SELECTION

• The exemplar pool contains 1,000 randomly sampled training examples per CFQ split, while COGS uses 89 manually selected examples covering verb and verb-phrase types.The COGS pool is used in addition to a static context of 28 exemplars.Source: §3.2 · p. 5
• Exemplar selection aims to cover as many input-tree nodes as possible through top-down matching of large subtrees and bottom-up coverage of leaf phrases.When a leaf phrase has multiple matches, the procedure prefers an exemplar containing it in a similar anonymized subtree.Source: §3.2 · p. 5

3.1 DECOMPOSITION USING LM-BASED SYNTACTIC PARSING

• Because no gold syntactic parses were available, parsing was not directly evaluated; manual inspection found desirable outputs for CFQ and COGS.The authors attribute this to breaking parsing into individually prompted steps with appropriate exemplars.Source: §3.2 · p. 5

3.3 SEQUENTIAL SOLUTION

• Figure 4 depicts dynamic least-to-most as sequentially solving subproblems extracted through a separate prompt before generating the final output.The illustrated prompt snapshot omits the dynamically selected exemplars and the fixed exemplar list prepended to each prompt.Source: Figure 4 · p. 6, §3.3 · p. 6
• Unlike traditional least-to-most prompting, constituents are not translated in isolation because they may be ill-formed or context-dependent; the tree is linearized into increasingly complex subproblems.Their solutions are predicted sequentially before the final output.Source: §3 · p. 4, §3.3 · p. 6

4 BASELINE: PROMPTING WITHOUT DECOMPOSITION

• Chain-of-thought generates intermediate steps before the final answer but lacks the decomposition step used by least-to-most prompting.The authors note that this absence may limit its effectiveness for compositional generalization.Source: §4 · p. 6

4.1 CHAIN-OF-THOUGHT PROMPT DESIGN

• The prompt first categorizes the query, then generates quasi-alignments between text and output statements.Its intermediate constituents and clauses need not map exactly to the input and output.Source: §4.1 · p. 6
• This flexible chain-of-thought design accommodates CFQ's variables, which are incompatible with synchronous grammar training and inference.The quasi-alignments represent synchronicity in the same spirit as other higher-performing semantic parsers.Source: §4.1 · p. 6

4.2 DYNAMICALLY SELECTING EXEMPLARS BASED ON LEXICAL SIMILARITY

• Without decomposition, chain-of-thought ranks exemplars by bag-of-words similarity and selects them iteratively to prioritize relevant words not yet covered.The selection is nondeterministic: early choices can change the resulting exemplar list.Source: §4.2 · p. 6
• Sampling multiple exemplar lists is useful for chain-of-thought, and error analysis found that a fixed list of basic exemplars greatly improved CFQ performance.The fixed-list finding is reported as an error-analysis result.Source: §4.2 · p. 6
• For vanilla and chain-of-thought prompts, the authors sample multiple outputs per exemplar list using temperature-based decoding, then aggregate predictions by plurality vote with self-consistency.The method cites Wang et al. (2022), Shi et al. (2022), and Li et al. (2022).Source: §4.2 · p. 7

5.1 DATA

• CFQ has three MCD splits with 95,743/11,968/11,968 train/validation/test examples, while COGS has 24,155/3,000/21,000 examples in those respective splits.Potential exemplar pools comprise 1,000 examples per CFQ split and 89 manually selected COGS examples.Source: §5.1 · p. 7
• Accuracy is exact match, with CFQ outputs normalized and COGS outputs given missing closing parentheses; CFQ also uses human-readable Freebase identifiers and removes redundant clauses.The CFQ-specific preprocessing and evaluation steps are tailored to prompting; T5-base reproduction matched published exact match within 1–2 points on each MCD split.Source: §5.1 · p. 7, §5.3 · p. 7

5.2 EXPERIMENTAL DESIGN CHOICES

• Dynamic least-to-most uses greedy decoding and a complexity-dependent number of exemplars and subproblems; vanilla and chain-of-thought use 4 lists of 15 exemplars and 4 sampled outputs per list.The latter's 16 outputs are aggregated with self-consistency; code-davinci-002 was used for all experiments.Source: §5.2 · p. 7

5.3 RESULTS

• 95.0% accuracy across CFQ's MCD test splits sets a new state of the art while using about 1% of training data; Table 1 reports the split-wise results.Traditional approaches are fully supervised.Source: §5.3 · p. 7, Table 1 · p. 8
• Vanilla few-shot and chain-of-thought perform worse without self-consistency.The comparison is reported in a results footnote.Source: §5.3 · p. 7
• 99.2% accuracy on the COGS generalization set is achieved with 0.4% of training data, despite its output space differing substantially from CFQ.Table 2 reports accuracy on the COGS generalization set.Source: §5.3 · p. 8, Table 2 · p. 8

Prompting methods

• 94.4% on CFQ MCD1 beats chain-of-thought (87.2%) and vanilla prompting (80.8%); it is also more than 2x faster than chain-of-thought.The comparison uses a random 500-sentence subset of the MCD1 test split.Source: §6 · p. 8

Exemplar pool

• Performance degrades especially when the exemplar pool contains only 15 examples, while methodical selection yielded high COGS performance with 89 instances.With 15 randomly selected exemplars, each input receives nearly the same examples and many properties go unrepresented.Source: §6 · p. 8

Model choice

• The authors chose Codex over text-based GPT-3 because initial experiments showed better performance, consistent with prior semantic-parsing work (Shin et al., 2021; Shin & Van Durme, 2022).Source: §6 · p. 8

Leakage and interpretation

• For pretrained models, test-data leakage is identified as a potential concern.Source: §6 · p. 8
• Given vanilla few-shot prompting's inferior performance, the authors attribute prompting success to in-context learning rather than memorization; least-to-most translates intermediate subproblems guaranteed unseen during training.Source: §6 · p. 9

Compositional generalization

• Although Zhou et al. (2022) report 99.7% accuracy on SCAN, prior prompting for semantic parsing still shows limited compositional generalization (Qiu et al., 2022b).Source: §7 · p. 9

Semantic parsing approaches

• Without specialized architectures or training algorithms, generic decomposition-based prompting achieves strong CFQ and COGS results and state-of-the-art performance on CFQ.Source: §7 · p. 9

Prompting

• Unlike SeqZero’s fixed, rule-based decomposition of SQL parts, this approach prompts from syntactic parses, applies beyond SQL-like outputs such as COGS, and uses LLM prompting without finetuning.Source: §7 · p. 9

8 CONCLUSION

• Using about 1% of traditional finetuning approaches’ training data, dynamic least-to-most prompting achieves state-of-the-art performance on a difficult semantic-parsing benchmark.The benchmark measures compositional generalization.Source: §8 · p. 9
• The authors expect dynamic least-to-most prompting to support quick adaptation across tasks and domains, especially knowledge-intensive applications using precise semantic parsing.They also expect the findings to facilitate research on compositional generalization through task decomposition.Source: §8 · p. 9

A.1.1 STEP 1: NOUN PHRASE IDENTIFICATION

• Prompting-based noun phrase identification decomposes CFQ questions by grouping noun phrases within parentheses.Examples include bracketing “a production company” and “M2” as separate phrases.Source: §A.1 · p. 16, §A.1.1 · p. 19

A.1.2 STEP 2: SUBCLAUSE IDENTIFICATION

• In two examples, the answers use nested parentheses to group an N1 phrase with a following clause introduced by that, while retaining the larger predicate directed M5 or meet N3.Source: §A.1.2 · p. 19

A.1.3 STEP 3: VERB PHRASE IDENTIFICATION

• Verb phrase identification segments predicates within subclauses, including coordinated verb phrases such as “influenced” and “was influenced by.”The examples show the verb phrases separated within a subclause introduced by “that.”Source: §A.1.3 · p. 21
• Verb-phrase identification groups coordinated predicates and distinguishes them from their arguments in example questions, including main-clause and relative-clause forms.Examples show both coordinated predicates such as “influence and marry” and separate predicate–argument groupings.Source: §A.1.3 · pp. 23–24

A.1.4 STEP 4: PART OF SPEECH TAGGING

• Part-of-speech tagging examples mark nouns, prepositions, modifiers, and verbs, with adjectives explicitly marked using /A.The examples use N, P, M, and V labels in varied noun-phrase and verb-phrase structures.Source: §A.1.4 · pp. 25, 27

A.1.5 STEP 5: VERB NORMALIZATION

• Verb normalization maps inflected forms to base forms, as in “edited by” to “edit by” and “written by” to “write by.”Source: §A.1.5 · p. 28

A.2 CFQ EXEMPLAR SELECTION: DETAILS

• CFQ inputs draw 4–35 exemplars dynamically from a 1,000-example training pool, selected to cover as many input decomposition-tree nodes as possible.The number selected depends on input complexity and the similarity of candidate exemplars.Source: §A.2 · p. 28

A.2.1 TOP-DOWN MATCHING

• Top-down matching converts exemplar and input decomposition trees into syntactic templates by anonymizing concrete leaves while retaining their types.For the Figure 6 question, the resulting template begins “Which N (V (N that …”.Source: §A.2.1 · p. 29
• Top-down matching first seeks exemplars for the full input template, then progressively collapses template nodes until selected examples cover it as well as possible.Source: §A.2.1 · p. 29

A.2.2 BOTTOM-UP MATCHING

• Bottom-up matching seeks exemplars covering each unanonymized leaf, preferring examples where the leaf appears in syntactically similar subtrees.For each leaf, other material is anonymized before searching for exemplars that share its subtrees.Source: §A.2.2 · p. 29

A.3 CFQ SOLUTION: DETAILS AND PROMPT

• Dynamic least-to-most prompting linearizes the composition tree into increasingly complex, well-formed subquestions because constituents may not translate correctly in isolation.A tree walk keeps the most generic variant of each top-level node, then expands these nodes step by step.Source: §A.3 · p. 30

A.3.1 PART 1: STATIC PROMPT CONTEXT.

• The static prompt context supplies 12 grounding examples with translation rationales to teach the model to translate simple subquestions, especially when selected exemplars are dissimilar.Source: §A.3.1 · p. 30

A.3.2 PART 2: DYNAMICALLY SELECTED EXEMPLARS

• After the static context, the prompt adds exemplars dynamically selected for the input question using the matching process described in Section A.2.Source: §A.3.2 · p. 31

A.3.3 PART 3: SEQUENTIAL QUESTIONS

• Sequential prompting grounds the simplest subquestion with “Partial Q:” and rationale-based reasoning, then asks increasingly complex questions while appending each answer to context.For strict extensions, comments highlight what to add; for each subquestion, the model is queried and its answer appended before the next request.Source: §A.3.3 · p. 33

B LEAST-TO-MOST PROMPTING FOR COGS

• The COGS application is a variation of the CFQ procedure, with its detailed prompts illustrating how dynamic least-to-most prompting processes nested-clause and nested-phrase examples.Source: B LEAST-TO-MOST PROMPTING FOR COGS · p. 34, Figure 7 · p. 34

B.1 COGS DECOMPOSITION: DETAILS AND PROMPTS

• COGS decomposition uses four steps, including iterative subclause and prepositional-phrase decomposition to handle nesting up to level 12.Subclauses and prepositional phrases are extracted one at a time until all are decomposed.Source: §B.1 · p. 34
• Verb normalization adds base forms without replacing the originals, supporting higher recall when selecting exemplars.For example, “shortened” yields “shorten”; in a nested-clause example, “said,” “liked,” “appreciated,” “believed,” and “was posted” are normalized to “say,” “like,” “appreciate,” “believe,” and “post.”Source: §B.1 · pp. 35–36

B.1.3 ITERATIVE PREPOSITIONAL PHRASE DECOMPOSITION AND NOUN PHRASE ANNOTATION

• Prepositional phrases are decomposed iteratively, while definite noun phrases are marked with a star.A nested prepositional phrase example requires five iterations; noun-phrase annotations mark “the atom” as “the * atom.”Source: §B.1 · pp. 35–36, §B.1.3 · p. 39

B.1.1 ITERATIVE SUBCLAUSE DECOMPOSITION

• For nested clauses, the prompt identifies a subject, verb, and embedded clause at each iterative decomposition step.Examples show successive extractions through clauses nested within clauses.Source: §B.1.1 · pp. 36–37

B.1.2 PHRASE IDENTIFICATION

• Phrase-identification prompts label subjects, verbs, objects, and prepositional arguments across varied sentence forms.Examples include passive constructions, ditransitive verbs, infinitives, and sentences with embedded prepositional phrases.Source: §B.1.2 · pp. 37–38

B.1.3 ITERATIVE PREPOSITIONAL PHRASE DECOMPOSITION AND NOUN PHRASE ANNOTATION

• Prepositional-phrase examples mark the noun-phrase head and preposition, leaving the remaining phrase as the PP constituent.For example, “the helmet on the smartphone” is represented as “(The * helmet) (on) P=(the smartphone).”Source: §B.1.3 · p. 40

B.1.4 VERB PHRASE NORMALIZATION

• Verb-phrase normalization maps “was snapped,” “was given,” “was lended,” and “was brought” to “snap,” “give,” “lend,” and “bring,” respectively.Source: §B.1.4 · pp. 41–42

B.2 COGS EXEMPLAR SELECTION: DETAILS

• CFQ uses a random 1,000-example candidate pool, whereas COGS exemplars were manually selected to cover verbs, including rare unergative and unaccusative verbs.COGS exemplars use parsed natural-language sentences with parentheses and other annotations to aid translation.Source: §B.2.1 · p. 42

B.2.1 EXEMPLAR POOL

• The COGS pool combines 62 exemplars covering its unergative and unaccusative verbs with 27 examples across nine verb-phrase types, for 89 total.COGS exemplars were manually selected, while CFQ used a random pool of 1,000 training examples; the COGS sentences use parsed versions with additional annotations.Source: §B.2.1 · pp. 42, 45
• The COGS exemplar pool contains annotated active and passive examples that map sentence forms to predicate roles such as agent, theme, and recipient.Examples include transitive clauses, passives with agents, and passives without an expressed agent.Source: §B.2.1 · p. 46

B.2.2 EXEMPLAR MATCHING

• The COGS procedure selects exemplars by the innermost subclause, using verb type to resolve subject roles or three examples when both subject and object are present.The matching heuristics resemble those for CFQ, and static prompt context handles nested clauses and prepositional clauses.Source: §B.2.2 · p. 47

B.3 COGS SOLUTION: DETAILS AND PROMPTS

• The COGS solution linearizes a composition tree into increasingly complex, well-formed sentence subproblems and queries the language model for each.Each query prompt consists of static context, dynamically selected exemplars, and sequential subproblems.Source: §B.3 · p. 47

B.3.1 PART 1: STATIC PROMPT CONTEXT

• A static prompt demonstrates systematic composition of subclauses and prepositional phrases, which sequential prompting reduces to one composition step at a time.The prompt demonstrates nesting only to level 2 and uses annotated natural-language sentences to aid translation.Source: §B.3.1 · p. 48

B.3.2 PART 2: DYNAMICALLY SELECTED EXEMPLARS

• After the static context, the COGS prompt adds exemplars dynamically selected for the input using the matching procedure described in Section B.2.The appendix refers to two running examples for the selected exemplars rather than repeating them here.Source: §B.3.2 · p. 49

B.3.3 PART 3: SEQUENTIAL SUBPROBLEMS

• Sequential prompting solves the simplest subproblem first, appends each model reply before the next request, and continues until the original problem is solved.The process builds up from a basic clause through nested clauses or increasingly complex prepositional phrases.Source: §B.3.3 · p. 49

C.1 BOOTSTRAP PROMPT

• The CFQ chain-of-thought procedure bootstraps rationale generation by annotating five exemplars, then uses them to prompt the model to generate rationales for the remaining pool.The bootstrap prompt combines 15 vanilla exemplars selected by bag-of-words similarity with five manually annotated chain-of-thought examples.Source: C CHAIN-OF-THOUGHT PROMPTING FOR CFQ · p. 51, §C.1 · p. 51

C CHAIN-OF-THOUGHT PROMPTING FOR CFQ

• The CFQ evaluation uses mixed vanilla and chain-of-thought exemplars, with chain-of-thought prompting evaluated only on CFQ, not COGS.Because chain-of-thought examples are verbose, the evaluation prompt mixes formats to keep it relatively short.Source: C CHAIN-OF-THOUGHT PROMPTING FOR CFQ · p. 51

C.1 BOOTSTRAP PROMPT

• Bootstrap examples show queries decomposed into query types and constituent relations before the semantic parse is assembled.The examples then combine those components into SELECT DISTINCT or count(*) queries.Source: §C.1 · p. 52

C.2 PREDICTION PROMPT

• The prediction prompt uses 10 vanilla input/output exemplars followed by 5 chain-of-thought exemplars.The chain-of-thought format is shown in reduced font size because it can be verbose.Source: §C.2 · p. 53
• A mapping from Freebase IDs to human-readable strings is applied to CFQ to make prompting more feasible.The table truncates the IDs to 120 characters.Source: Table 3 · p. 54

D.1 CFQ PROCESSING AND EVALUATION

• T5-base results matched previously published MCD1, MCD2, and MCD3 results within 1–2 points after the CFQ processing steps.The authors verified that these steps did not alter fully supervised performance.Source: §D.1 · p. 55

D.1.1 CFQ PREPROCESSING

• Preprocessing replaces excessively long Freebase identifiers with human-readable strings and strips FILTER statements associated with “sibling of” and “married to.”Those FILTER statements are considered trivial to add after prediction because the properties imply the self-inequality condition.Source: §D.1.1 · p. 55

D.1.2 CFQ POSTPROCESSING AND EVALUATION

• Clause sanitization discards malformed clauses, while inverse properties are standardized and their arguments flipped when needed.The “sequel of” error is not corrected because that property is absent from the original vocabulary.Source: §D.1.2 · p. 55
• Bidirectional-property arguments and SPARQL statements are sorted, while variables are relabeled and normalization repeats until ordering stabilizes.SPARQL statement order and variable labels are arbitrary, so the procedure canonicalizes them.Source: §D.1.2 · p. 55
• Postprocessing strips predicted types implied by relations, then measures accuracy using exact string match after applying the steps to gold and predicted parses.The authors note that retaining such a type can produce a semantically equivalent query but hurt language-model accuracy.Source: §D.1.2 · p. 55

D.2 COGS POSTPROCESSING AND EVALUATION

• 99.2% exact match on the COGS generalization test set followed a closing-parenthesis fix, up from 97.8%.The fix adds closing parentheses until every opening parenthesis is matched.Source: §D.2 · pp. 55–56
• The original COGS generalization test set has 50 mislabeled examples, so expected upper-bound performance is about 99.7%.The authors used the original data to compare with previous work, although a revised version had been released.Source: §D.2 · p. 56

D.3 HYPERPARAMETERS

• Initial hyperparameter tuning used a 100-sentence validation subset.The paper reports separate prompt and generation hyperparameters.Source: §D.3 · p. 56

D.3.1 PROMPT HYPERPARAMETERS

• Prompt settings used 12 static exemplars for CFQ and 28 for COGS, with 4–35 dynamic exemplars for CFQ and 1–3 for COGS.Dynamic exemplar counts are determined automatically from the decomposition tree.Source: §D.3.1 · p. 56
• Chain-of-thought prompts used five rationale exemplars and ten vanilla input/output exemplars, selected like the chain-of-thought exemplars.Only chain-of-thought exemplars that had produced the correct semantic parse were retained; 884 of 1,000 predictions succeeded.Source: §D.3.1 · p. 56

D.3.2 GENERATION HYPERPARAMETERS

• The authors did not conduct extensive hyperparameter search, using reasonable settings based on prior work and identifying temperature as the main toggle.Dynamic least-to-most did not use temperature.Source: §D.3.2 · p. 57
• Generation used code-davinci-002, temperature 0.7 when sampling or 0.0 when greedy, top-p 1.0, and zero presence and frequency penalties.Source: §D.3.2 · p. 57

E.1 INITIAL PROMPTING ATTEMPTS FOR CFQ

• 0% accuracy resulted from prompting without examples, despite SPARQL-like output; the authors infer that pretraining had not supplied specific CFQ examples.The prompt instructed the model to translate English to SPARQL.Source: §E.1 · p. 57
• The illustrated prompt pairs an English question with a generated SPARQL query.The question and query are shown in the example passages.Source: §E.1 · p. 57

E ADDITIONAL ANALYSIS

• Table 4 covers accuracy and error frequencies for dynamic least-to-most ablations on 100 MCD1 validation examples.The caption notes that CoT grounding is used only for CFQ.Source: Table 4 · p. 57

E.2 DYNAMIC LEAST-TO-MOST PROMPTING ON CFQ: ABLATIONS AND ERROR ANALYSIS

• The qualitative error analysis examined ablations on 100 MCD1 validation examples.It was conducted during development to examine the impact of components in dynamic least-to-most prompting.Source: §E.2 · p. 57
• The four setups progress from decomposition-based retrieval, to hints, sequential least-to-most prompting, and finally CoT grounding.The first setup retrieves exemplars without sequential prompting; the full implementation adds a constant prefix with basic subproblems solved in a chain-of-thought-like way.Source: §E.2 · p. 58
• 55% accuracy with retrieval alone rose to 67% with decomposition hints; the initial setup's 24 errors mostly involved types and Cartesian-product expansion, including 11 such expansion errors.The hints add parentheses to the question to indicate compositional structure.Source: §E.2 · p. 58
• 99% accuracy followed CoT grounding, mainly through eliminating type errors, in the 100-example CFQ validation ablation.The ablation was performed on a 100-example subset of the MCD1 validation set.Source: §E.2 · pp. 57–58
• Example parses disagree over whether a person-type predicate belongs in the generated query.One generated answer includes that predicate where the gold parse does not; another omits it where the gold parse includes it.Source: §E.2 · p. 58
• Other examples show extra or missing predicates and differences in which entity or relation a predicate attaches to.The examples include mismatches involving direction and the assignment of a film-director type.Source: §E.2 · pp. 58–59

E.3 FAIR COMPARISON AGAINST OTHER PROMPTING TECHNIQUES

• 94.4% accuracy for dynamic least-to-most exceeded chain-of-thought at 75.4% and vanilla few-shot at 69.8% under matched greedy decoding.The comparison used the same 500-sentence MCD1 validation subset without self-consistency.Source: §E.3 · p. 59

F REPRODUCIBILITY SUMMARY

• Exact prompts used to obtain predictions are provided for CFQ and COGS, alongside instructions for selecting their exemplars.The prompts include exemplars found from a concrete input.Source: F REPRODUCIBILITY SUMMARY · p. 59
• The summary directs readers to the preprocessing, output-normalization, and evaluation procedures for semantic parsing with Freebase identifiers.These procedures are described across the main text and appendices.Source: F REPRODUCIBILITY SUMMARY · p. 59
• Exemplar-pool construction and the hyperparameters used are documented, and the authors report that extensive hyperparameter search was unnecessary.The summary points to Section 3.2 for exemplar pools and Section 5.2 and Appendix D.3 for hyperparameters.Source: F REPRODUCIBILITY SUMMARY · p. 59