Ask Me Anything: A simple strategy for prompting language models

Background

• LLMs enable in-context learning by applying natural-language task specifications and demonstrations to new tasks without task-specific training [Brown et al., 2020, Bommasani et al., 2021].Prompts use templates containing placeholders for task descriptions and input-output demonstrations.Source: §1 · p. 1

Problem

• Small prompt changes can cause large performance variations, with additional dependence on LLM family and model size [Zhao et al., 2021, Holtzman et al., 2021, Ouyang et al., 2022, Sanh et al., 2022, inter alia., Wei et al., 2022a, Lampinen et al., 2022].This brittleness motivates painstaking prompt design and task-by-task manual exploration of large strategy spaces [Mishra et al., 2021, Wu et al., 2022].Source: §1 · p. 1

Method

• The strategy runs a collection of prompt chains that generate inputs into question-answer form, then combines the resulting noisy answers using weak supervision.The supplied method overview presents these as consecutive steps: generating inputs to question and answer, followed by weakly supervised combination.Source: §1 · p. 2

Motivation

• AMA addresses the limitations of same-format prompt voting, which produced only a 4% gain for CB and a 2% decline for RTE, by targeting prompt formats that work across tasks and models.The method also responds to the scalability challenge of manually rewriting varied task inputs into effective formats.Source: AMA PROMPTING · p. 2

Effective prompt formats

• AMA identifies open-ended question-answering prompts as more effective than restrictive-output prompts, with conversion of three SuperGLUE tasks yielding a 72% performance improvement.Its general structure forms questions from task inputs and prompts the LLM to answer them across diverse benchmark tasks.Source: AMA PROMPTING · p. 2

Scalable prompt collection

• AMA recursively reformats inputs through reusable question()- and answer()-prompts, whose varied pairings create prompt chains that collect multiple votes across inputs.The resulting pipeline is answer(question(x)); functional prompts are reused across inputs while different demonstrations provide variety.Source: AMA PROMPTING · pp. 2–3, Figure 1 · p. 2

Prompt aggregation

• Weak supervision aggregates noisy prompt-chain predictions by modeling varying accuracies and correlated errors, improving over majority vote by up to 8.7 points and recovering dependencies on 9 tasks for gains up to 9.6 points.Prompt accuracies varied by 9.5% on average, and error overlap was 69% higher than under independent errors, making majority vote unreliable.Source: AMA PROMPTING · pp. 2–3

Empirical results

• An average 10.2% ± 6.1% absolute (21.4% ± 11.2% relative) lift over the few-shot (k = 3) baseline across models enables a 30x-smaller open-source LLM to match or exceed GPT3-175B on 15 of 20 benchmarks.AMA was evaluated on 20 benchmarks and 14 open-source LLMs from four model families spanning 125M–175B parameters; gains were largest when required knowledge appeared in the provided context.Source: AMA PROMPTING · p. 3

Training based strategies

• AMA complements training-based approaches by improving prompting performance for off-the-shelf language models without additional fine-tuning.Prior work scales training or fine-tunes language models over prompts [Kaplan et al., 2020, Chowdhery et al., 2022; Wang et al., 2022a, Wei et al., 2022b, Sanh et al., 2022, Ouyang et al., 2022].Source: §2 · p. 3

Prompt-engineering

• Prior prompt-engineering methods use prescribed formats that may be narrow in task, model, or size, or require manual task-specific input rewriting, whereas AMA recursively uses the LLM to transform inputs.AMA’s recursive use of the LLM is similar to Jung et al. [2022], which focuses on commonsense reasoning.Source: §2 · p. 3

Task simplification

• Task-decomposition methods split explicitly compositional tasks across inference passes but do not support the single-step language tasks targeted by AMA.This line of work includes Creswell et al. [2022] and Wu et al. [2022].Source: §2 · p. 3

Prompt sensitivity

• AMA addresses prompt sensitivity by aggregating multiple prompt outputs, unlike prior strategies focused on improving individual prompts [Zhao et al., 2021, Liu et al., 2021].Prior prompt aggregation often trains models, while Self-Consistency Wang et al. [2022b] reports no improvements for small LMs below 10B parameters.Source: §2 · p. 3

Weak supervision

• AMA is the first described method to use Weak Supervision rather than Majority Vote for unsupervised prompt aggregation.Weak supervision learns source accuracies and correlations to aggregate noisy sources [Ratner et al., 2017, 2016, 2018, Varma et al., 2019, Fu et al., 2020], whereas prior prompting work used aggregation for distillation into smaller models [Smith et al., 2022].Source: §2 · pp. 3–4

3.1 Preliminaries

• AMA combines a collection of prompts P with an aggregator φ to transform multiple predictions for each input into one output.This framework separates prompt construction from prediction aggregation.Source: §3 · p. 4, §3.1 · p. 4

3.2 Effective Prompt Formats

• Open-ended prompts improve average accuracy from 41.7% to 71.5% (+72%) on CB, RTE, and WSC compared with restrictive formats.The authors attribute this advantage to alignment with next-token prediction.Source: §3.2 · p. 4
• Precise question reformulation raises WSC accuracy to 74.7% with a 49.4% lift, while specialized classes require mapping open-ended answers to valid categories.This mapping yields 33.3% and 11.1% lifts over few-shot baselines on DBPedia and AGNews, respectively.Source: §3.2 · p. 5
• Open-ended QA structures occur roughly 1000× more often than restrictive structures in a 2% sample of the approximately 200B-token Pile corpus.The authors also observe imbalances between yes/no and True/False frequencies that may induce restrictive-prompt biases.Source: §3.2 · p. 5
• AMA generates questions from inputs and answers them, producing complementary views such as who, whether, and where questions for each example.The two-step pipeline supports multiple prompt outputs without manually writing every prompt.Source: §3.2 · p. 5

3.3 Creating Prompt Collections at Scale

• A prompt()-chain recursively applies question() to generate an open-ended question and answer() to produce an intermediate answer from the input context.AMA varies chains through in-context demonstrations and question styles including yes/no, Wh-, multiple-choice, and cloze formats.Source: §3.3 · pp. 5–6

3.4 Prompt Aggregation

• AMA’s weak-supervision aggregator learns prompt dependencies and accuracies without labeled data, then predicts the label maximizing the learned conditional probability.It recovers a dependency graph, estimates prompt accuracies, and computes Pr(y|P(x)) for aggregation.Source: §3.4 · pp. 6–7
• Prompt collections exhibit a 12.1% best-to-worst accuracy gap, 9.7% average class-conditional variance, and correlated errors with Jaccard indices of 42.2 and 39.9.These properties make equal-weight, independence-assuming majority vote unreliable across general prompt collections.Source: §3.4 · p. 6

• Conditional entropy H(y|ŷ) decomposes into uncertainty controlled by prompt quality and additional information loss controlled by the aggregation method φ.The first term is H(y|P(x)); the second is H(y|ŷ) − H(y|P(x)).Source: §4 · p. 7
• Relevant and numerous prompts lower H(y|ŷ), while an aggregator that better matches Pr(y|P(x)) reduces compression loss during prediction.Because H(y|P(x)) ≤ H(y|p(x)), prompt-set quality and quantity affect the information available before aggregation.Source: §4 · p. 7
• AMA outperforms k-shot baselines in prompt-chain quality and scales with the quantity of aggregated prompts, as measured by reduced conditional entropy.Figure 4 (Left) evaluates H(y|P(x)) and shows the expected scaling for AMA No Agg and prompt quantity.Source: §4 · p. 7
• Weak supervision reduces AMA’s aggregation information loss relative to majority vote, whose conditional-independence and uniform-accuracy assumptions can create a large gap from optimal H(y|P(x)).The gap is evaluated in Figure 4 (Right), where misspecification makes MV insufficient.Source: §4 · p. 8
• Increasing the number of aggregated prompts reduces conditional entropy, suggesting prompt aggregation as another axis for understanding LLM scaling performance alongside parameter scale, training data, and compute [Kaplan et al., 2020, Hoffmann et al., 2022, Wei et al., 2022c].Appendix D theoretically characterizes H(y|ŷAMA) from individual prompt accuracies under the standard weak supervision model [Ratner et al., 2018].Source: §4 · p. 8

5.1 Main Results

• On 15 of 20 benchmarks, GPT-J-6B with AMA exceeds few-shot GPT3-175B, while improving 41% over GPT-J-6B few-shot performance on average.The comparison uses GPT3-175B results with 32–70 examples and GPT-J-6B with three examples.Source: §5.1 · p. 8, §5 · p. 8
• AMA’s lift is largest for input-supported natural-language-understanding tasks, while memorization-, temporal-, and domain-knowledge tasks expose limitations.Generated context helps on closed-book WebQ, but RealTimeQA and Amazon Instant Video remain difficult; retrieval may help.Source: §5.1 · p. 8

5.2 Evaluation across Models

• AMA’s prompt chains generalize across four open-source model families and seven tasks, with lift increasing by model size before leveling out.T0 shows lower lift, possibly because zero-shot fine-tuning compromises in-context learning.Source: §5.2 · p. 9
• Across models and tasks, AMA yields a 10.2% ± 6.1% absolute lift and a 21.4% ± 11.2% relative lift.Average absolute lift is 11.0% for EleutherAI, 11.0% for BLOOM, 11.9% for OPT, and 2.9% for T0.Source: §5.2 · p. 9
• Models with stronger question generation, answer generation, answer selection, and extraction obtain greater AMA lift; T0 underperforms especially on generative operations.Diagnostics use manually labeled samples across tasks corresponding to prompt-chain steps.Source: §5.2 · p. 9

5.3 Evaluation against other aggregation methods

• Weak supervision achieves up to 8.7 points over majority vote and is no worse on 16 of 20 tasks, while losses on the other four are at most 1.0 point.Modeling dependencies improves performance on nine tasks by up to 9.6 points and 2.2 points on average.Source: §5.3 · pp. 9–10
• For T0 on PromptSource prompts, majority vote gives a 3.7-point average lift and weak supervision gives a 6.1-point average lift.The evaluation uses 10 prompts each for CB, WIC, WSC, and RTE; Table 2 reports 6.1 points for weak supervision.Source: §5.3 · p. 10, Table 2 · p. 10

6 Conclusion

• AMA combines scalable generation of multiple prompts with weak supervision over their intermediate answers to produce final predictions.The method is motivated by the stronger performance of open-ended questions and by modeling varying prompt accuracies and dependencies.Source: §6 · p. 10
• AMA provides performance lift across four language model families and model sizes ranging from 125M-175B parameters.Source: §6 · p. 10
• A 30x smaller language model exceeds the average performance of few-shot GPT3-175B across 20 popular language benchmarks.This result highlights AMA’s potential for applications involving private or large-scale data, where accessing closed-source APIs or hosting large models is challenging.Source: §6 · p. 10

8 Ethics Statement

• AMA is intended to aid practitioners exploring and using LLMs, particularly smaller, open-source models.Source: §8 · p. 10
• AMA could be used for harmful or unethical tasks, and the authors recognize inherent risks and error modes in this proof-of-concept.The paper directs readers to Bommasani et al. [2021] and Weidinger et al. [2021] for detailed risk discussions.Source: §8 · pp. 10–11

A Experiment Details

• All experiments were run using NVIDIA A100 GPUs.Source: A Experiment Details · p. 14

A.1 Models

• The evaluation covers T0, BLOOM, EleutherAI, OPT, and GPT-3 model families.The listed variants include models ranging from 125M to 176B parameters, plus T0-3B and GPT-3 davinci.Source: §A.1 · p. 14
• EleutherAI variants include GPT-Neo-125M, GPT-Neo-1.3B, GPT-J-6B, and GPT-NeoX-20B.Source: §A.1 · p. 14
• BLOOM variants span BLOOM-560M, BLOOM-1.7B, BLOOM-7.1B, and BLOOM-176B, while OPT variants span OPT-125M, OPT-1.3B, OPT-6.7B, OPT-13B, and OPT-175B.Source: §A.1 · p. 14
• T0-3B and OpenAI’s davinci were also evaluated, with davinci representing the original GPT-3 175B model used in Brown et al. [2020].T0, BLOOM, OPT, and EleutherAI models came from the HuggingFace Model Hub, while davinci was accessed through the OpenAI API for a per-sample fee.Source: §A.1 · pp. 14–15

A.2 Metrics

• RealTimeQA GPT-3 performance is re-evaluated on the davinci endpoint using the benchmark’s few-shot prompt, rather than text-davinci-002 [2022].This aligns the comparison with the endpoint used for the paper’s other GPT-3 evaluations.Source: §A.2 · p. 15
• The evaluation follows the metrics used by Brown et al. [2020] across all tasks.Source: §A.2 · p. 15
• Most tasks are scored with matching accuracy.Source: §A.2 · p. 15
• DROP and RealTimeQA use text f1, while WebQ and NQ use span overlap accuracy.Source: §A.2 · p. 15
• MultiRC is evaluated with f1a accuracy.Source: §A.2 · p. 15

A.3 Weak Supervision

• Weak supervision uses unlabeled test data and 1,000 unlabeled training samples per task, applying Varma et al. [2019]’s default structure-learning parameters.If the recovered sparse matrix is entirely greater than 1, the method uses no edges; otherwise, it passes only the highest-valued edge to parameter learning.Source: §A.3 · p. 15

Additional Results

• Table 3 compares AMA on BLOOM-7.1B with few-shot GPT3-175B.The supplied passage identifies the comparison but does not provide the table’s numerical results.Source: Table 3 · p. 15
• GPT3-175B results are reported from Brown et al. [2020], with the number of shots shown in parentheses.Source: Table 3 · p. 15
• Classification-task baselines are taken from Zhao et al. [2021].Source: Table 3 · p. 15

B.1 BLOOM Model Results

• AMA consistently lifts BLOOM-7.1B performance over few-shot results across all 20 benchmarks, although it remains below the comparably sized GPT-J-6B model.The few-shot BLOOM-7.1B results are also often lower than the few-shot GPT-J-6B results.Source: §B.1 · pp. 15–16
• The comparison applies Majority Vote and Weak Supervision to three random few-shot prompts without prompt reformatting to the AMA QA template.Source: Table 4 · p. 16

B.2 AMA Ablations

• Using three few-shot prompts with k = 3 in-context examples, the study compares majority-vote and weak-supervision aggregation without applying AMA QA re-formatting.The prompts use different sets of in-context examples, and aggregation is evaluated independently of the proposed re-formatting.Source: §B.2 · p. 16
• Aggregation alone improves over average performance but still leaves large performance gaps.Source: §B.2 · p. 16
• Both prompt re-formatting and aggregation are critical and complementary for an effective prompting solution.Source: §B.2 · p. 16

Additional data

• Removing all additional unlabeled data preserves average accuracy on WSC and AGNews and reduces RTE accuracy by only 0.4 points, while still outperforming few-shot GPT3-175B.This result averages five runs across RTE, WSC, and AGNews with evaluation on a fixed test set.Source: §B.3 · p. 16, Figure 6 · p. 17

Latency

• Weak supervision takes 13.0 seconds without modeled dependencies and 84.3 seconds with dependencies modeled for RTE, adding comparatively low latency.Dependencies are ignored for WSC and AGNews because their recovered structured matrices are dense; GPT-J-6B inference times are provided as a comparison in Table 6.Source: §B.3 · p. 16, Table 6 · p. 17

Aggregation baselines

• Table 5 compares AMA aggregation with majority vote, weighted majority vote, Pick Best, and AMA without dependency edges for GPT-J-6B.Weighted majority vote uses 1,000 labeled examples and tunes its temperature on a 20% validation split; Table 5 also reports prompt-chain counts by task.Source: §B.3 · p. 16, Table 5 · p. 17

B.4 Additional AMA Baselines

• The Self-Consistency baseline aggregates multiple prompt outputs through temperature-based sampling over temperatures ∈{0.0, 0.3, 0.5, 0.6, 0.7}.Self-Consistency builds on Chain-of-Thought prompting, which generates reasoning paths alongside final predictions.Source: §B.4 · p. 17
• AMA outperforms Self-Consistency at the GPT-J-6B scale using the same number of prompts, consistent with limited gains reported for small LMs (<10B) [2022b, 2022a].The comparison uses overlapping benchmark tasks and the exact prompts from Wang et al. [2022b].Source: §B.4 · p. 17, Table 7 · p. 18

C Weak Supervision Algorithm

• Weak supervision learns a latent graphical model over labels and prompt votes, then predicts φ(x) = arg max_y Pr(y|P(x)).The model is parameterized to capture both variations in prompt accuracy and dependencies among prompts.Source: C Weak Supervision Algorithm · p. 18
• The dependency graph connects each prompt vote to y and connects prompt pairs that are not conditionally independent given y and the other prompt chains.Formally, the graph is G = (V, E), with V = {y, P(x)}.Source: C Weak Supervision Algorithm · p. 18
• The algorithm learns the prompt dependency structure by exploiting graph structure in the inverse covariance matrix and estimating sparse and low-rank components with RobustPCA.The recovered sparse component identifies conditional dependencies among prompts.Source: C Weak Supervision Algorithm · p. 18
• After recovering the graph, the algorithm estimates latent prompt accuracies from sparse inverse-covariance equations using the approach of Ratner et al. [2018].It then computes Pr_Ĝ,θ̂(y|P(x)) and aggregates the prompt predictions.Source: C Weak Supervision Algorithm · p. 18

D Information-Flow Theoretical Result

• AMA’s residual error reflects information lost when P(x) is converted into an independent label sample, and this loss increases with uncertainty in Pr(y|P(x)).The analysis assumes weak supervision exactly recovers Pr(y,P(x)) before sampling the predicted label.Source: D Information-Flow Theoretical Result · p. 18
• The theoretical model uses binary labels, an Ising dependency graph, canonical correlation parameters, and accuracy-scaled means to characterize prompt information.The parameters θ_i represent prompt–label correlation, while θ_ij represent dependencies between prompts.Source: D Information-Flow Theoretical Result · pp. 18–19
• Theorem 1 expresses H(y|P(x)) under the Ising model in terms of prompt–label correlations and a normalization term independent of prompt interactions.The result assumes binary labels and models prompt dependencies through an Ising graphical model.Source: D Information-Flow Theoretical Result · pp. 18–19

E AMA Diagnostics

• AMA diagnostics comprise eight tasks spanning question generation, answer generation, answer selection, and extraction.The diagnostic suite evaluates transformations into yes/no, wh-, and placeholder questions, candidate-constrained answers, sentence completions, and extracted spans.Source: E AMA Diagnostics · pp. 20–22
• Question-generation tasks use ROUGE, answer-selection tasks require outputs to match provided candidates, and extraction tasks use Text-F1 from SQuAD [Rajpurkar et al., 2018].The answer-generation task marks a response correct when it generates two candidate answers.Source: E AMA Diagnostics · pp. 20–22

F Understanding The Effectiveness of the Question-Answering Template

• QA patterns occur more frequently than restrictive prompts in a 2% sample of the roughly 200B-token Pile corpus, including yes/no questions followed by yes or no.This prevalence may explain why language models perform better with QA prompts.Source: F Understanding The Effectiveness of… · p. 23, Table 8 · p. 23, Table 9 · p. 23
• Restrictive prompting shows class-level F1 imbalances, motivating hypotheses that unequal label-word frequencies bias outputs and that frequent question words yield more context-specific representations.The paper concludes that designing templates from pretraining-corpus analysis may be promising future work.Source: F Understanding The Effectiveness of… · pp. 23–24, Table 10 · p. 24

Knowledge errors

• AMA gains most when task knowledge is explicitly provided in context, but provides less lift when models must recall factual or apply commonsense knowledge.Examples include GPT-J-6B lacking latent factual knowledge on Natural Questions and failing to recognize that smoked food is cooked on BoolQ.Source: G Error Analysis · pp. 24–25, Figure 7 · p. 25

Instruction-following errors

• On restrictive output tasks such as multi-way classification, AMA sometimes generates answers outside the permitted output space despite explicit instructions.GPT-J-6B fails to map descriptions such as “automobile” or “singer” to valid DBPedia classes.Source: G Error Analysis · p. 25

Long-context errors

• AMA question() struggles with accurate statement-to-question transformations for long or complex inputs, while answer() struggles to extract spans from contexts longer than 6 sentences.The paper illustrates these failures on SuperGLUE record and DROP QA examples using GPT-J-6B.Source: G Error Analysis · p. 27

H Datasets and Prompts

• The evaluation covers 20 datasets across SuperGLUE, NLI, classification, and question-answering categories.The listed benchmarks include eight SuperGLUE datasets, four NLI datasets, four classification datasets, and four question-answering datasets.Source: H Datasets and Prompts · p. 28

H.1 AGNews

• AGNews is a four-topic news classification dataset with 120,000 training examples and 76,000 test examples [Zhang et al. 2015].Its categories are World News, Sports, Business, and Technology and Science.Source: §H.1 · p. 29
• AGNews AMA prompts transform passages into short summaries and ask which category the summary fits.Examples map foreign direct investment to Business, the Arizona Diamondbacks to Sports, and a plant fire to World News.Source: §H.1 · pp. 29–30

H.2 ANLI R1

• ANLI R1 is an adversarially mined Wikipedia NLI dataset with 16,946 training examples and 1,000 test examples [Nie et al. 2020].Its examples use True, False, or Neither judgments, including statements supported, contradicted, or unresolved by the passage.Source: §H.2 · p. 31
• ANLI R1 AMA chains rewrite statements as yes/no questions, then answer them from context with Unknown when evidence is absent.The examples include transforming declarative statements into questions and returning yes, unknown, or related answers from context.Source: §H.2 · pp. 31–32

H.3 ANLI R2 / H.4 ANLI R3

• ANLI R2 and R3 use the same adversarial NLI format with 45,460/1,000 and 100,459/1,200 train/test examples, respectively [Nie et al. 2020].Their examples retain True, False, or Neither labels and apply AMA chains that convert statements into questions before contextual answering.Source: §H.3 · pp. 32–33, §H.4 · pp. 34–35

H.5 Amazon

• Amazon is a nine-class product classification dataset with 9,000 training examples and 9,000 test examples [He and McAuley 2016].The passage documents its dataset scope and split sizes but does not provide an accompanying prompt example here.Source: §H.5 · p. 36

H.6 BoolQ

• BoolQ is a yes/no question-answering task over small Wikipedia passages, with 9,427 training examples and 3,245 test examples.Its AMA examples use context-question-answer formats and include prompts such as “Answer the question using the context.”Source: §H.6 · pp. 36–37

AMA prompt chains

• Across these datasets, the displayed AMA prompt chains convert constrained classification or inference tasks into natural-language question-answering interactions with task-specific demonstrations.The examples include direct context questions, statement-to-question rewrites, causal completions, summary-to-category matching, and evidence-sensitive answers.Source: §H.6 · p. 36, §H.7 · p. 38, §H.8 · p. 39

H.7 CB

• CB is a three-class textual entailment task with 250 training examples and 56 test examples, prompted by rewriting statements as yes/no questions.The examples use True, False, or Neither labels and include demonstrations that convert declarative statements into questions.Source: §H.7 · pp. 37–39

H.8 COPA

• COPA asks models to select the alternative with the more plausible causal relation, using continuation and correct-ending prompt formats.Examples include choosing consequences such as “The motorcyclist died” after a crash and “she took medicine” as the cause of feeling better.Source: §H.8 · pp. 39–40

H.9 DBPedia

• DBPedia is a 14-class ontology classification dataset with 560,000 training examples and 70,000 test examples.AMA prompts ask for the correct category or summarize a passage before matching it to categories such as company, written work, album, or natural place.Source: §H.9 · pp. 40, 41, 43

H.10 DROP

• DROP is a reading-comprehension benchmark requiring discrete reasoning over paragraphs, with 77,409 training examples and 9,536 test examples.Its AMA prompt instructs the model to answer questions and return “Unknown” when the context lacks evidence.Source: §H.10 · pp. 43–45
• DROP examples pair passages with open-ended questions and answers, including an unknown response when the context does not support the requested fact.Examples ask where the plague was present, identify a game’s final touchdown scorer, and mark unsupported questions as Unknown.Source: §H.10 · p. 45

Answer formats

• The examples also show that answer formats vary by dataset, including free-form entities, binary judgments, qualified answers, and Unknown when evidence is insufficient.These conventions appear in DROP, MultiRC, Natural Questions, RTE, and ReCoRD examples.Source: §H.10 · p. 45, §H.11 · p. 46, §H.12 · p. 49

H.11 MultiRC

• MultiRC, a multi-sentence reading-comprehension dataset with 27,243 training and 953 test examples, asks whether candidate answers are correct [2019].Its AMA prompt-chain examples retain the instruction to judge whether a proposed answer correctly addresses the passage question.Source: §H.11 · pp. 45–46

Prompt-chain formats

• Across the illustrated tasks, AMA prompt chains preserve the source context while converting task-specific inputs into question-answer formats or answer-verification instructions.MultiRC uses correctness judgments, Natural Questions uses distinct-question generation, and RTE explicitly rewrites statements as questions.Source: §H.11 · p. 46, §H.12 · p. 49, §H.13 · p. 51

H.12 Natural Questions (NQ)

• Natural Questions, an open-domain user-question dataset with 307,373 training and 7,830 test examples, demonstrates few-shot and AMA prompt-chain formats [2019].The AMA examples transform or supplement questions with distinct questions and contextual answers, including factual entity, geography, and biography responses.Source: §H.12 · pp. 49–50

H.13 RTE

• RTE evaluates whether a premise entails a hypothesis using true-or-false examples, then rewrites statements as questions and answers unsupported cases as Unknown.The examples include direct entailments, negation-sensitive rewrites, and contextual answers such as Unknown, No, or Yes with supporting qualifiers.Source: §H.13 · pp. 50–51

H.14 ReCoRD

• ReCoRD is a commonsense reading-comprehension dataset with 100,730 training and 10,000 test examples [2019], illustrated through context-based entity answers.Examples include identifying BART’s response to a threatened website attack and completing an SNL hosting sequence with candidate entities.Source: §H.14 · pp. 51–53
• ReCoRD examples provide contextual passages about entities such as Obama, Saif al-Islam Gadhafi, the Beatles, and Tracy Morgan for question answering.The supplied ReCoRD passages include entity-rich contexts and candidate answers, but no dataset size or explicit prompt transformation is provided.Source: §H.14 · pp. 53–54

H.15 RealTime QA

• RealTime QA is a dynamic question-answering dataset about current-world facts, with 90 training examples and 187 test examples.Examples ask about facts such as Japan’s capital, a common charging standard, Walmart, JetBlue, and Yosemite National Park.Source: §H.15 · pp. 54–56
• RealTime QA prompts ask models to answer questions using multiple supplied articles, including questions whose answers require identifying information across current-news contexts.The examples present five articles followed by a question and a direct answer, such as Walmart, JetBlue, or Yosemite National Park.Source: §H.15 · pp. 55–56

H.16 SST2

• SST2 is a movie-review binary sentiment classification dataset with 6,920 training examples and 1,821 test examples.The task labels each text snippet as positive or negative, with demonstrations covering both sentiments.Source: §H.16 · p. 57

H.17 Story Cloze

• Story Cloze requires choosing the best ending from two candidate sentences for a four-sentence story, using 1,871 training and 1,871 test examples.Examples select endings consistent with story context, such as choosing recovery after illness or staying home after an embarrassing incident.Source: §H.17 · pp. 57–58
• Story Cloze AMA transformations rewrite statements as yes/no questions, then answer the resulting questions from passages.Examples convert statements about work, rest, bicycles, and staying home into questions such as “Did I stay home the next weekend?”Source: §H.17 · pp. 58–59

H.18 WSC

• WSC requires reading a sentence with a pronoun and selecting its referent from listed choices, with 554 training examples and 104 test examples.Examples ask whether “his” refers to Mark or Shakespeare, whether “he” refers to Ralph, and whether “it” refers to the table.Source: §H.18 · p. 59
• WSC AMA prompt transformations extract the phrase containing a pronoun and rewrite the input as a question about its referent.Examples include extracting phrases such as “her” first day and generating questions such as “Who drowned?” or “Whose cat is black?”Source: §H.18 · p. 60
• WSC examples recast pronoun-resolution statements as open-ended questions whose answers identify the relevant referent, such as Jane, Mrs. Jenna, Mark, or the styrofoam object.The examples include “Whose first day?”, “Who loved him?”, “Who could buy lunch?”, and “What was made of styrofoam?”Source: §H.18 · p. 61

H.19 WebQuestions (WQ)

• WebQuestions is a question-answering dataset with questions answerable using Freebase, comprising 3,778 training examples and 2,032 test examples (Berant et al. [2013]).Source: §H.19 · p. 61
• The WebQuestions AMA prompt()-chain asks the model to “Produce distinct questions,” then supplies varied question-answer examples and contextual question-answer pairs.Examples include questions about Sarah Jessica Parker, air composition, HP, FRIENDS, airports, degrees, rivers, currencies, and actors.Source: §H.19 · pp. 61–62
• Additional WebQuestions demonstrations show that contextual passages can directly support answers, including John Kasich as governor of Ohio and William Daniels as KITT’s actor.Other contexts identify Palm Springs International Airport, a Bachelor of Divinity degree, Libya, and the United States dollar.Source: §H.19 · p. 62

H.20 WiC

• WiC is a binary word-sense-disambiguation task over sentence pairs, with 5,428 training examples and 638 test examples, attributed to Wang et al. [2019].Its examples ask whether a shared word has the same sense across two sentences, with answers such as No or Yes.Source: §H.20 · pp. 62–63
• The WiC synonym examples extend to ambiguous words such as “class,” whose sentence context is used to elicit a meaning-specific synonym.The example begins with “An emerging professional class.” and requests synonyms for that occurrence.Source: §H.20 · p. 63
• The WiC AMA prompt()-chain transforms sense comparison into synonym generation by instructing the model to “Give synonyms of the word in the sentence.”Demonstrations provide synonyms such as “noise” for sound, “password” for code, and “show” for play.Source: §H.20 · p. 63