PaperlayerView exact revision on arXiv

Source-linked AI summary

Crosslingual Generalization through Multitask Finetuning

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, Colin Raffel

arXiv:2211.01786v2cs.CLcs.AIcs.LG

TL;DR

The paper addresses limited evidence on whether multitask finetuning transfers across languages in multilingual models. It trains BLOOM and mT5 variants on English, multilingual, and machine-translated prompt mixtures, finding broader zero-shot generalization across tasks and languages. The released xP3 corpus covers supervised datasets in 46 languages.

  • Problem

    Prior multitask finetuning research focused on English data and models, leaving crosslingual zero-shot generalization insufficiently explored.

    Source: §1 · p. 1

  • Method

    The paper finetunes BLOOM and mT5 on English P3, multilingual xP3, and machine-translated-prompt xP3mt mixtures.

    Source: §1 · pp. 1–2, §5 · p. 8

  • Results

    English-only finetuning transfers to non-English tasks, xP3 improves English and non-English zero-shot performance, and xP3mt improves human-written non-English prompts.

    Source: §1 · p. 2, §5 · pp. 8–9

  • Takeaways & Limitations

    Multilingual models can acquire task- and language-agnostic capabilities that support zero-shot generalization beyond intentionally trained languages.

    Source: §1 · p. 2, §5 · p. 9

    xP3 covers 46 languages although mT5 was pretrained on more than 101, and the authors expect broader language coverage could improve performance.

    Source: §6 · p. 9

Abstract

from arXiv · show

Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages that appear only in the pretraining corpus. Finetuning on multilingual tasks with English prompts further improves performance on English and non-English tasks leading to various state-of-the-art zero-shot results. We also investigate finetuning on multilingual tasks with prompts that have been machine-translated from English to match the language of each dataset. We find training on these machine-translated prompts leads to better performance on human-written prompts in the respective languages. Surprisingly, we find models are capable of zero-shot generalization to tasks in languages they have never intentionally seen. We conjecture that the models are learning higher-level capabilities that are both task- and language-agnostic. In addition, we introduce xP3, a composite of supervised datasets in 46 languages with English and machine-translated prompts. Our code, datasets and models are freely available at https://github.com/bigscience-workshop/xmtf.

1 Introduction

The paper asks whether multitask finetuning can improve multilingual models’ zero-shot transfer across both tasks and languages. It introduces progressively multilingual training mixtures and finds gains from English-only, multilingual, and machine-translated prompting.

1 Introduction

  • English-only multitask finetuning improves multilingual models’ performance on non-English held-out tasks.This tests transfer from English supervised data to languages represented only during pretraining.Source: §1 · p. 1
  • xP3 adds datasets from 46 languages and expands task coverage beyond P3, including translation and program synthesis.The multilingual mixture is compared against the English-only P3 baseline.Source: §1 · pp. 1–2
  • Multilingual finetuning with English prompts further improves zero-shot generalization on both English and non-English tasks.Models trained on xP3 perform best on English prompts, including for non-English samples.Source: §1 · p. 2
  • Machine-translated prompts improve performance on held-out tasks evaluated with human-written non-English prompts.The xP3mt variant translates prompts to match each dataset language, while reducing English prompts worsens English performance.Source: §1 · p. 2
  • xP3-trained models generalize to held-out tasks in languages never intentionally seen during pretraining or finetuning.A contamination analysis found only small amounts of these languages in pretraining, motivating the hypothesis of language- and task-agnostic capabilities.Source: §1 · p. 2

2 Related work

Prior work established prompted multitask finetuning for zero-shot task transfer, while multilingual modeling extended zero-shot capabilities across languages. This paper focuses specifically on crosslingual zero-shot generalization and evaluates broader prompting settings.

2.1 Multitask learning

  • Multitask finetuning uses prompted datasets to teach models task skills that transfer to held-out tasks.Zero-shot task generalization means performing a held-out task from prompted instructions alone.Source: §2.1 · pp. 2–3
  • Prior multitask finetuning largely used English prompts and English datasets, whereas this work examines crosslingual zero-shot generalization.The evaluation considers a wider variety of prompting settings and multilingual capabilities.Source: §2.1 · p. 3

2.2 Multilingual models

  • Multilingual pretrained models support monolingual and crosslingual tasks, but their zero-shot performance is generally below finetuned performance.This motivates improving zero-shot transfer for low-resource languages without requiring task-specific finetuning.Source: §2.2 · p. 3
  • BLOOM is pretrained on ROOTS in 46 natural and 13 programming languages, while mT5 is pretrained on a corpus covering 101 languages.The models provide decoder-only and encoder-decoder multilingual architectures for comparison.Source: §2.2 · pp. 3–4

3 Finetuning data and models

The paper constructs xP3 by extending P3 with multilingual tasks and trains BLOOM and mT5 variants under English and machine-translated prompting conditions. It evaluates zero-shot transfer on held-out tasks using standardized prompt and scoring procedures.

3 Finetuning data and models

  • The study trains BLOOM and mT5 models and releases P3, xP3, and xP3mt variants with English or machine-translated prompts.The variants are BLOOMZ-P3/mT0-P3, BLOOMZ/mT0, and BLOOMZ-MT/mT0-MT.Source: §3 · p. 4, §3.2 · p. 4

3.1 Finetuning data

  • xP3 extends P3 with 30 multilingual datasets spanning translation, simplification, program synthesis, and miscellaneous code tasks.Some prompts invert the original task, such as generating a question from an answer.Source: §3.1 · p. 4
  • xP3 matches ROOTS’s 46-language coverage, while xP3mt machine-translates monolingual prompts into each dataset’s language.Crosslingual datasets retain English prompts in xP3mt.Source: §3.1 · p. 4

3.2 Models

  • Models range from 560 million to 176 billion parameters and are finetuned for 13 billion tokens with loss computed only on target tokens.Target-length normalization gives short and long targets equal task weight; samples longer than 2048 tokens are skipped.Source: §3.2 · p. 4
  • Evaluation covers held-out coreference, sentence completion, NLI, and HumanEval tasks using language-averaged scores, rank classification, and BLEU or pass@k.Five randomly selected PromptSource prompts are used per evaluation dataset, and their median result is reported per language split.Source: §3.2 · p. 5, Figure 4 · p. 5

4 Results

Multitask finetuning improves multilingual zero-shot task and language generalization, with performance shaped by prompt language, model scale, generation-task characteristics, and language-resource proportions.

4.1 Task generalization

  • Finetuned BLOOMZ and BLOOMZ-P3 significantly improve over BLOOM and XGLM on held-out tasks, while 13B mT0 outperforms 176B BLOOMZ.The authors attribute mT0’s advantage to its encoder-decoder architecture, masked-language-modeling pretraining, and longer pretraining.Source: §4.1 · p. 5

4.2 Language generalization

  • Multitask finetuning on English data improves non-English task performance, including over 50% improvement for BLOOMZ-P3 on multilingual sentence completion versus BLOOM.This supports language transfer to languages seen during pretraining without task-specific finetuning.Source: §4.2 · p. 6
  • After xP3 finetuning, models generalize to unseen tasks in languages not intentionally trained on, including languages appearing only in tiny pretraining fractions such as Thai at 0.006%.The authors note that XNLI performance increases across all languages and examine whether superficial label cues explain this generalization.Source: §4.2 · p. 6

4.3 Multilingual prompting

  • BLOOMZ-MT raises XNLI accuracy on human-translated prompts from 41.13 to 45.55, while English-prompt performance falls from 53.58 to 49.74.Machine-translated prompt finetuning improves non-English prompting, although gains vary across tasks and human-translated prompts outperform machine-translated ones on XNLI.Source: Table 1 · p. 6, §4.3 · p. 7

4.4 Scaling

  • Multitask finetuning improves zero-shot generalization even at 560M parameters, and the pretrained-to-finetuned gap grows significantly with model size across evaluated languages.The underlying pretrained BLOOM performance is near random baselines for several tasks under zero-shot untuned prompting.Source: §4.4 · p. 7, Figure 6 · p. 7

4.5 Generation tasks

  • Natural-language-understanding performance continues increasing during training, whereas generative performance rises initially and then declines; BLOOMZ solves HumanEval problems with 70% fewer characters than BLOOM.The authors link this short-generation bias to finetuning mostly on single-sentence tasks.Source: §4.5 · pp. 7–8, Figure 7 · p. 7
  • Forcing a minimum generation length improves translation BLEU by 9 points, while mT0 models fail on HumanEval because they were not pretrained on code.Adding or upweighting long tasks helps somewhat, but minimum-length decoding is reported as most effective.Source: §4.5 · pp. 7–8

4.6 Effect of language proportions

  • Finetuned BLOOM models perform best on high-resource languages such as English, Spanish, and French, while XWinograd shows a less consistent language-resource trend.The xP3 language distribution closely follows pretraining, and differing XWinograd subset sizes and difficulty may explain its irregular pattern.Source: §4.6 · p. 8, Figure 8 · p. 8

5 Conclusion

The work shows that multilingual multitask finetuning improves zero-shot generalization across languages and tasks, while machine-translated prompts improve performance on human-written non-English prompts.

5 Conclusion

  • The study releases xP3, xP3mt, BLOOMZ, and mT0 artifacts to support future research on zero-shot generalization.Source: §5 · pp. 8–9
  • English-only finetuning enables multilingual pretrained models to generalize to tasks in other languages represented during pretraining.Source: §5 · p. 8
  • Multilingual xP3 finetuning improves performance beyond English-only P3, and machine-translated prompts further improve human-written non-English prompt performance.The resulting models include BLOOMZ and mT0, finetuned from BLOOM and mT5 on English-only, multilingual, and machine-translated-prompt corpora.Source: §5 · pp. 8–9
  • Finetuned models also generalize to new tasks in languages they have never intentionally seen.Source: §5 · p. 9

6 Limitations

The authors identify limitations in prompt formatting, language coverage, model quality, zero-shot failure modes, and evaluation of languages seen only during finetuning.

6 Limitations

  • Space-separated prompts can cause decoder-only models to continue the input before solving the task, whereas trailing punctuation or newlines can mitigate this behavior.Source: §6 · p. 9
  • The study evaluates only 46 languages, although mT0 was pretrained on more than 101, and its larger xP3x extension has not yet been used for finetuning.Source: §6 · p. 9
  • Zero-shot models retain numerous failure modes, including misunderstanding a fable’s moral and answering a request for explanation with a question.Source: §6 · p. 9
  • The BLOOM and mT5 base models and the multitask finetuning recipe remain suboptimal, leaving other procedure and base-model improvements unexplored.Source: §6 · p. 9

Learning new languages during finetuning

A Contributions

  • The project involved contributions spanning evaluation, dataset creation, training, code, prompting, checkpoint conversion, advising, and paper writing.Source: A Contributions · p. 17

B Task generalization breakdown

English held-out task performance improves with xP3 and with multilingual mT0, while multilingual task performance varies substantially by prompt and language.

B Task generalization breakdown

C Artifacts

  • The artifacts table lists the models and datasets used or released, with BLOOM(Z) under the RAIL license and mT5/mT0 under Apache 2.0.Source: C Artifacts · p. 19, Table 3 · p. 19

D ROOTS language contamination

A contamination analysis finds that ROOTS contains small amounts of unintentionally collected languages, which may contribute to observed language generalization.

D ROOTS language contamination

E Code generations

BLOOMZ generates shorter, more concise code than BLOOM despite both producing correct HumanEval solutions. Qualitative examples also show multilingual and unseen-task behavior, with language control and generation-length effects.

E Code generations

F Qualitative examples

  • BLOOMZ and mT0-13B handle non-English sentiment prompts without training on non-English prompts, although BLOOMZ answers in English.The example concerns a five-star review of Star Wars Episode IV.Source: Figure 13 · p. 21
  • On zero-shot query expansion, models sometimes fail to produce the requested minimum of five terms.The task was not included in training.Source: Figure 14 · p. 21
  • Specifying the language in a question-answering prompt effectively forces the output language, while minimum token length forces longer generations.The BLOOM output is shortened relative to the comparison generation.Source: Figure 15 · p. 22
  • BLOOMZ can generate an interesting fable with a requested moral, whereas mT0 is significantly worse at storytelling and BLOOMZ sometimes misinterprets the moral.The fable generations are cherry-picked from 16 outputs under non-greedy decoding.Source: Figure 16 · p. 22

G Increasing generation length

Generation quality deteriorates when finetuning produces very short outputs, so the authors test data-mixture changes and minimum-length decoding. Enforcing a minimum generation length provides the strongest reported improvement.

G Increasing generation length

  • A 13-billion-token finetuning run yields BLEU 0 and very short generations, motivating reduced short-task exposure and minimum-length decoding.The study uses a 7.1B-parameter BLOOM model and evaluates English-French and French-English translation.Source: G Increasing generation length · p. 23, Table 5 · p. 23
  • Early stopping, upweighting long tasks, and adding long tasks all lengthen outputs and slightly improve BLEU, but their effects remain small.Long tasks comprise 10% of finetuning data in the added-task ablation.Source: G Increasing generation length · p. 23
  • 768-token minimum generation raises MultiEURLEX BLEU to 9.05 for outputs targeting about 491 tokens.The final models combine early stopping and added long tasks, while minimum-length decoding is recommended for long generations.Source: G Increasing generation length · p. 23, Table 5 · p. 23
  • Upweighting longer tasks is excluded from the final recipe because it worsens NLU validation accuracy by 10%.Final checkpoints are early-stopped based on validation performance, with matched-step checkpoints for -MT and -P3 variants.Source: G Increasing generation length · p. 23, Table 6 · p. 23

H XNLI edit distances

Edit-distance patterns across XNLI labels are consistent with the proposed entailment, contradiction, and neutrality structure but are too small to explain language generalization alone. In unseen languages, translating prompts can substantially reduce BLOOMZ performance.

H XNLI edit distances

  • Neutral XNLI pairs have the largest and entailment pairs the smallest average edit distances across Thai, Turkish, and Greek.Each label has 830 samples per language subset.Source: H XNLI edit distances · p. 24, Table 8 · p. 24
  • Thai contradiction samples average only 2.5 more edits than entailment samples, so edit distance alone likely cannot explain language generalization.The aggregate differences across labels are generally only a few edits.Source: H XNLI edit distances · p. 24

I Multilingual prompting in unseen languages

  • BLOOMZ performance drops significantly when prompts are translated into languages unseen during both pretraining and finetuning.For these languages, BLOOMZ-MT performs worse than BLOOMZ, likely because it was not finetuned on prompts in those languages.Source: I Multilingual prompting in unseen… · p. 24, Table 9 · p. 25
  • mT0 shows less significant differences between English and machine-translated prompts in languages seen during pretraining but not finetuning.Table 9 compares English and machine-translated prompts for 176B BLOOMZ and 13B mT0.Source: I Multilingual prompting in unseen… · p. 24, Table 9 · p. 25

J Ideas that did not work

Several finetuning design changes failed to improve baseline performance. The clearest degradation came from special tokens and bias-only finetuning.

J Ideas that did not work

  • Non-causal finetuning does not improve over causal finetuning in these experiments.This differs from prior work reporting benefits for non-causal multitask finetuning.Source: J Ideas that did not work · p. 25
  • Replacing space separation with an end-of-sequence or newly learned special token significantly worsens results.The authors suggest these tokens may require training during pretraining or on more tokens.Source: J Ideas that did not work · p. 25
  • Leaving PromptSource prompts unchanged or substantially rewriting them does not significantly improve over separating inputs and targets with a space.The separator choice reflects adapting prompts written for encoder-decoder models to decoder models.Source: J Ideas that did not work · p. 25
  • Bias-only finetuning makes BLOOMZ-7.1B perform 15 absolute percentage points worse on average held-out tasks.This contrasts with prior reports that bias-only finetuning can support strong downstream performance.Source: J Ideas that did not work · p. 25

K Full results

This section provides full test and validation evaluation results, documents version updates, and describes the prompts and datasets covered in the appendices.

K Full results

L Version control

  • The version-control notes add validation evaluations, generation-length analysis, an xP3x mention, and an XNLI example.These additions are assigned to Appendix §§K, G, C, and H, respectively.Source: L Version control · p. 28

M Prompts used

  • The prompts appendix describes training and evaluation prompts, with dataset examples, task-formulation notes, references, input and target templates, and answer choices where applicable.It includes both human-translated and machine-translated prompt samples for xnli es.Source: M Prompts used · pp. 28–29

1 Prompts

  • The prompt inventory spans simplification, translation, program synthesis, coreference resolution, multiple-choice and extractive question answering, classification, coding, word-sense disambiguation, paraphrase identification, sentence completion, and natural language inference.The listed datasets include multilingual examples such as GEM/BiSECT, MLQA, XQuAD, XNLI, XCOPA, and PAWS-X.Source: §1 · pp. 29–30

1.1.1 GEM/BiSECT en

  • The appendix identifies GEM/BiSECT English as a training dataset from Kim et al. (2021).The dataset entry states that it was used in training.Source: §1.1.1 · p. 30

Data Example

This section presents multilingual prompt templates and representative data examples spanning summarization, translation, code generation, programming, coreference, question answering, classification, word-sense disambiguation, paraphrase identification, sentence completion, and natural-language inference. It includes examples and formulations across numerous languages, including English, Spanish, French, Bengali, Vietnamese, and Chinese, as well as programming languages, with some prompts differing from the datasets’ original intended tasks.

1.1.2 GEM/BiSECT es Data Example

  • GEM/BiSECT Spanish examples pair Spanish source sentences with simplified Spanish targets, while prompts request sentence splitting and simplification with equivalent meaning.The examples include a Spanish source about the end of Mass and its simplified target.Source: Prompts · pp. 31–32, §1.1.2 · p. 32

1.2.1 GEM/wiki_lingua en; 1.2.2 GEM/wiki_lingua es Data Example

  • GEM/wiki_lingua examples use English and Spanish articles or text pairs with prompts for summaries, abstracts, rephrasing, and short article generation.The English example is from lad (2020), and the Spanish example contains a web-search source and quiz-style target.Source: Prompts · p. 34, §1.2.2 · p. 36

1.2.3 GEM/xlsum bengali; 1.2.4 GEM/xlsum english Data Example

  • GEM/xlsum examples show Bengali and English news summarization, using prompts that provide article text or title-summary inputs and request outputs in the corresponding language.The Bengali dataset is from Hasan et al. (2021), while the English example concerns a tidal-turbine article.Source: §1.2.3 · p. 37, Data Example · p. 37, Prompts · p. 38

1.3.1 Helsinki-NLP/tatoeba_mt ben-eng

  • Translation examples cover Bengali-English and English-French pairs, with prompts explicitly directing translation from one named language to the other.The Bengali-English example comes from Tatoeba, while the French example translates “Aah. Now I understand.” as “Ah ! Maintenant, je comprends.”Source: §1.3.1 · p. 41, Data Example · pp. 41–42, Prompts · pp. 41–42

1.4.1 Muennighoff/mbpp sanitized

  • The programming examples include Python problem-solving prompts and Python solutions, including code requested in a Jupyter notebook style.The cited examples note that the prompts do not correspond to the original tasks intended by the dataset authors.Source: Prompts · pp. 45, 47, 53

Prompts

  • Prompt templates cover code-generation tasks in multiple languages, including C-Sharp, Java, Javascript, PHP, and Python.Examples ask models to solve a provided text or description in the target language, while some templates request descriptions of code or provide partial solutions.Source: Prompts · pp. 49, 50, 51, 52, 54

Data Example

  • The data examples pair programming problems with corresponding code solutions, including Java, Javascript, PHP, and Python instances.Examples include Java code for a programming problem, Javascript code for a function, PHP code for an array-rearrangement problem, and Python code for summing two cubes.Source: Data Example · pp. 50–52

1.5.1 Muennighoff/xwinograd en

  • The xwinograd example is a sentence-level two-option coreference task with English and French prompt variants.Prompts ask which option fills a blank or what the blank refers to, and the section notes that these prompts do not correspond to the original task intended by the dataset authors.Source: Prompts · pp. 53, 55, Data Example · pp. 55, 57

1.6 Question Answering Multiple Choice

  • The multiple-choice QA example contains a Chinese dialogue, question, answer choices, and answer, with prompts that ask for the correct choice or a question eliciting the answer.The prompt implementations include Chinese and English variants and state that some prompts do not correspond to the original task intended by the dataset authors.Source: Prompts · pp. 53, 59, 60, Data Example · p. 59

1.7 Question Answering Extractive

  • The extractive QA example supplies a Chinese passage, question, and answer span, with prompts asking models to generate questions or identify answers from context.Templates include Chinese and English formulations, including generating a question for a specified answer and extracting the answer from a provided passage.Source: Data Example · p. 63, Prompts · pp. 63–64

Prompts

  • Prompts instantiate question-answering tasks through context completion, answer extraction, question generation, and direct answering in English, Chinese, and Vietnamese.Templates include machine-translated or multilingual variants and use dataset answers and answer positions as inputs.Source: Prompts · pp. 64, 65, 69, 73
  • Several prompt families explicitly warn that the prompt does not correspond to the original task intended by the dataset authors.This note appears for examples associated with Lewis et al. (2019) and a keyword-evaluation task.Source: Prompts · pp. 53, 68

Data Example

  • Additional examples cover keyword-label prediction, news-title topic classification, and code-complexity data with fields such as abstracts, labels, topics, source code, and complexity.The examples include a keyword-matching label, a Chinese news title with label -1, and a Java Codeforces instance marked linear complexity.Source: Data Example · pp. 77, 83, Prompts · p. 81

Prompts

  • The prompts cover time-complexity classification, function-name prediction, buggy-line repair, and masked-token replacement tasks, using code snippets or placeholders as task inputs.They include {{ code }}, {{buggy_line_content}}, and {{mask}} placeholders.Source: Prompts · pp. 83–86

Data Example

  • A code-repair example records tokenized source code, repair candidates, repair targets, bug kind VARIABLE_MISUSE, graph edges, and dataset provenance.The example includes a test_get_params function and provenance metadata for the underlying record.Source: Data Example · p. 84

1.10 Word Sense Disambiguation

  • The word-sense-disambiguation prompts are provided in Chinese and French alongside dataset identifiers for xlwic_en_zh and xlwic_fr_fr.The prompts use localized wording while retaining placeholders for the two contexts and target word.Source: §1.10 · p. 89, Prompts · pp. 89, 93
  • The word-sense-disambiguation examples compare a target word across two contexts, with prompts asking whether its meaning or usage is the same.Examples include the English word “summer” and a French item labeled language FR with label 1.Source: Data Example · pp. 89, 93, Prompts · pp. 89, 90, 93

1.11 Paraphrase Identification

Data Example

  • The data examples show paired inputs such as two sentences about the 1975–76 NBA season and a Vietnamese premise with two candidate choices.A Chinese example presents the same causal-choice structure and records effect as the question type with label 1.Source: Data Example · pp. 97, 105, 107

Prompts

  • The prompts use generalized, natural-question, concatenation, and generative formats for paraphrase identification, with answer choices supplied separately.One note states that a prompt does not correspond to the original task intended by the dataset authors.Source: Prompts · pp. 53, 97, 98, 99

1.12 Sentence Completion

  • Sentence completion uses a dataset from Ponti et al. (2020) for evaluation and presents a premise with two candidate causes or effects.Vietnamese and Chinese templates ask users to select the more reasonable option, with variants using causal connectors and answer-choice placeholders.Source: §1.12 · p. 105, Prompts · pp. 105, 106, 107, 110

1.13 Natural Language Inference

  • The natural-language-inference dataset from Conneau et al. (2018) is used for evaluation, with premise–hypothesis examples shown in English and Spanish.The displayed examples include truncated premise and hypothesis text.Source: §1.13 · p. 111, Data Example · pp. 111, 116

Prompts

1.13.2.1 Human-translated prompts

  • Human-translated Spanish prompts preserve the NLI formats while translating questions and labels into Spanish, including verdadero, falso, ninguno, sí, tal vez, and no.Examples include translations of inference, certainty, and correctness formulations.Source: §1.13.2.1 · pp. 116–117

1.13.2.2 Machine-translated prompts

  • Machine-translated Spanish prompts provide alternative translated formulations with Spanish questions and answer choices, including sí, quizás, no, and correcto, poco concluyente, incorrecto.Some templates retain English answer-choice strings such as "guaranteed", "possible", and "impossible".Source: §1.13.2.2 · pp. 118–119
Loading 2211.01786v2…