PaperlayerView exact revision on arXiv

Source-linked AI summary

The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning

Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, Minjoon Seo

arXiv:2305.14045v2cs.CLcs.AIcs.LG

TL;DR

Smaller language models have difficulty applying chain-of-thought reasoning to unseen tasks, and prior instruction tuning covered few CoT tasks. This paper introduces CoT Collection, with 1.84 million rationales across 1,060 tasks, and uses it to fine-tune Flan-T5. The resulting CoT-T5 models improve average accuracy over Flan-T5 on BBH and few-shot domain-specific tasks.

  • Problem

    A comprehensive strategy to generalize CoT across unseen tasks in smaller LMs remains lacking, while reliance on models above 100B parameters brings computational-cost and accessibility concerns.

    Source: §1 · p. 1

  • Method

    The authors create CoT Collection, adding 1.84 million rationales across 1,060 tasks to Flan Collection, and use it to CoT-fine-tune Flan-T5 into CoT-T5.

    Source: §1 · pp. 1–2

  • Results

    On BBH, CoT-T5 beats Flan-T5 by +4.34% (3B) and +2.60% (11B) average accuracy; on four domain-specific few-shot tasks, average gains are +2.24% (3B) and +2.37% (11B).

    Source: §1 · p. 2

  • Takeaways & Limitations

    CoT-T5 provides a stronger base for learning from few examples and better zero-shot generalization in the evaluated settings.

    Source: §6 · p. 9

    CoT-T5 is not trained for long-form dialogue responses and therefore does not necessarily have the ability to serve in chat applications.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Language models (LMs) with less than 100B parameters are known to perform poorly on chain-of-thought (CoT) reasoning in contrast to large LMs when solving unseen tasks. In this work, we aim to equip smaller LMs with the step-by-step reasoning capability by instruction tuning with CoT rationales. In order to achieve this goal, we first introduce a new instruction-tuning dataset called the CoT Collection, which augments the existing Flan Collection (including only 9 CoT tasks) with additional 1.84 million rationales across 1,060 tasks. We show that CoT fine-tuning Flan-T5 (3B & 11B) with CoT Collection enables smaller LMs to have better CoT capabilities on unseen tasks. On the BIG-Bench-Hard (BBH) benchmark, we report an average improvement of +4.34% (Flan-T5 3B) and +2.60% (Flan-T5 11B), in terms of zero-shot task accuracy. Furthermore, we show that instruction tuning with CoT Collection allows LMs to possess stronger few-shot learning capabilities on 4 domain-specific tasks, resulting in an improvement of +2.24% (Flan-T5 3B) and +2.37% (Flan-T5 11B), even outperforming ChatGPT utilizing demonstrations until the max length by a +13.98% margin. Our code, the CoT Collection data, and model checkpoints are publicly available.

1 Introduction

The paper introduces CoT Collection, an instruction-tuning dataset with 1.84 million rationales across 1,060 tasks, and reports improved zero-shot and few-shot CoT performance for smaller language models.

  • CoT prompting benefits LMs above 100B parameters but not necessarily smaller models, while prior approaches lacked a comprehensive strategy for generalizing CoT to many unseen tasks.The introduction notes that single-task fine-tuning does not address broad generalization, and that using only nine CoT tasks amid 1,827 direct-finetuning tasks yielded poor CoT results for smaller LMs.Source: §1 · p. 1
  • CoT Collection adds 1.84 million rationales across 1,060 tasks to support CoT fine-tuning; Flan-T5 (3B and 11B) fine-tuned on it is called CoT-T5.Source: §1 · pp. 1–2
  • CoT-T5 outperforms Flan-T5 by +4.34% (3B) and +2.60% (11B) in average accuracy across 27 BBH datasets with CoT prompting.Source: §1 · p. 2
  • Low-compute ablations report +8.65% average accuracy for T0 (3B) on 11 P3 datasets and 2x–10x average-accuracy improvement for mT0 (3B) across five MGSM languages.The T0 experiment uses 163 training tasks; the multilingual experiment translates 80K instances into French, Japanese, Korean, Russian, and Chinese.Source: §1 · p. 2
  • CoT-T5 improves average performance over Flan-T5 by +2.24% (3B) and +2.37% (11B) across four domain-specific tasks, and over ChatGPT and Claude by +13.98% and +8.11%.The ChatGPT and Claude comparisons use in-context learning with demonstrations up to the maximum input length.Source: §1 · p. 2

2 Related Works

Prior work shows that CoT prompting benefits large language models, while smaller models need rationale-based instruction tuning to improve reasoning and generalization. This work extends CoT fine-tuning across 1,060 tasks and applies LoRA for few-shot adaptation.

2.1 Chain-of-Thought (CoT) Prompting

  • CoT prompting, including zero-shot “Let’s think step by step” and automated or tree-search demonstration variants, improves large LMs, but its effectiveness does not necessarily extend to smaller LMs.The work aims to equip smaller LMs with similar capabilities by instruction tuning on large amounts of rationales.Source: §2.1 · p. 2

2.2 Improving Zero-shot Generalization

  • Building on instruction tuning’s generalization to unseen tasks and CoT fine-tuning’s gains on single seen tasks, this work fine-tunes on 1,060 instruction tasks for zero-shot generalization.Prior instruction-tuning extensions target cross-lingual and label generalization, as well as modular expert LMs.Source: §2.2 · p. 2

2.3 Improving Few-Shot Learning

  • Applying LoRA to a CoT-capable LM and CoT fine-tuning it on a target task is a simple recipe for strong few-shot performance.The CoT Collection comprises 1.84 million rationales across 1,060 tasks, compared with nine datasets with publicly available rationales.Source: §2.3 · p. 2, Figure 1 · p. 3

3 The COT COLLECTION

The CoT Collection expands rationale-based instruction tuning with 1.84 million rationales across 1,060 tasks, generated for selected Flan Collection tasks using large language models. Its construction uses task-family demonstrations, filters task selection and generated rationales, and notes limitations related to the augmentation model.

The COT Collection

  • The CoT Collection adds 1.84 million rationales across 1,060 tasks to address the scarcity of rationale data.The dataset augments the existing Flan Collection with rationales for instruction-tuning tasks.Source: §3 · p. 3

Rationale Augmentation

  • Large language models generate rationales for existing instruction-instance-answer inputs through in-context learning, rather than generating new instances.The MCQA prompt conditions rationale generation on the ground-truth label.Source: §3 · p. 3, Figure 2 · p. 4

Source Dataset Selection

  • Source tasks come from the 1,836-task Flan Collection, with exclusions for long outputs, unavailable or mismatched datasets, and tasks producing uninformative rationales.Long-output tasks were excluded when rationale-plus-answer output would exceed the 512-token training limit; tasks such as sentiment analysis, sentence completion, coreference resolution, and word disambiguation were excluded to avoid negative transfer.Source: §3 · pp. 3–4

Creating Demonstrations for ICL

  • For 26 task-format families, three authors manually created 6–8 shared demonstrations per family using sampled Flan instances and rationale annotation with A/B testing.Two annotators wrote rationales for sampled instances, and a third compared the two options.Source: §3 · p. 4

Rationale Augmentation

  • The authors used OpenAI Codex, found label-before-rationale demonstrations crucial for quality, and filtered generated rationales because some tasks produced poor rationales.Codex was used due to budget constraints and the unavailability of other LLM services during augmentation; the paper points to appendices for filtering criteria and reproducibility analyses.Source: §3 · p. 4

4 Experiments

Experiments test CoT-T5 on zero-shot BBH, P3, and multilingual MGSM tasks, plus few-shot legal and medical adaptation. CoT fine-tuning improves BBH CoT evaluation and yields the strongest few-shot results when combined with LoRA.

4.1 Evaluation

  • Direct Evaluation selects classification verbalizers by logit probability and scores accuracy, while generation tasks use exact match; CoT Evaluation requires at least eight generated rationale tokens.For CoT classification, the model generates a rationale before verbalizer scoring, with an “[ANSWER]” indicator separating rationale and options.Source: §4.1 · pp. 4–5

Setup #1: CoT Fine-tuning with 1060 CoT Tasks

  • +4.34% (3B) and +2.60% (11B) CoT Evaluation gains over Flan-T5 accompany a +2.57% total average gain for CoT-T5-11B, which also outperforms roughly four-times-larger T0 and Tk-Instruct models in both evaluation modes.CoT-T5 is Flan-T5 fine-tuned on all 1,060 CoT Collection tasks; evaluation is zero-shot on BBH’s 27 unseen datasets.Source: Table 1 · p. 5, §4.2 · p. 5, Setup #1: CoT Fine-tuning with 1060… · p. 5

Setup #2: CoT Fine-tuning with 163 CoT Tasks (T0 Setup)

Setup #3: Multilingual Adaptation with CoT Fine-tuning

  • With 60K–80K target-language instances, CoT fine-tuning yields non-trivial MGSM gains across five languages, including Korean, Japanese, and Chinese; cross-lingual transfer was not tested.The experiments adapt and evaluate in a single target language, so the results do not establish transfer between languages.Source: Table 4 · p. 7, Setup #1: CoT Fine-tuning with 1060… · p. 7

4.3 Few-shot Generalization

  • Few-shot evaluation uses 64 randomly sampled training examples from each of four legal and medical datasets, with augmented rationales and regular, CoT, LoRA, and LoRA CoT fine-tuning compared.Results average accuracy across three random-seed runs; Claude and ChatGPT serve as ICL baselines with demonstrations appended up to their maximum context lengths.Source: §4.3 · pp. 7–8
  • CoT-T5 with CoT fine-tuning plus LoRA performs best overall; unlike Flan-T5, its LoRA adaptation beats full fine-tuning, and fine-tuning methods overall surpass ICL baselines.The authors attribute ICL’s lower results to long legal and medical inputs limiting the demonstrations that fit in context, while longer contexts make ICL computationally expensive.Source: Table 5 · p. 8, §4.3 · p. 8

5 Analysis of of CoT Fine-tuning

The analysis compares expanding task diversity with adding instances and tests whether CoT fine-tuning preserves accuracy on in-domain tasks.

5.1 Scaling the number of tasks & instances

  • 10K instances across 1,060 tasks outperform 180K instances across 9 tasks, indicating that task breadth is more crucial than instance count.The comparison uses randomly sampled CoT Collection instances against all 180K instances from the 9 existing CoT tasks.Source: Figure 3 · p. 9, §5.1 · p. 9

5.2 In-domain Task Accuracy of CoT-T5

  • CoT-T5 consistently improves CoT accuracy on the five tested in-domain tasks already used for fine-tuning.The authors conjecture this may be because Flan-T5 and CoT-T5 were fine-tuned on the same tasks; performance on additional tasks not used in training remains unexplored.Source: Figure 4 · p. 9, §5.2 · p. 9

6 Conclusion

The work introduces the CoT Collection, a large-scale instruction-tuning dataset of LLM-extracted rationales, and uses it to fine-tune Flan-T5 into CoT-T5. CoT-T5 shows better zero-shot generalization and serves as a stronger base model when training with few examples.

6 Conclusion

  • The CoT Collection contains 1.84M CoT rationales extracted across 1,060 NLP tasks using OpenAI Codex.The dataset augments instruction-tuning data with rationales generated using an LLM.Source: §6 · p. 9
  • CoT fine-tuning Flan-T5 with the collection produces CoT-T5, which shows better zero-shot generalization and serves as a better base model with few training instances.The conclusion presents these as improvements in reasoning capabilities for smaller language models.Source: §6 · p. 9

Limitations

COT-T5 is not designed for long-form dialogue or multilingual reasoning, and the authors identify reproducibility and rationale-quality limitations in their data-generation process.

Dialogue applications

  • COT-T5 was not trained on long-form dialogue, so it may not be suitable for chat applications despite focusing on academic-task zero- and few-shot capabilities.The authors identify combining long-form chat data with academic-task instruction data as future work.Source: Limitations · p. 10

Multilingual reasoning

  • Smaller LMs show nearly zero accuracy on math problems in different languages; CoT fine-tuning offers only slight improvement, motivating more comprehensive multilingual training.COT-T5 inherits Flan-T5’s limitation in step-by-step reasoning across diverse languages.Source: Limitations · p. 10

Reproducibility and data quality

  • Rationale augmentation raises reproducibility concerns because the proprietary Codex model used was shut down, and the authors see room to improve dataset quality with stronger models and prompting.They suggest GPT-4 and Tree of Thoughts prompting; later work explored GPT-4-augmented rationales and combining Chain-of-Thought with Program of Thoughts.Source: Limitations · p. 10

A Analysis of COT COLLECTION

The analysis finds that CoT Collection rationales are generally high quality, span more textual formats than earlier CoT datasets, and can be reproduced with different LLMs.

Diversity of Rationales

Quality of Rationales

Is COT COLLECTION Reproducible?

B Filtering COT COLLECTION

The CoT Collection filters generated rationales for consistency, length, duplication, repetition, and code contamination. The authors also report that nucleus sampling with a 0.8 probability threshold and a 3-gram repetition limit generated good-quality rationales.

Filtering

  • Rationales are filtered if they omit the ground-truth answer, exceed a combined rationale-and-answer length of 512 tokens, duplicate earlier augmentations, or contain repetitive sentences.The authors note that omitting the answer does not necessarily make a rationale bad, but excluding such instances was effective for removing inconsistent ones.Source: B Filtering COT COLLECTION · p. 15
  • Trigger-token filtering removes rationales when Codex continues by generating code after the rationale.The triggers were selected because they frequently appear at the start of code snippets.Source: B Filtering COT COLLECTION · p. 15
  • Sampling with p =0.8 and no_repeat_n_gram =3 was very effective in generating good-quality rationales.Source: B Filtering COT COLLECTION · p. 16

C Training and Evaluation Details of CoT-T5

This section describes how CoT Collection rationales were sampled and filtered, and how CoT-T5 was trained and evaluated. It also presents examples and analyses of the rationale data.

C Training and Evaluation Details of CoT-T5

  • CoT-T5 was trained for one epoch; AdamW worked well for the 3B model, while Adafactor optimized the 11B model.Training took one day for 3B and seven days for 11B using eight A100 80GB GPUs, with gradient accumulation of eight for both sizes.Source: C Training and Evaluation Details of… · p. 16
  • Filtering generated rationales for ground-truth-token consistency produced the CoT Collection of 1.84 million instances and rationales across 1,060 tasks.The collection was built by sampling from Flan, P3, nine existing CoT datasets, SNI, and other dialogue and code datasets, then generating five rationales per instance. Missing the ground-truth token could indicate an inconsistent rationale, often degeneration, a pattern mostly found in math datasets.Source: C Training and Evaluation Details of… · p. 16, Table 9 · p. 19
  • Figure 5 compares the top 20 common rationale root verbs and their top four noun objects for nine CoT tasks and the 1,060-task CoT Collection.The figure places the nine-task analysis on the left and the CoT Collection analysis on the right.Source: Figure 5 · p. 17
  • Figure 6 compares rationale augmentation quality across LLMs using reference-free ROSCOE scores for semantic alignment, semantic similarity, logical inference, and coherence.Source: Figure 6 · p. 17
  • Table 8 provides rationale examples for multiple-choice, extractive, and closed-book QA, formal logic, natural language inference, and arithmetic tasks.Source: Table 8 · p. 18

D Prompts used for Rationale Augmentation

This appendix section presents rationale-augmentation demonstrations across several task families, including MCQA, Commonsense, NLI, and arithmetic. The examples pair questions with answers and, in some cases, worked explanations that justify the answers.

Prompts 1: Demonstration used for tasks in family of MCQA (T0)

  • MCQA (T0) demonstrations pair questions and answer options with rationales, covering tasks such as science, medical, and everyday reasoning.Examples include judging whether bicycles can run on alcoholic fermentation, identifying a specialist for a ruptured cyst, and reasoning about the effect of no water on vegetables.Source: D Prompts used for Rationale… · p. 21

Prompts 2: Demonstration used for tasks in family of Commonsense (SNI)

  • Commonsense (SNI) demonstrations include evidence-bounded judgments, context-and-hypothesis generation, and verification of factual claims.Examples include rejecting a general trend based on data from one clinic, assessing whether a writer is popular, and confirming where a burger franchise originated.Source: D Prompts used for Rationale… · pp. 22–23

Prompts 3: Demonstration used for tasks in family of NLI (FLAN)

  • NLI demonstrations pair classification questions with rationales that use context to justify “No” or “it is not possible to tell.”One example contrasts a hypothesis with context stating the FBI list began in the 1950s; another notes that the premise does not specify how many trees are present.Source: D Prompts used for Rationale… · p. 23

Prompts 4: Demonstration used for tasks in family of Arithmetic (FLAN)

  • Arithmetic demonstrations show explicit addition and subtraction calculations to solve single-step word problems.Examples calculate 21 − 15 = 6 trees planted and 3 + 2 = 5 cars in the lot.Source: D Prompts used for Rationale… · p. 23
  • Other arithmetic rationales work through multistep problems by calculating intermediate totals, repeated additions, or costs before giving the result.Examples total chocolates before subtracting those eaten, multiply daily computer installations before adding them to the initial count, and calculate bagel costs before subtracting them from the starting money.Source: D Prompts used for Rationale… · pp. 23–24

Related papers

Loading 2305.14045v2…