PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model

Seongjin Shin, Sang-Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Woo Ha, Nako Sung

arXiv:2204.13509v2cs.CL

TL;DR

The paper asks how pretraining corpus properties relate to in-context learning, a relationship that had received limited in-depth analysis. It varies corpus sources and sizes in HyperCLOVA. It finds that source and corpus combinations matter, while task relevance and perplexity alone do not reliably indicate few-shot performance.

  • Problem

    How pretraining corpus choice relates to downstream in-context accuracy, and how validation perplexity relates to in-context performance, remain insufficiently investigated.

    Source: §1 · p. 1

  • Method

    The authors vary pretraining corpus sources and sizes in the Korean-centric HyperCLOVA model and analyze in-context learning effects.

    Source: §1 · p. 1

  • Results

    In-context learning varies with corpus source, can emerge from combining corpora, and is not reliably indicated by task relevance or perplexity in few-shot settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Corpus-source properties and combinations warrant attention when assessing in-context learning; task proximity or low perplexity alone may not signal competitive few-shot performance.

    Source: §1 · p. 2

    The study uses the HyperCLOVA corpus, and the authors identify investigation of other corpus sources as future work.

    Source: §1 · p. 1, §6 · p. 9

Abstract

from arXiv · show

Many recent studies on large-scale language models have reported successful in-context zero- and few-shot learning ability. However, the in-depth analysis of when in-context learning occurs is still lacking. For example, it is unknown how in-context learning performance changes as the training corpus varies. Here, we investigate the effects of the source and size of the pretraining corpus on in-context learning in HyperCLOVA, a Korean-centric GPT-3 model. From our in-depth investigation, we introduce the following observations: (1) in-context learning performance heavily depends on the corpus domain source, and the size of the pretraining corpus does not necessarily determine the emergence of in-context learning, (2) in-context learning ability can emerge when a language model is trained on a combination of multiple corpora, even when each corpus does not result in in-context learning on its own, (3) pretraining with a corpus related to a downstream task does not always guarantee the competitive in-context learning performance of the downstream task, especially in the few-shot setting, and (4) the relationship between language modeling (measured in perplexity) and in-context learning does not always correlate: e.g., low perplexity does not always imply high in-context few-shot learning performance.

1 Introduction

The paper addresses limited evidence on how pretraining corpus properties relate to in-context learning, and on whether validation perplexity predicts it. Using HyperCLOVA, it varies corpus sources and sizes and finds that source, combinations, downstream relevance, and perplexity relate to in-context performance in non-uniform ways.

1 Introduction

  • The relationship between pretraining corpus choice and in-context task accuracy was unknown, while prior domain-similarity findings concerned finetuned models.The paper also notes that explicit analysis of the relationship between validation perplexity and in-context learning was limited.Source: §1 · p. 1
  • The authors vary pretraining corpus sources and sizes in HyperCLOVA, a Korean-centric large language model, to analyze their effects on in-context learning.They focus mainly on few-shot learning and also explore zero-shot learning.Source: §1 · p. 1
  • In-context learning performance depends heavily on corpus source: Blog supports competitive few-shot learning, while Cafe and News hardly yield it.The paper also investigates corpus size, but this finding highlights differences across sources.Source: §1 · p. 2
  • Combining KiN and Ency produces few-shot in-context learning even though neither corpus alone showed that ability.KiN consists of QnA websites and Ency of encyclopedia websites.Source: §1 · p. 2
  • News-related pretraining seems to help zero-shot performance on a news task, but does not guarantee competitive few-shot learning.The example is news topic classification from titles on KLUE-YNAT.Source: §1 · p. 2
  • Across models trained on different corpora, low validation perplexity does not necessarily indicate strong few-shot learning.Cafe has the second-lowest validation perplexity across domain sources after Blog, yet fails to show few-shot learning.Source: §1 · p. 2

2 Related Work

Prior work studies in-context learning, domain relevance, pretraining-data quality, and multitask approaches, but leaves corpus-source effects on in-context learning less explored.

2.1 In-context Learning

  • In-context learning uses task descriptions and examples with a new input, without parameter updates; zero-shot omits examples, while few-shot includes them.The related-work discussion also notes studies seeking to improve or explain few-shot learning.Source: §2.1 · p. 2

2.2 Domain Relevance on Pretraining Corpus

  • Earlier domain-similarity findings concern finetuned models, while Codex studies code-comment and corresponding-code pretraining rather than varied domains.The authors position their study as examining multiple pretraining domains.Source: §2.2 · p. 2

2.3 Quantity and Quality of Pretraining Corpus

  • Prior work examines pretraining-data quantity and quality, including filtered C4 improving T5 downstream performance; this paper instead analyzes source and amount effects on in-context learning.Data filtering is also described as a general approach to improving language-model quality.Source: §2.3 · p. 2

2.4 Multi-task Learning

  • Where recent multitask approaches target zero- and few-shot transfer, this study reports that diverse pretraining corpora strengthen in-context learning.The authors contrast their focus with finetuning studies, which leave many corpus-related properties of in-context learning unknown.Source: §2.4 · pp. 2–3

3 Task Definition

The experiments vary HyperCLOVA’s model sizes and pretraining corpus sources, then evaluate corpus-specific models on four Korean downstream tasks. They also describe training settings and a shared validation procedure for measuring perplexity.

3.1 Model

  • The study mainly uses 1.3B-parameter HyperCLOVA models, includes 6.9B models, and holds the 2,048-token sequence limit and vocabulary constant.All variants use a morpheme-aware byte-level BPE tokenizer trained on the HyperCLOVA corpus.Source: §3.1 · p. 3

3.2 Pretraining with Different Corpus

  • Seven HyperCLOVA subcorpora span blogs, community sites, news, comments, QnA, encyclopedic text, and public datasets; heterogeneous Others is excluded from domain-source experiments.Each training corpus is capped at 150B tokens; individual validation sets contain 10,000 examples, and seven are combined for perplexity measurement.Source: §3.2 · p. 3

3.3 Downstream Tasks

  • Four Korean benchmarks cover sentiment (NSMC), reading comprehension (KorQuAD), translation (AI Hub), and seven-class topic classification (YNAT).KorQuAD draws from Wikipedia/Ency, YNAT and much of AI Hub relate to News, and KiN relates to translation; Figure 1 measures top-1,000-noun overlap.Source: §3.3 · pp. 3–4, Figure 1 · p. 4

3.4 Experimental Details

  • Training uses 72K steps and global batch size 1,024; Ency uses its 12K-step minimum-loss checkpoint, with task-specific shot counts, repeats, and decoding.AdamW at 2.0e-4 with cosine scheduling is used; few-shot counts are 70/4/4/70 for NSMC/KorQuAD/AI Hub/YNAT.Source: §3.4 · p. 4

3.5 Measuring Validation Perplexity

  • Validation perplexity uses the same vocabulary and a 70,000-example set drawn from seven corpus sources.Multi-domain model PPLs are italicized because they are generally lower than single-domain PPLs, cautioning against direct comparison.Source: §3.5 · pp. 4–5

4 Experimental Results

The experiments show that corpus source and combinations can shape in-context learning differently, while corpus size and domain relevance do not reliably predict few-shot performance. The relationship between perplexity and in-context performance also differs across corpora and training steps.

4.2 Effect of Corpus Source

  • Blog achieves competitive few-shot performance, while Cafe and News alone show little ability; Modu also outperforms Cafe and News despite using less than one-tenth their corpus size.Cafe+News also performs poorly despite matching Blog and ALL in size; finetuning performance differences are much smaller than in-context differences.Source: §4.2 · p. 5, Table 2 · p. 6

4.4 Effect of Combining Corpora

  • Combining KiN and Ency, or Cafe and KiN, yields few-shot ability on most tasks even though each component corpus alone fails on most tasks.Combining Cafe and News does not ensure competitive results: its few-shot scores remain below the basis on NSMC, KorQuAD, and YNAT, with NSMC and YNAT even decreasing.Source: §4.4 · p. 6, Table 4 · p. 7

4.3 Effect of Corpus Size

  • Reducing ALL from 150B to 56B tokens does not sharply lower few-shot performance, while 6B tokens causes a marked drop but still beats 150B-token Cafe+News.The 56B and 6B datasets are randomly sampled from the original corpus.Source: §4.3 · p. 5, Table 6 · p. 7

4.5 Effect of Domain Relevance

  • Task-related pretraining does not guarantee competitive few-shot results: KiN and Ency underperform on KorQuAD, and News underperforms on YNAT despite their domain overlap.High vocabulary overlap also does not ensure high task performance: Modu has large overlap with AI Hub but performs below Blog and KiN on translation tasks; KiN is competitive on AI Hub translation, best on Ko→En among seven corpora.Source: §4.5 · p. 7

4.6 Perplexity and Downstream Task

  • Across corpus models, perplexity is not a strong predictor of few-shot performance: Cafe and KiN have relatively low perplexity but show no few-shot ability.Blog has both the lowest perplexity and best in-context performance, while Ency has both the highest perplexity and worst performance; corpus size can affect few-shot performance more than perplexity.Source: Figure 2 · p. 8, §4.6 · p. 8

4.5 Effect of Domain Relevance

  • News-containing models outperform ALL on zero-shot KLUE-YNAT, while adding KiN consistently improves zero-shot AI Hub performance and the resulting models outperform ALL.For KiN on AI Hub, the zero-shot increase is less significant than the few-shot increase.Source: §4.5 · p. 8

4.6 Perplexity and Downstream Task

  • Zero-shot performance appears to correlate with perplexity more than few-shot performance, although Modu combines relatively high perplexity with relatively high zero-shot performance.Figure 2 plots normalized few-shot performance against perplexity for corpus models.Source: §4.6 · p. 8, Figure 2 · p. 8
  • For a single model across training steps, validation perplexity and few-shot performance correlate well, unlike the less explicit relationship across corpus domains.The comparison tracks five downstream tasks for the 1.3B-sized ALL model.Source: Figure 4 · p. 9, §4.6 · p. 9

5 Discussion

The discussion considers how corpus selection could inform pretraining when data are limited, while questioning whether perplexity reliably predicts few-shot performance.

5 Discussion

  • The authors propose using their findings to improve in-context learning when available or needed corpora are limited.Source: §5 · p. 9
  • The authors suggest their findings could guide corpus collection for multilingual and low-resource language models.They note that XGLM performs better for many languages but does not reach a single-language model's performance.Source: §5 · p. 9
  • Perplexity-based corpus reweighting lacks explicit evidence of improving downstream performance, and its relationship to few-shot in-context learning remains uncertain.The authors note that Gopher uses validation perplexity to set subcorpus ratios and question a strong perplexity–few-shot correlation.Source: §5 · p. 9

6 Conclusion

The conclusion emphasizes corpus-source effects on whether in-context learning emerges and identifies further work on source properties and broader corpus coverage.

6 Conclusion

  • Corpus source plays a crucial role in whether in-context learning ability emerges in a large-scale language model.Source: §6 · p. 9
  • Future work should investigate linguistic properties of corpus sources and study sources beyond the HyperCLOVA corpus.Quantifying differences between corpora could help NLP practitioners select suitable corpora.Source: §6 · p. 9

Broader Impact Statement

The broader-impact discussion links smaller-corpus training to potential energy benefits while flagging a possible domain-bias risk from corpus reweighting.

Broader Impact Statement

  • Models trained on part of the pretraining corpus can match whole-corpus models in in-context performance, though validation at tens of billions of parameters remains future work.The authors suggest this may help alleviate energy consumption associated with large-scale language models.Source: Broader Impact Statement · p. 10
  • Reweighting a domain-specific corpus might cause a language model to inherit bias present in that domain corpus.Source: Broader Impact Statement · p. 10

A Details on Experimental Results

The appendix points to supplementary experimental-result tables and a perplexity figure, describes the pretraining corpus and preprocessing, and details its deduplication procedure.

A Details on Experimental Results

  • Tables 8–10 and Figure 5 provide supplementary experimental-result details, including validation perplexity across models trained on different corpora.Table 10 adds score differences against ALL to in-context learning scores; Figure 5 supports Table 7.Source: A Details on Experimental Results · p. 12

B Details on Pretraining Corpus

B.1 Deduplication Preprocess

  • Deduplication removes near-duplicate documents within corpus subparts and then between them; overlap between subparts was 0.024% of the total.The 29 subparts are grouped into eight domains, including Blog, News, Cafe, Comments, KiN, Modu, Ency, and Others.Source: §B.1 · p. 12

C Experiments on LoRA

Table 11 reports LoRA fine-tuning results for some models from Tables 2 and 4.

D Examples of Few-shot Prompt

The appendix provides few-shot prompt examples and translated versions, explains the single-seed KorQuAD setting, and identifies supplemental results tables.

E Generalization to Other Languages

The authors leave experiments on non-Korean languages for future work, while arguing that their findings may generalize and describing practical challenges in controlling corpus sources.

  • The authors leave experiments on non-Korean languages, including English, for future work.Source: E Generalization to Other Languages · p. 13
  • They consider their findings basically generalizable because fundamental Korean-English differences for pretraining and in-context learning have been limitedly reported.They note that XGLM provides no critical evidence on language-specific properties.Source: E Generalization to Other Languages · p. 13
  • Corpus control is difficult because many in-context-learning corpora come from crawled websites whose original sources are hard to distinguish.The authors use a relatively refined corpus with subcorpora from one web service and identify controlling Pile subcorpora as future work.Source: E Generalization to Other Languages · p. 14
Loading 2204.13509v2…