Datasets for Large Language Models: A Comprehensive Survey

• Dataset development shifted from task-centric construction toward construction organized around both tasks and LLM stages, from pre-training through evaluation.Source: §1 · pp. 3–4
• The survey addresses the difficulty of navigating the LLM dataset explosion and the lack of systematic organization across dataset types, domain orientations, and real-world scenarios.Source: §1 · p. 4
• 444 datasets are covered in statistics spanning 8 language categories, 32 domains, and 20 dimensions, with over 774.5 TB of pre-training corpora and over 700M instances for other datasets.The survey also discusses bottlenecks and future research directions.Source: §1 · p. 4
• The survey summarizes five dataset dimensions: pre-training, instruction fine-tuning, preference, evaluation, and traditional NLP datasets.Source: §1 · p. 4
• Earlier surveys covered LLM developments or individual dataset stages, whereas this work focuses specifically on LLM datasets.The authors describe it as the first survey focused on LLM datasets, to their knowledge.Source: §1 · p. 4

2.1 General Pre-training Corpora

• General pre-training corpora combine diverse sources for broad language knowledge, while domain-specific corpora target specialized knowledge; eight general data categories are identified.The categories are webpages, language texts, books, academic materials, code, parallel corpora, social media, and encyclopedias.Source: §2 · p. 5, §2.1 · p. 6

2.1.1 Webpages; 2.1.2 Languages Texts

• Webpages are the most prevalent source, but their changing, multilingual, semi-structured, noisy content requires cleaning; language-text collections range from broad language corpora to domain-focused materials.Language texts include written and spoken materials as well as field-specific sources such as financial reports, legal regulations, and news.Source: §2.1.1 · p. 6, §2.1.2 · p. 9

2.1.3 Books; 2.1.4 Academic Materials

• Books provide long, relatively high-quality and diverse contextual text, while academic materials add rigorous specialist knowledge and are commonly included in mixed corpora.Books span many subjects and can be categorized into 30 domain classes; academic materials support learning professional terminology and knowledge.Source: §2.1.3 · pp. 9–10, §2.1.4 · pp. 10–11

2.1.5 Code; 2.1.6 Parallel Corpus

• Code corpora support programming-language understanding and code tasks, while parallel corpora pair translations to strengthen machine translation and cross-lingual performance.Source: §2.1.5 · p. 11, §2.1.6 · p. 12

2.1.7 Social Media; 2.1.8 Encyclopedia

• Social-media data offers interactive, real-time conversational language and social trends despite possible harmful content; encyclopedias contribute accessible, relatively authoritative knowledge.Reddit vote counts can also support construction of human-preference datasets.Source: §2.1.7 · p. 13, §2.1.8 · p. 13

2.1.9 Multi-category Corpora

• Multi-category mixtures can improve generalization, but their composition varies: RedPajama-V1 is 87.0% webpages, while The Pile is more balanced and Dolma combines six domains in 3T tokens.Chinese MNBVC targets 40TB; ArabicText 2022 is an open-source Arabic corpus; WanJuanText-1.0 combines Chinese-English data, while ROOTS spans 46 natural and 13 programming languages at 1.6TB.Source: §2.1.9 · pp. 14–15

2.2 Domain-specific Pre-training Corpora; 2.2.1 Financial Domain; 2.2.2 Medical Domain

• Domain-specific corpora extend a generally pretrained model through incremental training; financial and medical collections supply field-relevant materials such as reports, literature, and textbooks.Financial sources include news, statements, research reports, and market data; medical sources include diagnostic records, case reports, and biomedical publications.Source: §2.2 · p. 15, §2.2.1 · p. 16, §2.2.2 · pp. 16–17

2.2.3 Other Domains; 2.3 Distribution Statistics of Pre-training Corpora

• Specialized collections also cover legal texts, transportation information, and mathematical code, web data, and papers; the survey summarizes 59 pre-training corpora across six attributes.The attributes are release time, license, data category, construction method, language, and domain.Source: §2.2.3 · p. 17, §2.3 · p. 17

2.3 Distribution Statistics of Pre-training Corpora

• Open-source pre-training corpora grew explosively in 2023 after slow growth before 2018 and gradual acceleration following BERT; Apache-2.0, ODC-BY, CC0, and Common Crawl terms are commonly used.The survey advises reviewing each corpus’s license terms before use to ensure compliance.Source: §2.3 · p. 17
• Multi-category corpora are most numerous and preferred; webpages dominate single-type corpora, while English, Chinese, and multilingual resources outnumber corpora for other languages.General-purpose corpora lead, domain-specific open corpora are limited, and excess data from one domain can impair generalization; code corpora are also increasingly used.Source: §2.3 · pp. 18–19

2.4 Preprocessing of Pre-training Data

• Pre-training data preprocessing generally follows five stages: collection, filtering, deduplication, standardization, and review.The workflow aims to improve data quality and standardization while reducing harmful and sensitive content.Source: §2.4 · p. 19

2.4.1 Data Collection

• Collection begins by defining model-specific data requirements, then selecting reliable sources and a strategy before crawling and extracting text.The process also includes encoding and language detection, backups, privacy and legal compliance, and ongoing maintenance; crawling must follow applicable laws and site policies.Source: §2.4.1 · pp. 20–21

2.4.2 Data Filtering

• Filtering uses trained models or document- and sentence-level heuristics to remove low-quality, harmful, irrelevant, or otherwise unsuitable text.Model-based examples include separate safety and quality classifiers; heuristic criteria include language, text density, document length, sentence completeness, privacy, and unwanted markup.Source: §2.4.2 · pp. 21–22

2.4.3 Data Deduplication

• Deduplication removes duplicate or highly similar text using methods such as TF-IDF soft deduping, MinHash, SimHash, repeated-sentence checks, and URL matching.WanJuanText-1.0 combines MinHashLSH and n-grams, removing content with similarity greater than 0.8.Source: §2.4.3 · pp. 22–23

2.4.4 Data Standardization

• Standardization makes text more manageable for training through sentence splitting, Chinese-script conversion, spelling correction, and stop-word removal.Examples include converting traditional Chinese to simplified Chinese and removing spaces from Chinese text.Source: §2.4.4 · p. 23

2.4.5 Data Review

• Data review documents prior preprocessing methods, manually samples processed data against expected standards, and feeds identified issues back to earlier stages.Review can be conducted at the end of each preceding preprocessing step.Source: §2.4.5 · pp. 23–24

3.1 Instruction Category

• The survey groups single-turn instructions into 15 classes spanning reasoning, math, QA, coding, generation, transformation, translation, role-play, social norms, and other tasks.The categories also include extraction, summarization, and classification.Source: §3.1 · pp. 24–26

3.2 General Instruction Fine-tuning Datasets

• General instruction datasets cover tasks across domains to improve broad instruction following, and are classified by four construction methods: human, model, existing-dataset improvement, or combinations.The survey organizes general-dataset information in Tables 5 and 6.Source: §3.2 · pp. 26–27

3.2.1 Human Generated Datasets

• Human-generated datasets offer quality, interpretability, and flexible control, but require substantial labor and time and can reflect annotator subjectivity.Another route is to scrape human-generated web Q&A and standardize it as instructions.Source: §3.2.1 · pp. 28–30

3.2.2 Model Constructed Datasets

• Model-constructed datasets can provide abundant instructions efficiently, but generated content may be low-quality, biased, or hallucinatory and requires post-processing.Construction approaches include prompting models to generate instructions, using real human–LLM conversations, and generating dialogue through multiple LLM agents.Source: §3.2.2 · pp. 30–31

3.2.3 Collection and Improvement of Existing Datasets

• Repository datasets combine and modify existing open-source datasets, offering diversity, scale, and time savings while requiring quality and format standardization and attention to licenses.The survey compiles 16 such datasets, sourced primarily from traditional NLP and instruction fine-tuning datasets.Source: §3.2.3 · p. 32

3.2.4 Datasets Created with Multiple Methods

• Combined-method datasets use four pairings or groupings of human generation, model construction, and existing-dataset improvement to leverage their complementary strengths.Examples include manually supplementing or selecting collected data and using existing instructions as inputs for model-generated responses.Source: §3.2.4 · p. 34
• Multi-method datasets span translation, human annotation, model construction, and collection: COIG has 191K Chinese instructions, HC3 37K Q&A pairs, and Phoenix-sft-data-v1 464K multilingual dialogues.TigerBot datasets combine collected and self-developed data; Aya Collection includes templated, translated, and human-annotated data, with translations spanning 101 languages.Source: §3.2.4 · pp. 35–36

3.3.1 Medical Domain; 3.3.2 Code Domain

• Medical datasets include real doctor-patient dialogues, while code datasets target generation and tool use, including ToolAlpaca’s 3,928 tool-use instances.Medical fine-tuning datasets show generalization in specific scenarios; CommitPackFT filters instruction data across 277 programming languages, and ToolBench uses API collection, instruction generation, and solution annotation.Source: §3.3.1 · p. 36, §3.3.2 · p. 38

3.3.3 Legal Domain; 3.3.4 Mathematics Domain

• Legal instruction datasets are relatively scarce; mathematical datasets vary in construction, and high-difficulty math instruction data remain scarce.DISC-Law-SFT adds legal reasoning and external-knowledge use; OpenMathInstruct-1 generates question-solution pairs from GSM8K and MATH, while other math datasets include synthetic problems and reformatted NLP datasets.Source: §3.3.3 · p. 39, §3.3.4 · p. 39

3.3.5 Education Domain; 3.3.6 Other Domains

• Education datasets support course guidance, emotional support, learning, and related tasks through child-companionship dialogues, educational instructions, and Chinese-language teaching resources.Other domain-specific resources include DISC-Fin-SFT’s 246K financial instructions, GeoSignal’s 40K cleaned instances, and MeChat’s 56K psychological-health instructions.Source: §3.3.5 · pp. 39–40, §3.3.6 · p. 40

3.3.6 Other Domains

• Other domain-specific datasets address biology, IT, social norms, and transportation, using biological-data transformations, IT-task instructions, norm-guided dialogue, and document-derived traffic conversations.Owl-Instruction contains 18K instructions; PROSOCIALDIALOG covers problematic and harmful scenarios, and TransGPT-sft generates traffic dialogues from documents.Source: §3.3.6 · pp. 40–41

3.4 Distribution Statistics of Instruction Fine-tuning Datasets

• Instruction fine-tuning datasets are growing; Apache-2.0 is most common (43 datasets), and 63 datasets contain 10K to 1M instances.The survey attributes growth to attention to LLMs and instruction tuning; it notes fewer datasets at both small and large scales.Source: Figure 14 · p. 41, §3.4 · pp. 41–42
• Model-constructed instructions are the most prevalent method; 22 datasets combine methods, while Chinese and English and general-domain datasets predominate.The survey also reports multilingual and mixed Chinese-English datasets, scarcity of open programming-language resources, and 67 general-domain datasets, with niche fields awaiting exploration.Source: §3.4 · p. 42

4 Preference Datasets

• Preference datasets pair instructions with competing responses and human or model feedback; the survey organizes examples in Tables 9 and 10, whose legends explain dataset and evaluation-method fields.Source: §4 · pp. 42–43, Table 9 · p. 43, Table 10 · p. 44

4.1 Preference Evaluation Methods

• Human and model feedback can use voting, sorting, scoring, or other methods; human annotation is more realistic but slower and costlier, while model feedback is cheaper but may carry bias.The section suggests combining forms and sources of preference data may be advantageous.Source: §4.1 · p. 44, Figure 16 · p. 45

4.1.1 Vote

• Voting selects preferred responses simply and intuitively, but gives less granular information; examples include human-voted Chatbot Arena and HH-RLHF, plus model-voted CValues safety comparisons.Chatbot Arena includes responses from 20 models in 96 languages; CValues compares responses across three safety categories.Source: §4.1.1 · pp. 44–45

4.1.2 Sort

• Sorting ranks multiple responses by criteria, yielding relative-preference detail but requiring cumbersome collection and standardized criteria; OASST1 pairwise RLHF reward derives rankings from annotated human dialogues.Source: §4.1.2 · pp. 45–46

4.1.3 Score

• Scoring gives multiple responses graded values, representing preference intensity more flexibly, but criteria uniformity and scorer subjectivity remain concerns.Examples include human-scored Stack-Exchange-Preferences and model-scored UltraFeedback, which scores answers across four dimensions and supplies explanations.Source: §4.1.3 · p. 46

4.1.4 Other

• Other datasets include Medical-rlhf’s 4K Chinese medical-dialogue examples and PRM800K’s step-level correctness labels for mathematical solutions.Medical-rlhf contrasts authentic doctor replies with model-generated responses; its scale is relatively small and its high/low-quality categories are described as overly direct and absolute.Source: §4.1.4 · pp. 46–47

4.2 Distribution Statistics of Preference Datasets

• Preference datasets grew rapidly in 2023, are predominantly English, and focus mainly on general and social-norm domains, with non-English and other vertical-domain datasets relatively scarce.The statistics cover 16 datasets across release time, license, evaluation method, construction method, language, and domain.Source: §4.2 · pp. 47–48, Figure 17 · p. 47
• Human voting is the most common evaluation method; human and model scoring follow, while sorting and combined methods are rare, and human annotation or model-assisted generation is the leading construction approach.Scraped social-platform Q&A with upvotes is another construction approach; most datasets are commercially usable, with Apache-2.0 predominant.Source: §4.2 · pp. 47–48

5.1 Evaluation Domains

• The survey classifies 112 evaluation datasets into 20 domains, spanning general, exam, subject, NLU, reasoning, knowledge, long text, tool, agent, code, OOD, and specialized domains.The domains also include law, medical, financial, social norms, factuality, evaluation, multitask, multilingual, and other; the survey groups LLM evaluation into five categories: knowledge and capability, alignment, safety, specialized LLMs, and evaluation organization.Source: §5.1 · pp. 48–49, Figure 18 · p. 48

5.1.1 General

• General benchmarks assess versatility across instructions and domains, as well as instruction-following under complex directives.Vicuna Evaluation uses GPT-4 to judge outputs across nine question categories; SuperCLUE adds multiturn open-ended and objective multiple-choice questions, and CELLO tests complex task descriptions and inputs.Source: §5.1.1 · pp. 49, 51, 52

5.1.2 Exam; 5.1.3 Subject

• Exam datasets use significant questions from national exams to assess question-intent comprehension and exam knowledge, while subject benchmarks range from discipline-specific tests to broad curricula.AGIEval covers 20 official, public, and stringent exams, including the Gaokao, SAT, bar exam, and civil service exam; MMLU covers 57 subjects, while XiezhiBenchmark spans 516 subjects and about 250K questions.Source: §5.1.2 · p. 52, §5.1.3 · p. 54

5.1.4 Natural Language Understanding; 5.1.5 Reasoning; 5.1.6 Knowledge

• NLU benchmarks range from individual tasks to broad language-understanding suites, while reasoning and knowledge datasets test distinct inference skills and both retained and newly learned knowledge.Reasoning tasks include multi-step, decision, deductive, and mathematical reasoning; knowledge benchmarks assess memory, analysis, learning, and induction, with ALCUNA targeting understanding and association of unfamiliar knowledge.Source: §5.1.4 · p. 55, §5.1.5 · p. 55, §5.1.6 · pp. 55–56

5.1.7 Long Text

• Long-text benchmarks test document processing across tasks and lengths, with InfiniteBench raising average evaluation length to 200K tokens for Chinese and English.Other suites include L-Eval at 4K–60K words, LongEval at 5K–16K tokens, and LongBench’s English, Chinese, and code tasks, generally averaging 5K–15K tokens.Source: §5.1.7 · p. 56

5.1.8 Tool; 5.1.9 Agent

• Tool and agent benchmarks assess API use and tool operations, and agent competence across environments, planning, tool use, and memory.API-Bank provides 53 tools; ToolBench covers single- and multi-step actions across eight subtasks. AgentBench evaluates English scenarios in eight settings, while SuperCLUE-Agent covers three core abilities and ten tasks in Chinese.Source: §5.1.8 · p. 57, §5.1.9 · p. 57

5.1.10 Code; 5.1.11 Out-of-Distribution

• Code benchmarks assess generation and other programming tasks through single-task and multitask designs, while OOD datasets test robustness on unseen tasks after instruction fine-tuning.HumanEvalPack covers six programming languages and three code tasks; BOSS includes five tasks and 20 sub-datasets for OOD assessment.Source: §5.1.10 · pp. 57–58, §5.1.11 · p. 58

5.1.12 Law; 5.1.13 Medical; 5.1.14 Financial

• Legal, medical, and financial benchmarks assess domain-specific knowledge and tasks, with suites varying by language, breadth, and evaluation focus.Legal datasets span Chinese, English, and multilingual settings; integrated Chinese medical benchmarks cover multiple tasks, while FinBen collects 35 datasets across 23 financial tasks and three difficulty levels.Source: §5.1.12 · p. 58, §5.1.13 · p. 59, §5.1.14 · p. 59

5.1.15 Social Norms

• Societal-norm datasets test ethics, bias, toxicity, and safety; SafetyBench (Zhang et al, 2023n) covers seven safety dimensions in Chinese and English, while SuperCLUE-Safety56 uses adversarial and multi-turn challenges.Safety-Prompts (Sun et al, 2023a) covers 13 safety scenarios and prompt attacks, though ChatGPT-generated data can contain errors; TRUSTGPT (Huang et al, 2023d) assesses toxicity, bias, and value consistency.Source: §5.1.15 · p. 60

5.1.16 Factuality

• Factuality datasets assess hallucination through choice-based tests such as FACTOR (Muhlgay et al, 2023), open-ended generation such as FActScore (Min et al, 2023), and targeted questions such as TruthfulQA (Lin et al, 2022).HaluEval (Li et al, 2023e) tests hallucination recognition across Q&A, dialogue, and summarization; FactualityPrompt (Lee et al, 2022) tests genuine and non-genuine prompts. HalluQA (Cheng et al, 2023) targets Chinese models, while FreshQA (Vu et al, 2023) tests evolving knowledge and rejection of false premises.Source: §5.1.16 · pp. 60–61

5.1.17 Evaluation

• LLM-evaluator datasets probe assessment reliability: FairEval (Wang et al, 2023b) tests LLM assessors, PandaLM testset (Wang et al, 2023g) uses human annotations, and LLMEval2 spans 15 tasks and 8 abilities.Source: §5.1.17 · p. 61

5.1.18 Multitask

• English multitask benchmarks range from BIG-Bench’s 95 task types and 204 tasks to HELM’s 73 scenarios and 65 metrics; Chinese benchmarks include CLEVA’s 370K instances across 31 assessment tasks.CLEVA has 11 application and 20 capability tasks; CLiB, LLMEVAL-1, and FlagEval provide additional Chinese or bilingual proficiency and capability assessment.Source: §5.1.18 · pp. 61–62

5.1.19 Multilingual

• Multilingual benchmarks extend traditional NLP tasks across languages: XNLI (Conneau et al, 2018) covers 15 languages, while XTREME (Siddhant et al, 2020) translates four tasks into 40 languages across 12 language families.Source: §5.1.19 · p. 62

5.1.20 Other

• Specialized evaluations cover few-shot Chinese learning, geoscience, IT operations, multi-turn tool use, adversarial robustness, and emotional empathy, including FewCLUE (Xu et al, 2021), GeoBench (Deng et al, 2023), and MINT (Wang et al, 2023e).PromptBench (Zhu et al, 2023) examines seven adversarial prompt types across eight tasks; EmotionBench (Huang et al, 2023a) includes more than 400 scenarios. Other examples include Owl-Bench (Guo et al, 2023b) for bilingual IT operations.Source: §5.1.20 · pp. 62–63

5.2 Evaluation Methods

• Code and model evaluation automate scoring, whereas human evaluation suits open-ended or complex questions but incurs substantial cost and potential subjective bias.Code evaluation compares model responses with annotations using metrics such as accuracy, F1, BLEU, and ROUGE, but is less effective for open-ended questions. Model evaluation uses an LLM to assess responses, reducing manual effort but depending on evaluator performance and potentially diverging from human judgments; combining methods is increasingly common.Source: §5.2 · pp. 63–65

5.3 Distribution Statistics of Evaluation Datasets

• 56.4% of evaluation datasets contain fewer than 10K samples; most fall below 100K, while a few exceed 1M, and dataset counts rose sharply in 2023.Evaluation sets are commonly built manually or compiled from open-source datasets; model-generated data is less common. English datasets predominate, Chinese datasets are also significant, and low-resource languages remain underrepresented. Coverage is broad but long-tailed, with niche domains limited and ancient texts and cultures lacking benchmarks.Source: Figure 20 · p. 65, §5.3 · p. 66

6 Traditional NLP Datasets

• The survey organizes traditional NLP datasets into 15 task categories and notes that they are often converted into instruction-tuning data or used to evaluate LLMs.These datasets were originally designed to train, optimize, and test traditional NLP models; the categories cover a broad range of text-processing tasks.Source: §6 · p. 67

6.1.1 Reading Comprehension

• Reading-comprehension datasets cover selection and judgment, cloze completion, answer extraction, and unrestricted QA, ranging from exam questions and fill-in-the-blank tasks to open-ended reasoning.Some answer-extraction datasets add unanswerable or adversarial questions, while unrestricted QA requires generating answers that cannot be copied directly from the text.Source: §6.1.1 · pp. 68–69

6.1.2 Knowledge QA; 6.1.3 Reasoning QA

• Knowledge QA tests world or domain knowledge without a reference passage, whereas reasoning QA targets logical, multi-step, contextual, and causal inference.Datasets span general and specialist knowledge, including science, law, medicine, and psychology, and probe varied reasoning dimensions.Source: §6.1.2 · p. 69, §6.1.3 · pp. 69–70

6.2 Recognizing Textual Entailment; 6.3 Math; 6.4 Coreference Resolution

• RTE datasets classify premise–hypothesis relations, math datasets range from elementary calculations to proofs, and coreference datasets identify referential links or explain them.RTE variants include three-way labels and adversarial examples; coreference variants include fill-in-the-blank and explanation formats.Source: §6.2 · pp. 70–71, §6.3 · p. 72, §6.4 · p. 72

6.5 Sentiment Analysis; 6.6 Semantic Matching

• Sentiment datasets classify emotional polarity, while semantic-matching datasets assess similarity or correspondence between text sequences.Sentiment examples draw on real-world text; semantic-matching resources include multilingual data and continuous similarity labels, and LCQMC emphasizes question-intent matching.Source: §6.5 · pp. 72–73, §6.6 · pp. 73–74

6.7 Text Generation; 6.8 Text Translation; 6.9 Text Summarization

• Text-generation benchmarks map words to sentences or structured triples to descriptions, translation benchmarks cover multilingual transfer, and summarization datasets span news and other text styles.NLLB provides evaluation benchmarks for over 200 languages; summarization sources include dialogues, reviews, social media, interviews, and emails.Source: §6.7 · p. 74, §6.8 · p. 75, §6.9 · p. 75

6.10 Text Classification; 6.11 Text Quality Evaluation

• Text-classification datasets vary in category granularity, while text-quality evaluation ranges from grammaticality judgments to spelling and grammar correction.AGNEWS and TNEWS use at most 15 categories, CSLDCP uses 67, and IFLYTEK uses 119; Chinese correction datasets also target errors caused by Pinyin input.Source: §6.10 · pp. 75–76, §6.11 · p. 76

6.12 Text-to-Code; 6.13 Named Entity Recognition

• Text-to-code datasets test natural-language-to-code generation, including Python and SQL, while NER datasets identify and classify entities across general and domain-specific texts.NER resources range from benchmarks with four entity classes to one with 18, and include datasets for entertainment, e-commerce, and social media.Source: §6.12 · p. 77, §6.13 · p. 77

6.14 Relation Extraction

• Relation-extraction datasets span dialogue, document-level, and few-shot settings, with distinct annotation, inference, and out-of-distribution evaluation capabilities.Dialogue RE (Yu et al, 2020a) is the first entirely human-annotated dialogue RE dataset and includes 36 relationship types; DocRED (Yao et al, 2019) requires document-level information aggregation, while FewRel (Han et al, 2018) combines few-shot learning with RE and its 2.0 version evaluates OOD capability.Source: §6.14 · p. 78

6.15 Multitask

• Multitask datasets reuse shared text through varied configurations, transformations, and annotations for training or evaluation across NLP task categories.Examples include CSL (Li et al, 2022b) for title prediction, keyword generation, and paper classification; QED (Lamm et al, 2021), which adds explanatory annotations to Natural Questions (Kwiatkowski et al, 2019); and METS-CoV (Zhou et al, 2022), which supports NER and sentiment analysis.Source: §6.15 · pp. 78–79

7.1 Pre-training Corpora

• Pre-training corpora need broader, more balanced language and domain coverage, finer-grained data categories, and investigation of synthetic data and underrepresented vertical domains.The section notes that English and popular web content dominate, while some languages and specialized areas remain poorly represented.Source: §7.1 · pp. 79–80
• Outdated corpora can yield inaccurate or stale generations, motivating dynamic, automatic updates and research into models’ self-learning of new knowledge.Common Crawl continually collects current web data but is mostly English; updating other data types requires reacquisition and preprocessing.Source: §7.1 · p. 80
• Pre-training corpus quality lacks a systematic assessment method, while preprocessing standards are absent and cleaning trades reduced harmful output against weaker discrimination and data diversity.The section leaves unresolved how to compare corpus quality and how extensively to filter harmful or low-quality data.Source: §7.1 · p. 80
• The pre-training corpus ecosystem lacks shared preprocessing, evaluation, release, and maintenance standards.The section describes community-wide coordination and data management as still undeveloped.Source: §7.1 · p. 80

7.2 Instruction Fine-tuning Datasets

• Instruction datasets need finer task and domain labels and greater coverage of low-resource and niche domains, though detailed categories can raise standardization and annotation costs.Quality assessment also lacks universal standards and spans model performance, annotation consistency, bias, timeliness, and subjective review.Source: §7.2 · pp. 81–82
• Instruction dataset repackaging without adequate source and copyright labels raises risks of data leakage and biased behavior, prompting calls for transparency and usage standards.Longpre et al (2023b) provides a dataset audit and data provenance explorer tool.Source: §7.2 · p. 82

7.3 Preference Datasets

• Open preference datasets are scarce, especially for non-English and domain-specific uses; evaluation also needs consistent, multidimensional standards and explanations for preference judgments.The section suggests weak supervision or model-assisted voting and scoring as possible ways to reduce annotation burdens.Source: §7.3 · p. 82

7.4 Evaluation Datasets

• Evaluation datasets need contamination-aware sourcing, balanced and suitably difficult questions, and broader coverage of low-resource domains, languages, multi-turn interaction, and changing knowledge.Static evaluations can become training targets or outdated; model-based scoring and centralized, standardized evaluation frameworks are discussed as directions.Source: §7.4 · pp. 83–84

• The survey categorizes and summarizes LLM datasets across pre-training, instruction fine-tuning, preference, evaluation, and traditional NLP.Source: §8 · p. 84
• It outlines challenges and potential directions for pre-training, instruction fine-tuning, reinforcement learning, and model evaluation datasets.Source: §8 · p. 84
• The survey aims to provide a reference for researchers and practitioners while supporting a robust, standardized dataset ecosystem and continued LLM advancement.Source: §8 · p. 84

• Appendix A gives detailed information on each pre-training corpus discussed in the main text.Source: Appendix A · p. 85

A.1.1 Webpages

• Web corpora range from English-focused C4 variants to multilingual collections such as CulturaX, which covers 167 languages and 6.3T tokens.Other examples include Common Crawl-derived RedPajama-V2, with over 100 billion text documents, and RefinedWeb, the Falcon pre-training dataset.Source: §A.1.1 · pp. 85–86

A.1.2 Languages Texts

• Language-text resources include American and British English corpora and news crawled from online newspapers in 59 languages.The BNC comprises 4,124 texts and over 100M words, with written language accounting for 90% and spoken language 10%.Source: §A.1.2 · p. 86

A.1.3 Books

• Book resources include public-domain collections and crawled e-books; PG-19 selects 28,752 pre-1919 books to avoid international copyright issues.The Toronto Book Corpus contains 11,038 e-books and is not currently publicly available.Source: §A.1.3 · pp. 86–87

A.1.4 Academic Materials

• Academic materials include arXiv preprints and S2ORC, which provides structured full text for 8.1M open-access papers.S2ORC covers 81M English-language academic papers across disciplines and includes metadata, abstracts, and resolved references.Source: §A.1.4 · p. 87

A.1.5 Code

• Code corpora range from six-language BIGQUERY and curated Python data for phi-1 to The Stack’s over 6TB of permissively licensed code in 358 languages.The Stack serves as a pre-training corpus for Code LLMs.Source: §A.1.5 · pp. 87–88

A.1.6 Parallel Corpus

• Parallel corpora include MTP’s 300M aligned Chinese-English pairs and ParaCrawl’s 223M filtered sentence pairs spanning 42 languages.MultiUN draws on United Nations documents in six official languages, with some German versions also available.Source: §A.1.6 · p. 88

A.1.7 Social Media

• Social-media sources include Reddit-derived OpenWebText, which contains 8,013,769 documents after filtering, deduplication, and tokenization.Other listed sources include Reddit, StackExchange Q&A, and WebText, which excludes Wikipedia-related documents.Source: §A.1.7 · pp. 88–89

A.1.8 Encyclopedia

• Encyclopedia sources include Chinese collections such as Baidu Baike and TigerBot-wiki, alongside multilingual Wikipedia.Baidu Baike had accumulated more than 27M entries as of April 2023.Source: §A.1.8 · p. 89

A.2.1 Financial Domain

• Financial pre-training resources include Chinese corpora of financial news, announcements, and reports, while medical resources include Chinese reference entries and regularly updated biomedical literature.Medical-pt has approximately 360K encyclopedia entries and 8,475 textbook entries; PubMed Central comprises approximately 5M articles.Source: §A.2.1 · p. 90, §A.2.2 · pp. 90–91

B.1.3 Collection and Improvement of Existing Datasets

• The appendix also points to further information on other pre-training domains and on collection and improvement of existing instruction datasets.Its multiple-method examples include LIMA-sft’s 1,330 human-selected instructions and Bactrain-X’s 3.5M instructions across 52 languages, with quality dependent on translation accuracy and generated responses.Source: §A.2.3 · p. 91, §B.1.3 · p. 95, §B.1.4 · pp. 95, 97

B.2.6 Other Domains

• The cited passages direct readers to Sections 2.2.3 and 3.3.6 for details, but do not specify in these excerpts what topics those sections cover.Source: §A.2.3 · p. 91, §B.2.6 · p. 101

B.1.1 Human Generated Datasets

• Appendix B details instruction fine-tuning datasets, including human-generated collections such as Aya’s over 204K instances across 65 languages and OASST1’s 161K messages across 35 languages.The Human Generated Datasets subsection also lists Chinese conversational resources, including LCCC’s 6.8M and 12M dialogue versions.Source: Appendix B · p. 91, §B.1.1 · pp. 91–92

B.1.2 Model Constructed Datasets

• Model-constructed instruction datasets include Alpaca’s 52K examples and Self-Instruct’s approximately 52K English instructions expanded from seed instructions.Wizard Evol-Instruct datasets apply four evolutionary stages to 175 human-created seed instructions to increase instruction difficulty and complexity.Source: §B.1.2 · pp. 92, 94, 95

B.2.1 Medical Domain

• Medical instruction collections combine authentic dialogues, model-generated data, and reorganized resources; MedDialog contains 3.4M Chinese and 0.26M English doctor-patient dialogues.Other examples include DISC-Med-SFT’s 465K samples and Huatuo-26M’s 26M medical Q&A pairs.Source: §B.2.1 · pp. 97–98

B.2.2 Code Domain

• Code-domain datasets cover programming instructions, contest problems, and tool use, including ToolBench’s 126K instances built around 16,464 APIs in 49 categories.The legal-domain examples include DISC-Law-SFT’s 403K instructions; its Pair and Triplet subsets target legal reasoning and use of external knowledge, respectively.Source: §B.2.2 · p. 99, §B.2.3 · p. 99

B.2.4 Mathematics Domain

• Mathematics datasets include synthetic arithmetic, word problems, and generated solutions; Goat is limited to addition, subtraction, multiplication, and division.Education examples include Educhat-sft-002-data-osm’s 4.28M Chinese and English dialogues, while TaoLi data comprises 88K instances and may contain model-generated errors.Source: §B.2.4 · p. 100, §B.2.5 · pp. 100–101

C.1 Vote

• Vote-based preference datasets include hh-rlhf’s approximately 170K selected-versus-rejected response pairs and PKU-SafeRLHF’s 362K human-annotated English examples.Other examples include Chatbot Arena conversations and SHP, where user-favored Reddit responses form preference pairs.Source: §C.1 · pp. 101–102

C.2 Sort

• OASST1 pairwise rlhf reward converts OASST1’s human quality ratings into 19K preference examples with responses sorted by preference.Source: §C.2 · p. 102

C.3 Score

• Score-based resources include Alpaca comparison data, where GPT-4 scores responses, and UltraFeedback, whose model ratings cover four response dimensions.Stack-Exchange-Preferences assigns answer scores using upvotes and acceptance; WebGPT uses manual ratings to determine preference scores.Source: §C.3 · pp. 102–103

C.4 Other

• The appendix points to Section 4.1.4 for information on other preference datasets.Source: §C.4 · p. 103

D.1 General

• General benchmarks assess open-ended, conversational, complex-instruction, and broad Chinese- and English-language capabilities, often using model-based scoring.The descriptions include GPT-4 or ChatGPT scoring, code assessment, and manual evaluation.Source: §D.1 · pp. 103–104

D.2 Exam; D.3 Subject

• Exam and subject benchmarks cover admission tests, school examinations, and academic disciplines, with formats and assessment methods ranging from multiple choice and code scoring to expert review.The catalog includes multilingual and multimodal exam questions as well as Chinese and English subject tests across STEM, humanities, and professional fields.Source: §D.2 · pp. 104–105, §D.3 · pp. 105–106

D.4 Natural Language Understanding; D.5 Reasoning

• Natural-language-understanding benchmarks span Chinese and English datasets for classification, matching, entailment, reading comprehension, and related language abilities.Reasoning benchmarks address mathematical, logical, decision, and multi-step reasoning through varied task formats, including few-shot chain-of-thought prompting.Source: §D.4 · pp. 107–108, §D.5 · pp. 108–109

D.6 Knowledge; D.7 Long Text

• Knowledge and long-text evaluations probe expertise across domains and the ability to reason over extended inputs, including retrieval, summarization, question answering, and inference.The long-text datasets described use inputs ranging from 1K to over 100K tokens or words, with InfiniteBench reaching a maximum of 2M.Source: §D.6 · pp. 109–110, §D.7 · pp. 110–111

D.8 Tool; D.9 Agent

• Tool and agent benchmarks test API invocation, retrieval, and orchestration, alongside agent planning, tool use, and memory across varied environments.The agent examples include operating systems, databases, knowledge graphs, web browsing, and shopping; Chinese agent tasks additionally cover planning and memory.Source: §D.9 · pp. 111–112

D.10 Code

• Code benchmarks cover code generation, repair, translation, and documentation, with tasks ranging from function-writing tests to database querying and multi-turn synthesis.Some datasets also test multilingual coding or generation across multiple programming languages.Source: §D.10 · pp. 112–113

D.11 Out-of-Distribution; D.12 Law

• Out-of-distribution benchmarks assess transfer across classic NLP tasks, while legal datasets evaluate legal reasoning and NLP capabilities across languages and application settings.The legal evaluations include tasks such as entity recognition, judgment prediction, document handling, information retrieval, and consultation.Source: §D.11 · p. 113, §D.12 · pp. 113–114

D.13 Medical

• Medical benchmarks assess Chinese and English medical understanding through clinical, exam, question-answering and prompt-based tasks.CBLUE (Zhang et al, 2022) covers five task categories and eight sub-datasets; CMB (Wang et al, 2023c) contains 281K questions, while MultiMedQA (Singhal et al, 2023) combines six public medical datasets with an in-house English Q&A dataset. PromptCBLUE100 converts 16 existing medical NLP tasks into generation tasks using 94 templates.Source: §D.13 · pp. 114–115

D.14 Financial

• Financial benchmarks range from focused language-understanding tasks to broad financial reasoning, forecasting and trading evaluations.FinBen (Xie et al, 2024) integrates 35 datasets across 23 tasks and three difficulty levels; FinancelQ102 contains 7,173 Chinese multiple-choice questions, while FinEval (Zhang et al, 2023d) has 4,661 questions across four domains and 34 subjects. BBF-CFLEB combines six task-specific datasets, and FLUE (Shah et al, 2022) standardizes six English finance NLP datasets.Source: §D.14 · pp. 115–116

D.15 Social Norms

• Social-norm evaluations test stereotypes, safety, toxicity, bias and value alignment across English and Chinese prompts, questions and adversarial dialogues.CrowS-Pairs (Nangia et al, 2020) contains 1,508 instances across nine stereotype categories; SafetyBench (Zhang et al, 2023n) has 11,435 bilingual questions across seven safety dimensions, and Safety-Prompts (Sun et al, 2023a) includes 100K Chinese prompts paired with ChatGPT responses. SuperCLUE-Safety evaluates multi-turn adversarial conversations, while TRUSTGPT assesses toxicity, bias and value alignment.Source: §D.15 · p. 116

D.16 Factuality

• Factuality benchmarks examine factual accuracy and hallucination in generation and question answering, including changing facts, false premises and misleading prompts.FActScore (Min et al, 2023) decomposes generated text into facts and scores those supported by a knowledge source; FreshQA (Vu et al, 2023) tests four answer-change or incorrect-premise categories. HaluEval (Li et al, 2023e) contains 35K English samples spanning Q&A, dialogue and summarization, while HalluQA (Cheng et al, 2023) tests hallucination with 450 Chinese questions.Source: §D.16 · p. 117

D.17 Evaluation

• LLMEval2 contains 2,553 samples across 15 task types and 8 abilities, while FairEval and PandaLM testsets assess LLMs’ response-evaluation capabilities.FairEval uses 80 instructions from the Vicuna Evaluation dataset; the survey cautions that LLMs should be used carefully as evaluators.Source: §D.17 · p. 118

D.18 Multitask

• Multitask benchmarks range from BIG-Bench’s 95 task types and 204 tasks to CLEVA’s 370K Chinese test samples across 11 application and 20 capability tasks.The section also describes benchmarks including BBH, CLiB, decaNLP, FlagEval, HELM, LLMEVAL-1, and LMentry.Source: §D.18 · pp. 118–119

D.19 Multilingual

• XNLI evaluates cross-lingual sentence classification in 15 languages, while XTREME covers four NLP tasks across 40 languages and 20 language families.XNLI has 7.5K evaluation samples per language, with non-English data derived from translation.Source: §D.19 · pp. 119–120

D.20 Other

• For other datasets, the section points readers to Section 5.1.20 for details.Source: §D.20 · p. 120

Appendix E Traditional NLP Dataset Information

• Appendix E provides detailed information on each traditional NLP dataset mentioned in the main text.Source: Appendix E · p. 120

E.1.1 Reading Comprehension

• Reading-comprehension benchmarks range from yes/no and multiple-choice questions to multi-sentence and multi-document reasoning, conversational QA, and multilingual extractive tasks.The datasets also cover Chinese and English comprehension, opinion polarity, robustness, and unanswerable questions.Source: §E.1.1 · pp. 120, 121, 123, 124

E.1.2 Knowledge QA

• Knowledge-QA datasets assess scientific, medical, legal, and commonsense knowledge, with formats including multiple-choice exams, real-world questions, and explanation-supported answers.Examples range from 792,099 Chinese medical Q&A pairs to 11K CommonsenseQA-derived pairs with explanations.Source: §E.1.2 · pp. 125–127

E.1.3 Reasoning QA

• Reasoning-QA benchmarks span causal, logical, physical, qualitative, social, and procedural inference, using tasks such as multi-hop fact integration and predicting process outcomes.PROST contains 18,736 questions across ten physical-reasoning concepts; STRATEGYQA provides steps and evidence for 2,790 strategy-focused questions.Source: §E.1.3 · pp. 127–129

E.2 Recognizing Textual Entailment

• Textual-entailment datasets include adversarial inference, Chinese idiom-pair semantics, translated Chinese NLI, physician-annotated medical inference, and English data spanning ten genres.CommitmentBank evaluates entailment, neutral, and contradiction judgments, while MultiNLI supports cross-genre generalization.Source: §E.2 · p. 129
• Textual-entailment benchmarks include native-Chinese OCNLI, English RTE collections, large-scale SNLI, and WANLI’s hybrid human-and-model construction.OCNLI has 56K pairs and is integrated into CLUE; RTE is included in GLUE and SuperGLUE, while SNLI has 570K pairs. WANLI uses challenging MultiNLI samples, GPT-3-generated instances, automated filtration, and human labeling and refinement.Source: §E.2 · p. 130

E.3 Math

• Math datasets cover elementary word problems, algebra, competitive problems, and proofs, with annotations and designs targeting solution steps, reasoning, or model biases.Ape210K includes solutions, equations, and templates; AQUA-RAT and MATH provide step-by-step explanations or solutions, while NaturalProofs covers propositions and proofs. MathQA introduces an operational representation, and SVAMP tests sensitivity to problem-solving rather than shallow heuristics.Source: §E.3 · pp. 130–131

E.4 Coreference Resolution

• Coreference datasets test pronoun resolution through varied formats, while WiC tests contextual word-meaning identity; WinoGrande expands WSC data and WinoWhy adds rationale selection.CLUEWSC2020 uses expert-annotated sentences from contemporary Chinese literature. DPR uses target pronouns and candidate antecedents; WinoGrande reformulates pronoun disambiguation as fill-in-the-blank, and WinoWhy adds 4,095 constructed reference reasons to WSC data.Source: §E.4 · pp. 131–132

E.5 Sentiment Analysis

• Sentiment benchmarks use product, movie, and social-media reviews, with binary labels; SST-2 adds sentence-level annotations and appears in GLUE (Socher et al, 2013).Examples include EPRSTMT (Xu et al, 2021), IMDB (Maas et al, 2011), and Sentiment140 (Go et al, 2009).Source: §E.5 · pp. 132–133

E.6 Semantic Matching

• Semantic-matching resources span Chinese banking, financial, and question matching, while PAWS targets lexical-overlap traps and PAWS-X extends them to six languages (Zhang et al, 2019; Yang et al, 2019).STSB uses continuous 0–5 similarity scores and covers 10 languages (Cer et al, 2017).Source: §E.6 · pp. 133–134

E.7 Text Generation

• Generation benchmarks cover concept-to-sentence commonsense tasks, structured-record-to-text, and restaurant-domain generation (Lin et al, 2020; Nan et al, 2021; Novikova et al, 2017).WebNLG also maps DBpedia triplets to text (Gardent et al, 2017).Source: §E.7 · pp. 134–135

E.8 Text Translation

• Translation resources include recurring IWSLT tasks across English, French, German, and Arabic, alongside NLLB benchmarks spanning over 200 languages and multi-year WMT datasets (Cettolo et al, 2017; Costa-juss`a et al, 2022).WMT draws on sources including news commentaries and parliamentary records.Source: §E.8 · p. 135

E.9 Text Summarization

• Summarization datasets span news, email, social media, interviews, messaging, and procedural guides, including cross-lingual and professionally annotated collections (Zhang and Tetreault, 2019; Ladhak et al, 2020; Hasan et al, 2021).CNewSum contains 304K long documents with manually written summaries and inferential test annotations (Wang et al, 2021a); WikiLingua covers 18 languages, while XL-Sum covers 45.Source: §E.9 · pp. 135–137

E.10 Text Classification

• Classification benchmarks cover four-category news topics, 67 scientific-literature categories, and 119 app-description categories, alongside multilingual product reviews and Chinese news headlines (Zhang et al, 2015; Xu et al, 2021).AGNEWS contains 497K articles, while TNEWS includes 73.3K headlines (Xu et al, 2020b).Source: §E.10 · pp. 137–138

E.11 Text Quality Evaluation

• Text-quality resources evaluate grammaticality, Chinese spelling correction, and sentence acceptability or fluency, with correction and fluency annotations in YACLC (Wang et al, 2021b).CoLA is a binary grammaticality task, while CSCD-IME targets Pinyin-input errors (Warstadt et al, 2019; Hu et al, 2022b).Source: §E.11 · pp. 138–139

E.12 Text-to-Code

• Text-to-code resources include cross-domain Text-to-SQL datasets and MBPP, a Python benchmark with 974 crowdsourced questions, solutions, and three automated tests per question (Austin et al, 2021).DuSQL includes 200 databases and 23,979 question-SQL pairs across 164 domains (Wang et al, 2020a).Source: §E.12 · p. 139

E.13 Named Entity Recognition

• NER datasets range from finely annotated Few-NERD categories to multilingual OntoNotes and domain-specific e-commerce and emerging-entity resources (Ding et al, 2021; Weischedel et al, 2012; Jie et al, 2019; Derczynski et al, 2017).Few-NERD has 188,200 sentences and 491,711 entities; OntoNotes 5.0 covers three languages and 18 entity categories.Source: §E.13 · p. 140

E.14 Relation Extraction

• Relation-extraction datasets include dialogue-based labeling, document-level inference across multiple sentences, few-shot and out-of-domain evaluation, and 106,264 TACRED instances (Yu et al, 2020a; Yao et al, 2019; Han et al, 2018; Zhang et al, 2017).Dialogue RE annotates 36 relationship types; DocRED contains 5,053 documents.Source: §E.14 · p. 141

E.15 Multitask

• Multitask resources include CSL scientific papers used for title prediction, keyword generation, and classification, and METS-CoV tweets annotated for medical entities and sentiment (Li et al, 2022b; Zhou et al, 2022).CSL incorporates data from 396K papers; METS-CoV manually annotates 10K tweets.Source: §E.15 · p. 141