PaperlayerView exact revision on arXiv

Source-linked AI summary

Skywork: A More Open Bilingual Foundation Model

Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuanhai Dong, Yanqi Sun, Yifu Chen, Yongyi Peng, Xiaojuan Liang, Shuicheng Yan, Han Fang, Yahui Zhou

arXiv:2310.19341v1cs.CLcs.AI

TL;DR

The growing commercialization of LLMs has coincided with reduced transparency in the open-source community. Skywork-13B uses two-stage training on a segmented corpus, and its reported results include the lowest average perplexity of 9.42 and lowest scores in four listed domains.

  • Problem

    Commercialization has coincided with reduced transparency in the open-source community.

    Source: §1 · p. 1

  • Method

    Skywork-13B uses a two-stage training methodology that leverages a segmented corpus to enhance model capability in specific domains.

    Source: §7 · p. 13

  • Results

    Skywork-13B obtains the lowest average perplexity score of 9.42 and the lowest scores in tech (11.58), movie (21.84), government (4.76), and finance (4.92).

    Source: §4.3.3 · p. 10

  • Takeaways & Limitations

    The authors present their transparent development approach as a resource for researchers and describe training-progress monitoring as a practical solution for tracking model improvement.

    Source: §7 · p. 13

    It remains unclear whether two-stage training produces a model on par with or superior to one-stage training on a mixed corpus.

    Source: §6 · p. 13

Abstract

from arXiv · show

In this technical report, we present Skywork-13B, a family of large language models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both English and Chinese texts. This bilingual foundation model is the most extensively trained and openly published LLMs of comparable size to date. We introduce a two-stage training methodology using a segmented corpus, targeting general purpose training and then domain-specific enhancement training, respectively. We show that our model not only excels on popular benchmarks, but also achieves \emph{state of the art} performance in Chinese language modeling on diverse domains. Furthermore, we propose a novel leakage detection method, demonstrating that test data contamination is a pressing issue warranting further investigation by the LLM community. To spur future research, we release Skywork-13B along with checkpoints obtained during intermediate stages of the training process. We are also releasing part of our SkyPile corpus, a collection of over 150 billion tokens of web text, which is the largest high quality open Chinese pre-training corpus to date. We hope Skywork-13B and our open corpus will serve as a valuable open-source resource to democratize access to high-quality LLMs.

1 Introduction

The report responds to declining openness and transparency in LLM development by presenting Skywork-13B and documenting its development. It pairs the model with training artifacts and data, plus a method for detecting in-domain data use.

1 Introduction

  • The report identifies diminishing openness and withheld reproduction information as problems that hamper progress in the field.Source: §1 · p. 1
  • Skywork-13B is a 13-billion-parameter bilingual model family whose Base version has state-of-the-art Chinese language modeling capability and whose Chat version is fine-tuned for conversation.The family was trained on more than 3.2 trillion English and Chinese tokens.Source: §1 · pp. 1–2
  • The report discloses training-process and data-composition details and releases intermediate checkpoints for studying capability development and supporting researchers’ specific use-cases.Source: §1 · p. 2
  • The released corpus portion exceeds 150 billion tokens and is, to the authors’ knowledge, the largest open Chinese pre-training corpus to date.Source: §1 · p. 2
  • The authors introduce a method for detecting the level of in-domain data usage during training and release relevant data to facilitate experiment reproduction.Source: §1 · p. 2

2 Methodology

Skywork-13B uses a segmented corpus for general-purpose pre-training followed by mixed-corpus STEM enhancement. The report monitors training with language-model loss across distribution-specific validation sets, alongside benchmark checks.

2.1 Two Pre-training Stages

  • The model is first trained on SkyPile-Main, then continually pre-trained on a SkyPile-STEM and SkyPile-Main mix to add STEM knowledge while addressing catastrophic forgetting.SkyPile-STEM is an exercise-and-solution subset; its complement is SkyPile-Main.Source: §2.1 · p. 2
  • Separating the stages lets the authors assess general-purpose versus targeted pre-training, since STEM data overlap with supervised data related to popular benchmarks.The report also describes the general-purpose-only model as an alternative to targeted enhancement for applications not requiring STEM competencies.Source: §2.1 · p. 2

2.2 Training Progress Monitoring

  • Training loss may overstate performance when data are reused or duplicated, while intermediate-checkpoint benchmarks can be variable, insensitive to gradual gains, and costly to run.The report also notes that weak models may not follow instructions well, making benchmark results an imperfect reflection of learning progress.Source: §2.2 · p. 3
  • The authors track language-model loss across reserved validation sets with distinct distributions; English-web validation loss closely correlates with average scores on five English tasks during pre-training.Figure 1 compares English-web validation loss with the average metric for BoolQ, PIQA, Winogrande, TriviaQA, and RACE.Source: §2.2 · p. 3, Figure 1 · p. 3
  • The report describes constructing multiple validation sets as relatively effortless and calculating validation loss as straightforward, reducing monitoring overhead.This approach supports evaluation of language-model performance across varied domains.Source: §2.2 · p. 3

3 Pre-training

Skywork-13B uses SkyPile, a large bilingual corpus, and a two-stage pre-training design. Its tokenizer and LLaMA-inspired architecture are tailored to English-Chinese modeling.

3.1 SkyPile Corpus

  • SkyPile contains over 6 trillion tokens, and more than 150 billion tokens of web text have been released.The corpus is built primarily from publicly accessible web pages and processed for structure, distribution, duplication, and quality.Source: §3.1 · pp. 4–5

3.2 Training Data Composition

  • Stage-1 data comprises 49.8% English, 39.6% Chinese, 8.0% code, and 2.4% other languages.The authors generally limit repetition of any source to five times.Source: §3.2 · p. 5

3.3 Tokenizer

  • The tokenizer extends LLaMA’s vocabulary with 8,000 single-character Chinese tokens and 25,000 frequent Chinese multi-character words, reaching 65,536 tokens.Numbers are split into individual digits, with bytes used for unknown UTF-8 characters.Source: §3.3 · pp. 5–6

3.4 Architecture

  • Skywork follows LLaMA-inspired transformer modifications, including RoPE, RMSNorm with pre-normalization, and SwiGLU.The authors report that a 7-billion-parameter LLaMA model trained on 200 billion tokens achieves lower training loss than GPT; Skywork also uses a deeper, narrower network than LLaMA2-13B.Source: §3.4 · p. 6, Figure 2 · p. 6

3.5 Infrastructure

Skywork-13B is trained through general-purpose pre-training followed by STEM-oriented continual pre-training. Across these stages, the model’s reported Chinese language-modeling and STEM benchmark performance improves, while Stage-2 maintains validation-loss stability.

3.5 Infrastructure

  • Skywork-13B’s training used DP256, PP2, and ZeRO-1, reaching 1,873 tokens per GPU per second and 56.5% MFU over 39 days.The training cluster comprised 512 A800-80G SXM GPUs across 64 nodes.Source: §3.5 · pp. 6–7

3.6 Training Details

  • The report defines two pre-training stages: general-purpose training on SkyPile-Main, then STEM-oriented continual pre-training on SkyPile-STEM.Both stages use a standard autoregressive language-modeling objective with a 4,096-token context length.Source: §3.6 · p. 7

3.6.1 Stage-1 Pre-training

  • Stage-1 trains from scratch on SkyPile-Main for over three trillion tokens in two sessions, covering 0–2T and 2–3T tokens.The second session uses a constant 6e-5 learning rate; performance continues improving, though at a decelerating pace.Source: §3.6.1 · p. 7, Figure 4 · p. 9
  • By the end of Stage-1, Skywork trails comparable open LLMs in English language modeling but surpasses them in Chinese language modeling.The report says the Chinese advantage also holds across test sets from diverse domains.Source: §3.6.1 · p. 8

3.6.2 Stage-2 Pre-training

  • Stage-2 training maintains language-modeling validation-loss stability while enabling STEM knowledge transfer; CEVAL and GSM8K scores show similar improvement trends.The Stage-2 mix begins with 10% SkyPile-STEM and rises to 40%; the authors report roughly 130 billion tokens total.Source: §3.6.2 · pp. 8–9

4 Evaluation

Skywork-13B is evaluated against similarly sized open models on standard benchmarks and on Chinese language-modeling tests from diverse domains. It leads on CEVAL, MMLU, GSM8K, and overall domain-test perplexity, but not on CMMLU.

4.1 Baselines

  • The authors compare Skywork-13B with open models of similar size, including LLaMA-13B, LLaMA2-13B, Baichuan-13B, Baichuan2-13B, Xverse-13B, and InternLM-20B.Model details include training-token quantities and supported primary languages.Source: §4.1 · p. 9, Table 4 · p. 9

4.2 Benchmark Evaluation

  • Skywork-13B scores highest on CEVAL (60.6), MMLU (62.1), and GSM8K (55.8), while Baichuan2-13B leads CMMLU (62.0).CEVAL, CMMLU, and MMLU use 5-shot accuracy; GSM8K uses 8-shot accuracy.Source: §4.2 · pp. 9–10

4.3.1 LM as a solution to benchmark overfitting

  • The report motivates language-modeling evaluation as an alternative to static human-annotated benchmarks, whose test sets can become overfit and yield misleading results.It notes that language-modeling test sets can be sampled and revised using newly published natural text.Source: §4.3.1 · p. 10

4.3.2 Construction of diverse LM testsets

  • The Chinese language-modeling evaluation uses domain-labeled text collected from varied websites, with samples published after September 1, 2023.The date cutoff is intended to ensure samples were not included during evaluated models’ pre-training.Source: §4.3.2 · p. 10

4.3.3 Results

  • Skywork-13B has the lowest average perplexity, 9.42, and the lowest reported scores in tech (11.58), movie (21.84), government (4.76), and finance (4.92).The report attributes its language-modeling performance to training-corpus quality and says it outperforms even the larger InternLM-20B and Aquila2-34B.Source: §4.3.3 · p. 10

5 Discussion

The discussion examines how pre-training on task-related data can improve task performance, while weighing its effects on general language modeling and fair benchmarking. It also considers loss comparisons as a way to investigate possible benchmark-data exposure.

5 Discussion

  • For foundation-model task performance, pre-training on in-domain data is described as more efficient than general-purpose web-text training.The discussion frames foundation models as general-purpose backbones whose quality is commonly evaluated through downstream task performance.Source: §5.1 · p. 11

5.1 Effect of pre-training on in-domain data

  • 1B tokens of in-domain data let a 0.5T checkpoint surpass the 3T Skywork-13B backbone on CEVAL and GSM8K, but increased held-out English and Chinese web-text loss.The reported improvement concerns benchmark performance; the accompanying higher losses indicate a trade-off in language modeling.Source: §5.1 · pp. 11–12, Table 7 · p. 12

5.2 Pre-training on in-domain data: a common practice?

  • Most evaluated models had nearly identical GSM8K training- and test-split losses, while ChatGLM3-6B, Baichuan2-13B, Qwen-7B/14B, and Aquila2-34B had lower training-split losses.The authors postulate that these models may have been substantially pre-trained on GSM8K training data or similar data.Source: §5.2 · p. 12, Table 8 · p. 13
  • A lower GSM8K test loss than reference-set loss appears as an anomaly that the authors identify for further study.The reference set consists of GPT-4-generated samples designed to mimic GSM8K.Source: §5.2 · p. 12, Table 8 · p. 13

5.3 Pre-Training or Supervised Fine-Tuning?

  • Because targeted pre-training can improve specific tasks, the authors warn that benchmark scores may overstate capability on unseen tasks and make model comparisons unfair.They connect this issue to the growing availability of supervised or in-domain data, including data generated by LLMs, for use during pre-training.Source: §5.3 · pp. 12–13

6 Limitation

The authors identify two unresolved limitations: the comparative effectiveness of two-stage versus mixed-corpus training, and the scope of language-modeling loss as a proxy for downstream performance.

6 Limitation

  • It remains unclear whether two-stage training can match or outperform one-stage training on a mixed corpus.The report says further investigation is needed to compare these approaches.Source: §6 · p. 13
  • Language-modeling loss or perplexity on one data distribution may not predict performance on other tasks, and the correlation can vary across distributions and tasks.Evaluation depends on the specific test-data distribution, of which the authors note there are infinitely many possibilities.Source: §6 · p. 13

7 Conclusion

The conclusion presents Skywork-13B’s transparent development and two-stage training as contributions to open LLM research. It also positions the work as an appeal for renewed fairness, transparency, and sharing in the NLP community.

7 Conclusion

  • The report presents its transparent development as a resource for researchers and its two-stage training and monitoring methods as practical contributions.The authors say the segmented-corpus approach enhances capability in specific domains and that monitoring training progress helps track model improvement.Source: §7 · p. 13
  • The authors call for renewed fairness, transparency, and sharing of ideas in the NLP community.They hope Skywork-13B will encourage openness and cooperation in developing future models.Source: §7 · pp. 13–14
  • The conclusion frames the work as more than the creation of a new LLM.It links the project to a broader appeal for community openness and cooperation.Source: §7 · p. 14

A Details on GPT-7B vs. LLaMA-7B Experiment

The appendix describes a preliminary controlled comparison of GPT and LLaMA architectures using two 7B models.

A Details on GPT-7B vs. LLaMA-7B Experiment

B Preliminary Experiments on Distributed Training

Preliminary distributed-training experiments found that DP256 with PP2 and ZERO-1 achieved the best throughput for both tested model architectures.

B Preliminary Experiments on Distributed Training

  • DP256 with PP2 and ZERO-1 achieved the best throughput for both LLaMA-13B and Skywork-13B.The report describes these as preliminary results across distributed-training configurations.Source: B Preliminary Experiments on… · p. 17

C More Benchmark Results

The appendix reports additional English benchmark results and describes the datasets used to evaluate question answering, commonsense reasoning, and reading comprehension.

C More Benchmark Results

D Details on LM Test Sets

The report describes independently collected, domain-categorized Chinese language-model test sets, drawing on articles and user posts from several websites.

D Details on LM Test Sets

  • A daily crawl of articles and user posts from selected Chinese websites supplies independent, domain-categorized language-model test sets, collected through a process distinct from SkyPile construction.Source: D Details on LM Test Sets · p. 17
  • The technology test set uses AI-related articles from 36kr, a source covering startups, technology, and business trends in China.Source: D Details on LM Test Sets · p. 17
  • The movie test set uses user-written reviews from Douban, a Chinese social platform for movie, book, and music content.Source: D Details on LM Test Sets · p. 17
  • The government test set draws on People’s Daily news, described as formal Standard Mandarin with an authoritative tone.Source: D Details on LM Test Sets · pp. 17, 19
  • The game test set draws on Gcores, a Chinese digital media platform covering video games, technology trends, and geek culture.Source: D Details on LM Test Sets · p. 19
  • Finance test data comes from Sina’s finance section, while general-news data comes from Jiemian News.Sina covers financial topics; Jiemian covers areas including politics, economy, culture, technology, finance, and lifestyle.Source: D Details on LM Test Sets · p. 19
  • Table 12 provides details on CEVAL benchmark results.Source: Table 12 · p. 20
  • Figure 6 charts Skywork-13B performance during Stage-1 pre-training on BoolQ, PIQA, Winogrande, TriviaQA, RACE, and CMRC.Source: Figure 6 · p. 21
Loading 2310.19341v1…