PaperlayerView exact revision on arXiv

Source-linked AI summary

Entropy Law: The Story Behind Data Compression and LLM Performance

Mingjia Yin, Chuhan Wu, Yufei Wang, Hao Wang, Wei Guo, Yasheng Wang, Yong Liu, Ruiming Tang, Defu Lian, Enhong Chen

arXiv:2407.06645v3cs.LGcs.CL

TL;DR

Sample-wise quality selection can miss redundancy and contradiction among training examples, complicating data selection under limited computational resources. The paper proposes ZIP, a multi-stage greedy method guided by an entropy law relating performance to compression ratio and training loss. Experiments report that ZIP outperforms other selection approaches across backbones and alignment settings.

  • Problem

    Most data-selection methods assess individual sample quality, leaving the combinatorial effects of redundancy and inconsistency among samples insufficiently addressed.

    Source: §1 · p. 2

  • Method

    The paper formulates an entropy law relating model performance to compression ratio and training loss, then uses ZIP’s multi-stage greedy selection to prioritize low-compression data.

    Source: §1 · pp. 2–3

  • Results

    ZIP outperforms other data-selection approaches on Mistral-7B and Llama-3-8B and yields the best downstream performance in preference alignment.

    Source: §5.1.2 · p. 7, §5.2.2 · p. 9

  • Takeaways & Limitations

    The reported results support modeling combinatorial effects when selecting data for LLM alignment across backbones and alignment stages.

    Source: §5.2.2 · p. 9, §6 · p. 11

    The theoretical account rests on assumptions about data correctness, diversity, and consistency, and illustrates its information argument with non-independent question-answer pairs.

    Source: §3 · p. 4

Abstract

from arXiv · show

Data is the cornerstone of large language models (LLMs), but not all data is useful for model learning. Carefully selected data can better elicit the capabilities of LLMs with much less computational overhead. Most methods concentrate on evaluating the quality of individual samples in data selection, while the combinatorial effects among samples are neglected. Even if each sample is of perfect quality, their combinations may be suboptimal in teaching LLMs due to their intrinsic homogeneity or contradiction. In this paper, we aim to uncover the underlying relationships between LLM performance and data selection. Inspired by the information compression nature of LLMs, we uncover an ``entropy law'' that connects LLM performance with data compression ratio and first-epoch training loss, which reflect the information redundancy of a dataset and the mastery of inherent knowledge encoded in this dataset, respectively. Through both theoretical deduction and empirical evaluation, we find that model performance is negatively correlated to the compression ratio of training data, which usually yields a lower training loss. Based on the findings of the entropy law, we propose a quite efficient and universal data selection method named \textbf{ZIP} for training LLMs, which aim to prioritize data subsets exhibiting a low compression ratio. Based on a multi-stage algorithm that selects diverse data in a greedy manner, we can obtain a good data subset with satisfactory diversity. Extensive experiments have been conducted to validate the entropy law and the superiority of ZIP across different LLM backbones and alignment stages. We also present an interesting application of entropy law that can detect potential performance risks at the beginning of model training.

1 Introduction

Selecting useful LLM training data is difficult under computational constraints, and sample-level quality assessments miss redundancy and inconsistency between samples. The paper links compression ratio to model performance and introduces ZIP, a greedy data-selection method that prioritizes low-compression subsets.

1 Introduction

  • Automatic data selection is needed because trial-and-error selection is cumbersome and often impractical under limited computational budgets.Not all collected data is useful for teaching LLMs.Source: §1 · p. 2
  • Sample-level quality evaluations overlook redundancy and inconsistency between samples, so individually strong examples can form a suboptimal training set.The paper contrasts such sets with lower-quality but more diverse samples that may convey more information.Source: §1 · p. 2
  • The paper formulates an entropy law relating model performance to data compression ratio, provided selected samples’ overall quality and consistency remain unchanged.The analysis is motivated by the view of autoregressive language modeling as information compression.Source: §1 · p. 2
  • ZIP uses a multi-stage greedy strategy to select heterogeneous, low-compression data, and experiments report its superiority over quality-based baselines across backbones and alignment stages.The paper also describes using the entropy law to detect potential performance risks at the beginning of training.Source: §1 · p. 3

2 Related Works

Related work connects language modeling with information compression, describes LLM alignment data and its selection, and identifies sample-wise quality assessment as a limitation of existing selection approaches.

2.1 Large Modeling and Information Compression

  • An optimal lossless compressor’s expected coding length is expressed through the corpus distribution, and estimating that distribution with a language model yields the cross-entropy loss used to train LLMs.This establishes the paper’s connection between language modeling and information compression.Source: §2.1 · p. 3

2.2 Alignment of Large Language Models

  • LLM alignment adapts models to end tasks using instruction datasets, with prior work constructing or expanding such datasets through crowd-sourcing, self-instruction, and dataset combination.Alignment includes supervised fine-tuning and reinforcement learning with human feedback.Source: §2.2 · p. 3

2.3 Data selection for LLM alignment

  • Prior alignment-data selection largely scores individual samples using heuristics or model-based assessments, while this paper focuses on dataset quality measured through compression metrics.Examples include response-length heuristics, IFD and SuperFiltering loss measures, ChatGPT quality scoring, #InsTag tagging, and DEITA’s hybrid scores.Source: §2.3 · pp. 3–4

3 Entropy Law: Connecting Model Performance with Data Compression

The entropy law models LLM performance as related to compression ratio, training loss, and data consistency. Its deductions connect higher compression ratios with usually worse performance and interpret higher loss at a fixed ratio as evidence of lower consistency.

3 Entropy Law: Connecting Model Performance with Data Compression

  • The theoretical account treats performance as influenced by data correctness and diversity, while poor consistency or weak mastery of encoded information may also limit performance.The authors state these relationships as assumptions for their analysis.Source: §3 · p. 4
  • Compression ratio measures pre- versus post-compression size, training loss indicates how hard data is to memorize, and consistency reflects next-token uncertainty given prior context.The authors say average loss over a small number of steps in the first epoch can provide an indicative value without overfitting.Source: §3 · p. 4
  • For non-independent question-answer pairs, the paper states that joint mutual information is less than the sum of the pairs’ separate mutual information, narrowing the knowledge learned.This example illustrates how inconsistent answers to similar questions can constrain the learned knowledge.Source: §3 · p. 4
  • The entropy law states that model performance is correlated with data compression ratio and training loss.The analysis also treats average sample-level quality as approximately constant when selection does not substantially change it.Source: §3 · p. 5
  • With data consistency held constant, higher compression ratio is associated with usually worse performance; at a fixed ratio, higher training loss indicates lower consistency and potentially more limited learned knowledge.The paper distinguishes this performance connection from Pandey’s data-dependent scaling law.Source: §3 · p. 5

4 ZIP: Lightweight Data Selection for LLM Alignment

ZIP approximates low-compression data selection with a multi-stage greedy procedure. It first identifies information-dense candidates, then accounts for redundancy with the selected set and among candidates themselves.

4 ZIP: Lightweight Data Selection for LLM Alignment

  • ZIP uses a multi-stage greedy algorithm to approximate low-compression selection without searching the full space of sample combinations.The stages are repeated until the selected dataset reaches the desired size.Source: §4 · p. 5

4.1 Global Selection

  • Global selection initializes each sample’s redundancy state from its compression ratio and retains the K1 lowest-scoring candidates for local selection.The method treats high intra-sample redundancy as a sign that a sample may have poor global diversity.Source: §4.1 · p. 5

4.2 Local Coarse-grained Selection

  • Coarse-grained local selection recomputes compression ratios after adding each candidate to the selected set, then keeps the K2 candidates with the lowest scores.This stage narrows the candidate pool while accounting for relations with the already selected data.Source: §4.2 · p. 5

4.3 Local Fine-grained Selection

  • Fine-grained local selection repeatedly adds the candidate with the lowest compression ratio when combined with the local set, yielding samples diverse from both that set and the selected dataset.The process continues until the overall data budget is reached; the method is model-free and can use multiple threads.Source: §4.3 · p. 6

5 Experiments

Experiments evaluate ZIP in SFT and RLHF and test the entropy law’s relationship to model performance, compression ratio, and training loss. ZIP leads the reported selection comparisons, while entropy-law measurements also flag potential performance risks during incremental data updates.

5.1.1 Setup

  • SFT uses a 300K-sample pool and an approximately equal token budget across selection methods, then evaluates fine-tuned Mistral-7B and Llama-3-8B on MT-bench.The token budget is set using ZIP’s 10,000 selected samples, after which other methods select data to match its token count.Source: §5.1.1 · pp. 6–7

5.1.2 Results

  • ZIP yields the best reported downstream performance in both SFT comparisons and RLHF preference alignment.The SFT comparison covers Mistral-7B and Llama-3-8B; the RLHF comparison uses Llama-3-8B.Source: Table 1 · p. 7, §5.1.2 · p. 7, Table 2 · p. 9
  • ZIP selects shorter samples without the lengthy-sample bias reported for DEITA and SuperFiltering, and takes the least selection time except Random using CPU-only computation.The token-length distributions of DEITA and SuperFiltering are described as severely skewed from the original distribution.Source: §5.1.2 · p. 8, Figure 2 · p. 8
  • Selected datasets receive highly similar quality scores, while DEITA’s selected data contains 2.7 times ZIP’s token count under the compared setting.The authors argue that controlling data budgets by token count gives methods a more comparable compute budget.Source: §5.1.2 · p. 8

5.2.1 Setup

  • For RLHF, the study selects approximately equal token counts from around 60K chosen-rejected pairs and uses KTO for preference alignment, evaluated on MT-bench.The baselines are Random and Score, which selects pairs with the highest chosen-scores.Source: §5.2.1 · p. 9

5.3 Empirical Validation of Entropy Law

  • Across the Mistral-7B and Llama-3-8B analyses, lower data compression ratios typically accompany better performance and usually higher training loss.The authors interpret lower compression ratios as indicating more knowledge in the data, which is more challenging for models to absorb.Source: Figure 3 · p. 9, §5.3 · p. 10, Figure 4 · p. 10
  • Random, Cluster, and Perplexity show similar performance when their compression ratios and training losses are comparable, supporting the entropy law’s potential as a performance criterion.The paper proposes that this criterion could guide LLM training.Source: §5.3 · pp. 10–11
  • In incremental updates, rising loss and compression ratio in version x5 flag potential performance degradation, later confirmed by post-training evaluation.For versions x1 through x4, decreasing compression ratios support an expectation of improved performance, assuming data quality does not significantly decay.Source: Figure 5 · p. 10, §5.3 · p. 11

6 Conclusion

The paper frames data selection through compression and proposes an entropy law relating model performance, data compression ratio, and training loss. It presents ZIP as a lightweight, versatile selection method and reports its effectiveness across LLM backbones and alignment stages.

6 Conclusion

  • The entropy law connects model performance with data compression ratio and training loss, grounding data selection in a data-compression perspective.The paper also reports empirical analysis supporting the law.Source: §6 · p. 11
  • ZIP selects data with nearly the lowest compression ratio and is described as model-free, content-agnostic, lightweight, and versatile.The paper reports that experiments demonstrate its effectiveness and efficiency across LLM backbones during SFT and RLHF.Source: §6 · p. 11
  • The entropy law is proposed as a criterion for predicting LLM performance on specific data.Source: §6 · p. 11

A Derivations of joint mutual information of two QA pairs

The derivation expands the joint mutual information of two question-answer pairs and bounds it by the sum of their separate mutual informations. Equality holds when the answers are independent, with the corresponding independence condition for the questions.

A Derivations of joint mutual information of two QA pairs

B Training Details

Experiments used a platform with 64 Intel Xeon Gold CPU cores, two high-performance GPUs, and 500GB memory. Data was formatted with model-specific conversation templates, while ZIP, SFT, and RLHF settings are specified.

B Training Details

  • Experiments ran on 64 Intel Xeon Gold 6326 CPU cores, two high-performance GPUs, and 500GB memory.The training code used the Axolotl open-source framework.Source: B Training Details · p. 14
  • Mistral-7B data used the Vicuna-style template and Llama-3-8B data used the Llama-3 template; samples exceeding the maximum input length were truncated.SFT data was packed to speed up training.Source: B Training Details · p. 14
  • ZIP used K1=10000, K2=200, and K3=100; SFT used batch size 128, four epochs, sequence length 2048, and warm-up ratio 0.1.SFT learning rates were 4e-6 for Mistral-7B and 1e-5 for Llama-3-8B; KTO used learning rate 1e-6 and batch size 128.Source: B Training Details · p. 14

C Token length distribution of more backbones

Figure 6 depicts the token-length distribution of data selected for Llama-3-8B, which is described as similar to the Mistral-7B distribution.

C Token length distribution of more backbones

D Hyper-parameter sensitivity

The section tests ZIP’s three efficiency hyperparameters and shows that their settings can limit combination-aware selection, causing performance to fall or plateau.

Perceived sample number in global selection K1

  • As K1 rises, performance initially improves, but setting it to 20,000 degrades performance as unupdated samples’ compression ratios may be underestimated.K1 controls how many samples are updated in global selection; a small value can reduce ZIP to individual compression-ratio selection.Source: D Hyper-parameter sensitivity · p. 14, Figure 7 · p. 15

Data pool size of local selection K2

Data budget of local selection K3

  • K3 shows a trend similar to K1: large values reduce selection to individual compression ratios, while small values can underestimate some inferior samples’ ratios.K3 controls how many samples are selected from the K2 samples; small values prompt more frequent compression-ratio updates.Source: D Hyper-parameter sensitivity · p. 15, Figure 7 · p. 15
Loading 2407.06645v3…