Language Model Decomposition: Quantifying the Dependency and Correlation of Language Models

• The paper identifies a gap: despite many LM variants differing in data, multilingual support, size, objectives, architecture, attention, and sequence length, their relationships lacked mathematical study.Source: §1 · p. 1
• It models an LM as a vector-valued function whose sequence embedding is obtained by mean-pooling the last layer’s token embeddings.Source: §1 · p. 1
• Language Model Decomposition (LMD) represents one LM as a linear combination of other LMs and provides a framework for analyzing their linear dependency.Source: §1 · p. 1
• The paper defines a goodness-of-fit metric based on LMD to quantify LM dependency and correlation.Source: §1 · p. 1
• The paper argues that linear dependence indicates redundancy, while more independent LMs could support distilling knowledge from diverse models and combining them into more powerful models.Source: §1 · p. 1

2.1 Notations and Definitions

• The framework defines LM linear dependence through matrix-valued coefficients, extending vector and function linear dependence to model outputs.A set is dependent when one LM is zero or expressible as a linear combination of the others.Source: §2.1 · p. 2

2.2 Language Model Decomposition

• LMD approximates a target LM as a linear combination of basis LMs by minimizing expected squared residual error.The optimization is convex; when the basis covariance is rank-deficient, infinitely many optima may exist and the minimum-norm solution is selected.Source: §2.2 · p. 2
• Closed-form LMD solutions use a Moore–Penrose inverse, with regularization providing a unique optimum and improved numerical stability.The unregularized solution assumes full rank for uniqueness; rank deficiency instead yields a minimum-norm optimum.Source: §2.2 · pp. 2–3

2.3 Quantitative Measure of Dependency and Correlation

• R2 measures how well a target LM is approximated by basis LMs, with larger values indicating stronger linear dependency.R2 compares residual and total variation and lies between 0 and 1.Source: §2.3 · p. 3
• 91% “correlation” among BERT and its variants indicates substantial linear redundancy under the proposed framework.The section’s measures quantify how well target LMs are approximated by basis LMs.Source: §2.3 · p. 3
• The group correlation measure aggregates each LM’s dependency on the remaining models, while pairwise correlation symmetrizes directional dependencies.Pairwise R2 is asymmetric, whereas ρ is symmetric and bounded in [0, 1].Source: §2.3 · p. 3

3.1 Experiments

• The study evaluates twelve BERT and BERT-like encoders on English Wikipedia and BooksCorpus using sequence lengths from 16 to 512 tokens.It samples 512,000 training sequences and 51,200 test sequences, and reports R2 and ρ on the test data.Source: §3.1 · pp. 3–4, §3.2 · p. 4

3.2 Results

• Distilled models are highly correlated with their original full models, whereas multilingual models show lower correlation with other language models.The proposed explanation is that distillation transfers knowledge from full models, while multilingual training may encode knowledge beyond a single language.Source: §3.2 · p. 4
• Figure 1 visualizes symmetric pairwise correlation among the twelve models at text sequence length T = 512.The pairwise correlation is computed on English Wikipedia and BooksCorpus.Source: Figure 1 · p. 4, §3.2 · p. 4
• 91.25% group correlation indicates that the twelve language models are highly linearly dependent and contain redundancy.R2 is around 90% for all models at T = 512.Source: §3.2 · p. 4
• XLNet and ELECTRA have lower dependency than most models, especially for shorter sequences, consistent with their distinct pretraining objectives.They use Permutation Language Modeling and Replaced Token Detection rather than the Masked Language Modeling used by most other models.Source: §3.2 · p. 4
• Shorter sequence lengths reduce R2, which the authors attribute to greater variation in mean-pooled sequence embeddings.The reported explanation is that larger variation makes target embeddings harder to approximate using other models as bases.Source: §3.2 · p. 4

4 Conclusion and Future Work

• LMD formalizes linear combination and linear dependency among language models and provides a metric for quantifying their relationships.The framework represents one model using other models as bases.Source: §4 · pp. 4–5
• Figure 2 reports multi-model dependency R2(ui, {u−i}) and group correlation ρ({ui}n) on English Wikipedia and BooksCorpus.The figure summarizes the dependency and correlation measures used in the experiments.Source: Figure 2 · p. 5
• The experiments report 91% correlation among BERT and its variants, suggesting redundancy in state-of-the-art pretrained language models.The paper describes these results as preliminary evidence for using LMD to analyze relationships among language models.Source: §4 · p. 5
• Future work asks whether linear dependency can improve pretraining or fine-tuning and whether R2 relates to language-model performance.It also asks how to learn complementary representations, how dependency changes after fine-tuning, and how multilingual models relate to monolingual ones.Source: §4 · p. 5

• The experiments study encoder LMs similar to BERT, leaving encoder-decoder, decoder, pre-BERT, and domain-specific LMs for future investigation.Source: §5 · p. 5
• The study uses Wikipedia and BooksCorpus, while evaluation on a neutral corpus not used for pre-training is identified as worthwhile.Source: §5 · p. 5
• The mean-pooled final-layer sequence embedding makes LMD model agnostic but leaves layer-wise and token-level correlation for future work.Source: §5 · p. 5

Pairwise Dependency of Language Models

• Pairwise analysis uses asymmetric dependency R2(ui, uj), while symmetric correlation ρ(ui, uj) provides a complementary relationship measure.Source: A Details for Experiment Results · p. 7

Dependency and Correlation of Multiple Language Models

• Table 2 reports exact numbers for dependency R2(ui, {u−i}) between language models, defined in Eq (9), while Figure 2 presents the visualization.Source: A Details for Experiment Results · p. 7
• The experiments provide exact multi-model dependency values in Table 2 alongside their visualization in Figure 2.Source: A Details for Experiment Results · p. 7

• The raw Wikicorpus experiment uses the same twelve language models, with 128,000/12,800/12,800 train/validation/test samples and λ = 10−6.Source: B Additional Experiment Results · p. 7

Pairwise Dependency of Language Models

• On raw English Wikicorpus, pairwise dependency R2(ui, uj) increases with sequence length, making shorter-sequence embeddings harder to approximate from basis LMs.Source: B Additional Experiment Results · p. 7

Dependency and Correlation of Multiple Language Models

• The multi-model analysis reports dependency R2(ui, {u−i}) and group correlation ρ({ui}n) in Figure 5 and Table 3.Source: B Additional Experiment Results · p. 7, Table 3 · p. 8, Figure 5 · p. 10
• The raw English Wikicorpus results are similar to those obtained on English Wikipedia and BooksCorpus.Source: B Additional Experiment Results · p. 7

• Each model is treated in turn as the target, with the remaining eleven models serving as basis models for LMD.Source: B Additional Experiment Results · pp. 8, 10

Pairwise Dependency of Language Models

• Pairwise dependency on raw English Wikicorpus is visualized by Figure 4, with target models on the y-axis and basis models on the x-axis.Source: Figure 4 · p. 9