Paloma: A Benchmark for Evaluating Language Model Fit

Motivation and benchmark

• PALOMA measures language-model perplexity on distributions from 16 sources spanning 546 textual domains, rather than extrapolating trends from a single prescriptive mix of domains.Source: §1 · pp. 1–2

Case studies

• Case studies find that training without heterogeneous sources beyond Common Crawl can produce inconsistent domain perplexities as token counts increase.Figure 1 isolates changes in fit associated with the pretraining corpus while other factors are controlled.Source: §1 · p. 2, Figure 1 · p. 2

Contributions

• The benchmark adds evaluation data for the 100 most popular subreddits and 100 programming languages.These datasets are sampled from DOLMA and target sources common in contemporary language-model training.Source: §1 · p. 2
• PALOMA releases six controlled 1B models and standardized comparison tools covering corpora, hyperparameters, token budgets, decontamination, and training order.The controls support fair comparisons of pretraining data and provide code for reproducing them.Source: §1 · p. 2

Case studies

• Few vocabulary types account for most of the loss measured by perplexity.This finding motivates inspecting fine-grained loss composition rather than relying only on aggregated perplexity.Source: §1 · p. 2

Sampling and evaluation controls

• PALOMA targets at least 100 thousand tokens per domain and 1 million per source, balancing inference cost against metric variance.The subsampling target relies on the simplifying assumption that variance observations from C4 extend to other domains and models.Source: Table 1 · p. 3

Domain construction

• PALOMA defines sources as datasets and domains as metadata-based partitions intended to represent distinct language distributions.The benchmark contains 16 sources divided into 546 domains, using inherited or newly constructed metadata operationalizations.Source: §2 · p. 3

Interpretation and scope

• Domain perplexity should not be treated as direct evidence of the specific human-valued skills represented by that domain.Preprocessing differences can produce large discrepancies, such as perplexity of 391,171 on REDPAJAMA arXiv versus 14 on overlapping DOLMA peS2o data.Source: §2 · p. 3

Limitations

• PALOMA focuses on English and code data, leaving fine-grained fit in other languages for future work.The authors identify this as the benchmark’s largest limitation because current LMs also emphasize these data types.Source: §2 · p. 3

Standard language-model sources

• The benchmark includes standard corpora such as C4, MC4-EN, WIKITEXT-103, PENN TREEBANK, REDPAJAMA, FALCON REFINEDWEB, and DOLMA.These sources span filtered web text, Wikipedia, classic evaluation sets, mixtures of web and reference data, and corpora containing books, papers, code, and Reddit.Source: §2 · p. 4

Fine-grained domain sources

• PALOMA adds fine-grained datasets for internet domains, the top 100 subreddits, and the top 100 programming languages.The subreddit set is ranked by posts, while the programming-language set is ranked by tokens in THE STACK.Source: §2 · p. 4

Disparities between speech communities

• The benchmark includes dialectal English and fringe-community corpora to examine disparities and model exposure to distinct social contexts.TWITTERAAE represents African-American and White-aligned English, while fringe sources include MANOSPHERE, GAB, and 4CHAN CORPORA.Source: §2 · p. 4

Training controls

• PALOMA removes evaluation-data contamination from pretraining data and fixes training-data order to reduce validity and recency confounds.It detects paragraph-level overlaps, removes contaminated documents, and fixes tokenization, sequence length, random seed, and device-invariant data loading.Source: §3 · p. 5

Evaluation controls

• PALOMA equalizes domain sampling by targeting 100,000 tokens per domain, balancing inference cost against perplexity variance.The benchmark selects 1 million tokens for sources and 100,000 tokens for domains based on observed variance trends.Source: §3 · p. 5
• PALOMA recommends fixing model vocabulary and uses bits per byte when vocabularies cannot be matched, improving comparability of likelihood measurements.Vocabulary affects the event space and segmentation of sequences, so byte-level normalization only partially addresses differences.Source: §3 · p. 5
• PALOMA standardizes inference by evaluating documents individually and splitting long documents into disjoint maximum-length inputs.This follows THE PILE’s input format and controls engineering differences in sequence handling.Source: §3 · p. 5

Benchmark comparison

• Table 2 shows PALOMA is the first perplexity benchmark to combine contamination removal, fixed training order, equal domain sampling, fixed vocabulary, and controlled inference format.The comparison positions PALOMA’s guidelines against practices in previous language-model benchmarks.Source: §3 · pp. 5–6, Table 2 · p. 6

Comparability

• These controls enable comparisons of compute-matched performance, compute efficiency, pretraining-corpus quality, and performance scaling.PALOMA measures cost using model parameters and training tokens, while intermediate checkpoints should be matched by completed optimization progress.Source: §3 · p. 6

4 Case studies

• The case studies compare six controlled 1B-parameter baselines trained for approximately 150B tokens on common pretraining corpora.Training controls include decontamination, fixed data order, and a shared tokenizer for comparability.Source: §4 · p. 6, §4.1 · p. 7

4.1 Pretraining Beyond Common Crawl Shows Improved Stability of LM Fit

• Per-source macro averaging gives every domain equal weight, exposing robustness differences that ordinary token-weighted perplexity can obscure.PALOMA reports these macro averages separately by source to make distribution-specific fit easier to interpret.Source: §4.1 · p. 7
• Common Crawl-only baselines exhibit high, sometimes non-monotonic perplexity on specific sources, including 391,171 for C4 on arXiv and 21,652 for FALCON REFINEDWEB on Max.Source: §4.1 · p. 7
• Curated-data baselines maintain relatively stable perplexity gaps during training, whereas Common Crawl-only models more often show irregular gaps across domain difficulty.The PILE is an exception, with erratic below-median gaps on some scientific and programming sources.Source: §4.1 · pp. 8–9

4.2 Common Vocabulary Types Dominate Perplexity

• Five percent of vocabulary types account for roughly 50% of overall perplexity because frequent types contribute high total loss despite lower average loss.Across PALOMA domains, vocabulary IDs and evaluation frequency correlate at Pearson’s r = –0.522±0.087.Source: §4.2 · p. 9, Figure 4 · p. 9

5 Conclusion

• Perplexity benchmarks can use real-world source data and be updated by scraping more material, avoiding the annotation costs of downstream evaluations.They also let researchers from other fields contribute domains without understanding LM architectures.Source: §5 · pp. 9–10
• PALOMA sources correlate with some downstream tasks and anticorrelate with others, so lower perplexity does not universally indicate better downstream performance.The paper calls for experiments across broad scales and data recipes to distinguish useful learning from superficial train-validation overlap.Source: §5 · p. 10

• PALOMA focuses on English and code, leaving fine-grained language-model fit across other languages for future work.The authors identify this scope as PALOMA’s largest limitation.Source: §6 · p. 10
• Perplexity can reflect context length and frequent strings rather than model fit to salient domain features.The authors therefore recommend decomposing surprise over specific vocabulary strings rather than relying only on domain-level aggregates.Source: §6 · p. 10
• PALOMA proposes simple, scalable mitigations for benchmark leakage and subsampling variance, but their efficacy requires further comparison with alternatives.Source: §6 · p. 10
• Its 546 metadata-defined domains are heuristic and may overlap across sources, because PALOMA treats similar domains as fully distinct.The authors call for better ways to identify fine-grained domains and separate authorship from data-collection shifts.Source: §6 · p. 10

Checklist

• The paper reports limitations, potential negative societal impacts, and the challenge of evaluating diverse preferences, with further discussion of personally identifiable information and licenses.Source: Checklist · p. 17
• The paper provides code, data, reproduction instructions, training details, and compute information through its repository, appendices, and main text.Source: Checklist · p. 18
• The authors did not report error bars because multiple pretraining runs were not feasible.Source: Checklist · p. 18
• Existing assets are cited and their licenses are discussed, while new assets are released through the project URL.Source: Checklist · p. 18
• The checklist states that consent discussions were not included because the data came from publicly distributed research datasets.Source: Checklist · p. 18

• No single perplexity evaluation correlates well with all downstream tasks, supporting evaluation across diverse perplexity sources rather than one validation loss.The analysis reports relationships for specific pairs, including m2d2 wikipedia with Arc Challenge and c4-en with Hellaswag.Source: A Downstream Correlation Analysis · p. 19, Table 3 · p. 20
• The correlations should not be overinterpreted because adjacent-checkpoint downstream rankings have only moderate average correlation of 0.513 across the six pretraining experiments.Source: A Downstream Correlation Analysis · p. 19

B Additional Metrics

• Bits per byte normalizes document log likelihood by UTF-8 byte count when model vocabularies differ.Source: B Additional Metrics · p. 20
• Average likelihood per vocabulary type records per-type corpus likelihoods and occurrence counts to expose effects hidden by frequent strings.The metric distinguishes vocabulary types from their corpus occurrences, called tokens.Source: B Additional Metrics · p. 20

C.1.1 Decontamination

• PALOMA requires sub-document contamination checks because document-level splits can miss leaked spans and underestimate perplexity.Its Bloom-filter approach is designed to scale more cheaply than pairwise minHash or memory-intensive suffix-array methods.Source: §C.1.1 · p. 21
• Paragraph matching ignores short or formatting-only spans, and PALOMA does not decontaminate its code sources because repeated short lines make the heuristic unsuitable.Source: §C.1.1 · p. 21
• Removing whole documents containing contaminated paragraphs removes at most 2.753% of documents in the six baseline training datasets.The authors caution that this can disproportionately remove long, frequently quoted documents and may be inappropriate for production models.Source: §C.1.1 · pp. 21–22, Table 4 · p. 21

C.1.2 Data Order

• Fixed training order improves comparability and checkpoint data accounting, requiring deterministic instance ordering with shared tokenization and sequence length.Source: §C.1.2 · p. 22

C.2.1 Subsampling

• Perplexity variance decreases as evaluation samples and training data grow, although the sampling-size relationship is treated as a heuristic that may depend on domain and model properties.The mean is relatively stable across subsample sizes, while variability shrinks with larger evaluation samples and later checkpoints.Source: §C.2.1 · p. 23, Figure 5 · p. 23
• Subsampling uses equal token targets per domain to reduce the risk that sparsely represented domains disappear from evaluation.The benchmark targets at least 100 thousand tokens per domain and 1 million tokens per source where possible, with exceptions for smaller or specially treated datasets.Source: §C.2.1 · p. 23

C.2.2 Vocabulary

• PALOMA recommends a shared vocabulary when possible and BPB when comparing models with different tokenizers, while acknowledging residual comparability issues.The fixed-vocabulary option uses the GPT-NeoX-20B vocabulary with three added DOLMA special tokens; BPB is the alternative for differing vocabularies.Source: §C.2.2 · p. 24

C.2.3 Evaluation Format

• Documents are evaluated individually and split into disjoint maximum-length inputs, because concatenation can alter subsampling-variance behavior and provide cross-document context.Nonoverlapping inference is more efficient than sliding-window inference, while concatenation may function like few-shot conditioning and should be intentional.Source: §C.2.3 · pp. 24–25

D Additional Case Studies

• The benchmark’s additional case studies examine unequal domain scaling, vocabulary-type dynamics, and auxiliary sources removed from the main release because access restrictions prevent rehosting.These analyses extend the controlled baseline comparisons with domain- and type-level diagnostics.Source: D Additional Case Studies · p. 25

D.1.1 Scaling Tokens Seen

• C4 and MC4-EN worsen on 65 and 43 domains respectively with longer training, whereas only six other model-domain pairs deteriorate.The authors note that FALCON REFINEDWEB does not show the same erratic behavior, potentially because its cleaning heuristics preserve different content.Source: §D.1.1 · p. 25
• 1.57x separates the median improvement of the most- from least-improved domains, rising to 1.94x when C4 and MC4-EN baselines are excluded.Improvement is estimated from log-loss change per 10× increase in tokens between approximately 20B and 150B tokens.Source: §D.1.1 · pp. 25–26, Figure 6 · p. 26
• Unequal token-scaling rates can reflect worsening fit, slower improvement, or saturation, and are presented as diagnostics rather than problems requiring correction.A least-improved domain can still have lower absolute perplexity than a most-improved domain.Source: §D.1.1 · p. 26, Figure 7 · p. 27

D.1.2 Scaling Model Parameters

• Larger Pythia models consistently achieve lower macro-averaged perplexity across sources and improve faster per token seen, despite contamination-control limitations.These public models address the other evaluation guidelines but are not controlled for contamination.Source: §D.1.2 · p. 27
• 2.02× separates the median domain-level improvement from increasing model size between the most- and least-improved domains.Unlike some token-scaling results, parameter scaling improves every domain in this analysis, but at substantially different rates.Source: §D.1.2 · p. 27
• The case study concludes that aggregate fit should be decomposed by domain because scale does not improve all domains uniformly.The authors leave the desirability and mitigation of these inequalities for future work.Source: §D.1.2 · p. 27

D.2 Common Vocabulary Types Dominate Perplexity, Others Have Inverse Scaling

• 8.5% to 32.1% of vocabulary types favor Pythia-1B over Pythia-7B across sources, even though Pythia-7B has lower perplexity on every domain.The proportion is largest for TWITTERAAE, and about 10% of high-frequency types still favor the smaller model.Source: §D.2 · p. 28, Figure 10 · p. 29
• Type-level advantages arise through multiple training trajectories, including constant gaps, degradation in the larger model, improvement only in the smaller model, or opposing loss trends.The final pattern resembles inverse scaling reported for other language-model behaviors, but the authors describe this type-level phenomenon as previously unreported.Source: §D.2 · pp. 28–29, Figure 12 · p. 30
• Inverse-scaling prevalence varies by domain: in TWITTERAAE, Pythia-1B is better on 77.6% of favored African American-domain types versus 71.3% of favored White-aligned-domain types.The larger model can therefore retain an advantage on the White-aligned domain while inverse scaling appears on the African American domain.Source: §D.2 · pp. 29–30
• The smaller model’s advantage increases with type ID: Pythia-7B usually wins on high-frequency types, while Pythia-1B wins on many low-frequency types.Perplexity is influenced little by these low-frequency types, so the aggregate metric does not capture the pattern well.Source: §D.2 · p. 28, Figure 11 · p. 30
• These findings show that reporting only aggregated perplexity misses subtler source-, domain-, and vocabulary-level behavior.The broader analysis connects type-level effects to the observation that a few common vocabulary types dominate measured loss.Source: §D.2 · pp. 27, 30

E Evaluation Data Source Details

• Table 5 summarizes 16 research data sources used to construct PALOMA’s language-modeling evaluations.The sources include standard corpora, web and code data, social-media datasets, academic collections, and specialized discourse corpora.Source: E Evaluation Data Source Details · pp. 30–31, Table 5 · p. 31

E.1 Removed sources

• PALOMA includes varied Common Crawl, Wikipedia, academic, code, Reddit, and social-media sources, while two early sources are retained only as auxiliary results because access restrictions prevent rehosting.THE PILE and ICE are therefore excluded from the main PALOMA body but documented separately.Source: E Evaluation Data Source Details · pp. 31, 32, 35

E Evaluation Data Source Details

• The benchmark adds balanced evaluations for the top 100 subreddits and top 100 programming languages, extending coverage to social-media and code domains.Reddit posts remain separate documents rather than reconstructed conversational threads, while code is sampled from DOLMA-contained THE STACK data.Source: E Evaluation Data Source Details · pp. 33–34, Table 11 · p. 34, Table 12 · p. 34

F Reweighting Perplexities

• Domain-level negative log likelihoods can be reweighted by another corpus’s token proportions to estimate perplexity under a different domain distribution.This avoids rerunning inference when the desired target corpus differs from PALOMA’s equal-token-per-domain sampling.Source: F Reweighting Perplexities · p. 36

G Baseline Models

• The six trained baselines share a controlled 1B-parameter architecture and training setup, while Pythia results provide additional comparisons but do not conform to all training guidelines.Pythia uses the same core tokenizer vocabulary apart from three DOLMA special tokens, enabling partially trained-checkpoint comparisons.Source: G Baseline Models · pp. 36–37

H Formatting and Subsampling

• Preliminary evidence indicates that the observed decrease in perplexity variability with more evaluation or training data depends on evaluating documents separately rather than concatenating them.The authors hypothesize that random boundary placement and cross-document influence contribute to the breakdown under concatenated inputs.Source: H Formatting and Subsampling · p. 37, Table 17 · p. 37

• Unequal improvement in language-model fit across domains can reflect different training dynamics.The appendix uses performance curves over the underlying scaling factor to investigate these differences more specifically.Source: I Most and Least Improved Domains · p. 37

Tokens seen

• Token-scaling analysis compares the most and least improved domains using changes from approximately 20B to 150B tokens seen.Figure 13 presents perplexity curves for these domains across baseline models.Source: I Most and Least Improved Domains · p. 37, Figure 13 · p. 38

Model size

• Parameter-scaling analysis compares the most and least improved domains across 85M–805M and 805M–6.4B parameter ranges.Figure 14 presents perplexity curves across model-size comparisons and evaluation sources.Source: I Most and Least Improved Domains · p. 37, Figure 14 · p. 39