PaperlayerView exact revision on arXiv

Source-linked AI summary

Adam's Law: Textual Frequency Law on Large Language Models

Hongyuan Adam Lu, Z. L., Victor Wei, Zefan Zhang, Zhao Hong, Qiqi Xiang, Bowen Cao, Wai Lam

arXiv:2604.02176v3cs.CL

TL;DR

The paper addresses limited evidence about whether textual frequency affects LLMs, especially when equivalent inputs use different expressions. It proposes TFL, TFD, and CTFT to estimate frequency and apply it to prompting and fine-tuning. Across math reasoning, machine translation, commonsense reasoning, and agentic tool calling, the authors report effective results and positive relationships with textual frequency.

  • Problem

    Textual frequency is understudied in LLMs, despite evidence of its relevance to human cognition and differences in LLM performance across meaning-equivalent prompts.

  • Method

    The framework estimates sentence-level frequency from online corpora, optionally refines it through story-completion generations, and trains in increasing frequency order.

  • Results

    Across math reasoning, machine translation, commonsense reasoning, and agentic tool calling, experiments suggest the textual-frequency framework is effective, with outputs positively related to frequency.

  • Takeaways & Limitations

    The paper supports preferring high-frequency inputs for prompting and fine-tuning, and combining textual frequency with curriculum learning to improve final performance.

  • Takeaways & Limitations

    Textual Frequency Distillation incurs computational costs, although it avoids requiring access to closed-source LLM training corpora.

Abstract

from arXiv · show

While textual frequency has been validated as relevant to human cognition in reading speed, its relatedness to Large Language Models (LLMs) is seldom studied. We propose a novel research direction in terms of textual data frequency, which is an understudied topic, to the best of our knowledge. Our framework is composed of three units. First, this paper proposes Textual Frequency Law (TFL), which indicates that frequent textual data should be preferred for LLMs for both prompting and fine-tuning. Since many LLMs are closed-source in their training data, we propose using online resources to estimate the sentence-level frequency. We then utilize an input paraphraser to paraphrase the input into a more frequent textual expression. Next, we propose Textual Frequency Distillation (TFD) by querying LLMs to conduct story completion by further extending the sentences in the datasets, and the resulting corpora are used to adjust the initial estimation. Finally, we propose Curriculum Textual Frequency Training (CTFT) that fine-tunes LLMs in an increasing order of sentence-level frequency. Experiments are conducted on our curated dataset Textual Frequency Paired Dataset (TFPD) on math reasoning, machine translation, commonsense reasoning and agentic tool calling. Results show the effectiveness of our framework.

1 Introduction

The paper investigates whether textual frequency matters when LLM inputs preserve meaning but use different expressions. It proposes TFL, TFD, and CTFT to estimate and exploit sentence-level frequency for prompting and fine-tuning.

  • Figure 1 illustrates selecting higher-frequency paraphrased prompts, estimating sentence frequency from word frequency, and a toy framework example.
  • Previous work studies data quality, quantity, difficulty, and length, but the preferred textual frequency of training data remains overlooked.
  • Paraphrases preserve meaning while changing language expression, motivating investigation of which expressions are most useful under limited training and prompting budgets.
  • Textual Frequency Law recommends higher-frequency data over lower-frequency paraphrases with the same meaning for both prompting and fine-tuning.
  • Textual Frequency Distillation enhances online frequency estimates by using story completion to collect generations from LLMs whose training corpora are inaccessible.
  • Curriculum Textual Frequency Training fine-tunes LLMs with data ordered from lower to higher sentence-level frequency.

2 Prior Works

Prior work connects textual frequency with human neural responses, semantic similarity, and LLM prediction behavior, while also using paraphrases for evaluation, contamination analysis, and data augmentation. These findings motivate studying frequency differences among meaning-preserving paraphrases.

  • High-frequency words generally evoke stronger neural responses in human reading tasks.
  • High-frequency target words have higher semantic similarity with their nearest-neighbor words.
  • Prior studies use paraphrases to create diverse references, evaluate contamination, and augment training data across language tasks.
  • The paper asks which same-meaning paraphrases are more useful when computational budgets limit training and prompting.

3 Proposed Approach

The approach selects higher-frequency paraphrases for prompting and fine-tuning, estimates sentence frequency from corpora, optionally refines it through model-generated data, and orders training from lower to higher frequency.

  • Textual Frequency Law: Textual Frequency Law selects the highest-frequency paraphrase for prompting and fine-tuning when meaning is preserved.Sentence frequency is estimated from word-level frequencies in an arbitrary text corpus.
  • Textual Frequency Distillation: Textual Frequency Distillation queries LLMs with story-completion prompts to create a distilled dataset for refined frequency estimation.The resulting dataset is denoted D′, and the refined estimate is used to select the highest-frequency expression.
  • Textual Frequency Distillation: The distilled frequency is optional because obtaining it from LLM-generated data is relatively computationally expensive, while the original estimate remains effective alone.
  • Curriculum Textual Frequency Training: Curriculum Textual Frequency Training orders training instances from lower to higher sentence-level frequency for each epoch.Unlike paraphrase selection, CTFT does not require training instances to be paraphrases.
  • Textual Frequency Paired Dataset: TFPD pairs high- and low-frequency sentences while retaining the original datasets’ ground-truth answers.The dataset covers math reasoning, machine translation, commonsense reasoning, and tool calling.
  • Textual Frequency Paired Dataset: GPT-4o-mini generates 20 paraphrases per sentence, after which the lowest- and highest-frequency candidates are human-checked for equivalent meaning.Samples are retained only when the annotators validate that the compared sentences have the same meaning.

4 Experimental Setup

The experiments evaluate textual-frequency effects across LLMs, languages, tasks, and training-order baselines, using established task-specific metrics and prompting templates.

  • Metrics: Translation experiments use chrF, BLEU, and COMET, while math reasoning uses accuracy.COMET supports 37 of the 100 studied languages.
  • Models: Experiments use closed- and open-source LLMs, including GPT-4o-mini and DeepSeek-V3, with additional translation baselines.DeepSeek-V3 is described as a 671B-parameter mixture-of-experts model.
  • Baselines: The study compares CTFT with reverse high-to-low-frequency fine-tuning and easy-to-hard curriculum learning based on Max Dependency Tree Depth.
  • Languages: Prompting experiments sample 100 FLORES-200 languages, more than half of which are classified as relatively low-resource.CTFT experiments use Kabuverdianu, Kikuyu, Pangasinan, and Standard Latvian.
  • Math Reasoning: Figure 2 reports higher math-reasoning accuracy for the high-frequency TFPD partition than for the low-frequency partition.The figure also tracks models correct on both partitions through the high-frequency ∩low-frequency category.

5 Results

Across math reasoning, machine translation, commonsense reasoning, and analysis experiments, high-frequency textual partitions generally improved or maintained LLM performance. TFD and CTFT contributed additional gains, while textual frequency showed weak association with complexity and strong association with translation performance in some settings.

  • Math Reasoning: 8.0–8.26 percentage-point gains improved accuracy across DeepSeek-V3, GPT-4o-mini, and Llama3.3-70B-Instruct on math reasoning.DeepSeek-V3 increased from 63.55% to 71.54%, GPT-4o-mini from 60.70% to 68.70%, and Llama3.3-70B-Instruct from 80.49% to 88.75%.
  • Math Reasoning: High-frequency inputs preserved correct low-frequency answers while improving samples that were initially incorrect.The reported intersection analysis found maintained performance for originally correct samples and improvements concentrated among originally incorrect samples.
  • Machine Translation: 99 of 100 language pairs improved in BLEU on DeepSeek-V3, including 63 by more than 1 point, 31 by more than 3 points, and 12 by more than 5 points.The pattern was consistent across BLEU, chrF, and COMET for DeepSeek-V3 and GPT-4o-mini; degradations were below 1 point.
  • Machine Translation: 29.96% BLEU improvement on pag_Latn was obtained by CTFT over high-frequency fine-tuning without CTFT.The BLEU score increased from 3.7781 (+0%) to 4.9102 (+29.96%), and CTFT produced the best metric in all 8/8 reported comparisons.
  • TFD Analysis: TFD improved performance, with 100% of language pairs better on DeepSeek-V3 COMET scores, and greater gains when more distillation data were used.The ablation removed TFD and observed a performance drop; the data-volume analysis reported increasing improvement with more TFD data.
  • Frequency Analysis: Textual complexity had very weak correlation with frequency, while high-frequency prompts performed better in most restricted-complexity bins.Only the 50%–55% bin favored low-frequency prompts on BLEU and chrF, and it contained 21 samples.

6 Conclusions

The paper presents a three-unit textual-frequency framework for LLMs and evaluates it across several tasks. Its experiments and analyses support using high-frequency inputs in prompting and training, including when inputs are not paraphrases.

  • Framework and Scope: The framework combines Textual Frequency Law, Textual Frequency Distillation, and Curriculum Textual Frequency Training across four LLM tasks.The evaluated tasks are math reasoning, machine translation across hundreds of language pairs, commonsense reasoning, and agentic tool calling.
  • Conclusion: High-frequency inputs are supported for both LLM tuning and training, and can be combined with curriculum learning to improve final performance.The conclusion reports extensive experimental analysis supporting the effectiveness of the textual-frequency framework.
  • Conclusion: Final LLM outputs were positively related to textual frequency even when compared inputs were not paraphrases.The paper presents this analysis as further support for the proposed framework.

Limitations

The framework avoids requiring closed-source LLM training corpora by estimating frequency through story completion, but that workaround introduces computational costs.

  • Limitation: Story-completion-based frequency estimation adds computational costs, although it avoids obtaining closed-source LLM training corpora.The paper describes access to those training corpora as often unrealistic.

Ethical Statement

The supplied passages primarily document the paper’s ethics statement and supporting experimental materials rather than a substantive ethical analysis. The authors report using established datasets and acknowledge risks from offensive content generated by end-to-end pretrained LLMs.

  • Ethics: The datasets are described as well-known and widely used, with preprocessing that used no external textual resource.The authors report no known ethical issue from the datasets or preprocessing.
  • Ethics: End-to-end pretrained LLMs may generate offensive context, and generated content does not reflect the authors’ views.The statement characterizes these risks as commonly known for such models.

A Scope and Proof Strategy

The proof formalizes the Textual Frequency Law: among paraphrases, higher sentence-level frequency tends to yield lower autoregressive NLL under stated assumptions. It derives token-level rank–loss behavior using Zipf’s law, then lifts it to sentences while explicitly limiting strict ordering in the distributional tail.

  • Scope: The central claim is that a higher-frequency paraphrase tends to incur lower NLL under cross-entropy-trained language models.The claim concerns text sequences with the same meaning and is stated as a tendency under the proof’s assumptions.
  • Proof strategy: The proof first relates token-level NLL to frequency rank under Zipf’s law, then extends the result to sentence-level frequency while accounting for marginal–conditional gaps.The sentence-level step introduces a sentence-frequency measure and an explicit contextual discrepancy term.
  • Definitions and assumptions: Sentence frequency is defined as the geometric mean of marginal token frequencies, deliberately ignoring word order and inter-token dependencies.This unigram-style measure is intended to capture vocabulary-frequency differences between paraphrases.
  • Token-level result: The token-level result is semi-log linear: ideal NLL has slope s and intercept C as a function of ln r, within rank-dependent error ε(r).Here r is token frequency rank, with smaller ranks denoting more frequent tokens.
  • Token-level result: Strict token-level monotonicity is guaranteed only when the Zipf-induced rank gap exceeds the combined approximation errors.For similar-frequency tail tokens, the condition can fail because the rank gap shrinks while approximation quality may worsen.

E.3 Sentence-Level Results

The sentence-level theorem bounds average NLL by negative log sentence frequency plus approximation and contextual errors. It yields a sufficient condition for higher-frequency paraphrases to have lower loss, while emphasizing that the condition is conservative and error-dependent.

  • Sentence-level theorem: Sentence-level NLL is approximately equal to negative log sentence frequency, with total error bounded by ¯εx + ηx.The bound combines averaged token-level approximation error with marginal–conditional discrepancy.
  • Monotonicity condition: A higher-frequency paraphrase has lower NLL when its log-frequency advantage exceeds the approximation and contextual error bounds for both paraphrases.This is a sufficient condition, not a necessary one.
  • Caveats: The sufficient condition may hold even when its worst-case bound fails because token errors can cancel and averaging across tokens can tighten the effective error.For high-frequency paraphrases, contextual discrepancy may further reduce actual sentence loss.
  • Caveats: Whether the condition is satisfied depends on the number and frequency gap of changed tokens, model approximation quality, and marginal–conditional discrepancy.Paraphrases that share most function words may differ only modestly in sentence frequency.
  • Empirical connection: Empirical validation is reported to show the law across settings, suggesting that the relevant error terms are often small enough for the condition to be effective.The theorem itself supplies the analytical framework, while the empirical claim concerns practical behavior.

F Discussion: From Loss Ordering to Task Performance

The paper distinguishes its formal result—loss ordering—from the stronger hypothesis that higher-frequency paraphrases improve downstream task performance. It motivates that hypothesis for prompting and fine-tuning but identifies the connection as empirical rather than theorem-level.

  • Formal result versus performance: The formal theorems establish lower NLL for higher-frequency paraphrases, whereas better task performance remains an empirically supported extension.The paper cites higher math-reasoning accuracy and higher BLEU/chrF in machine translation as examples of the stronger claim.
  • Prompting: For prompting, lower input NLL is interpreted as the model assigning higher probability to an input region with better-calibrated internal representations.The proposed interpretation connects frequent inputs with representations shaped by more similar training examples.
  • Fine-tuning: For fine-tuning, higher-probability inputs are proposed to provide more stable gradients and reduce catastrophic-forgetting risk because they are closer to the pre-training distribution.These are presented as motivations for the performance connection rather than as part of the formal theorem.

H Limitations

The theoretical framework is established under explicit assumptions, but several assumptions and scope boundaries constrain its guarantees. The paper distinguishes provable loss–frequency relationships from empirical downstream-performance evidence.

  • Theoretical assumptions: Assumption 2 is empirically motivated but not implied by cross-entropy minimization, and its guarantees weaken for low-frequency tokens.Cross-entropy controls P-weighted expected loss rather than per-token log-domain error.
  • Theoretical assumptions: Contextual discrepancy ηx lacks a general data-independent bound because it depends on sentence context and learned conditional distributions.Estimating it would be model- and data-specific.
  • Measurement scope: The sentence-frequency measure is a unigram approximation that may miss commonness when paraphrases differ substantially in syntax or length.It ignores word order and inter-token dependencies.
  • Interpretation boundary: Loss ordering does not formally imply task-performance ordering; higher accuracy or BLEU remains empirically supported rather than proven by the framework.The formal result concerns lower NLL for higher-frequency paraphrases.
  • Measurement scope: The proof assumes semantic equivalence between paraphrases, although practical paraphrasing can introduce subtle meaning shifts.A formal semantic-similarity treatment is outside the theorem’s scope.
  • Supported conclusion: The framework concludes that higher-frequency paraphrases can have provably lower loss only when frequency differences exceed the error bounds.The sufficient condition is conservative, while empirical evidence indicates it is often effectively satisfied.
Loading 2604.02176v3…