Learning to Learn a Language

1 Introduction

• Children bring expectations about what languages can be, whereas language models must infer language regularities from their corpus alone.The passage frames this gap as a difference in prior knowledge, not a fact about language.Source: §1 · p. 3
• PFLM trains on a fresh synthetic language for every sequence, so it must infer the language from a prefix rather than memorize one.The method is meta-learning by construction and approximates amortized Bayesian inference over the prior.Source: §1 · p. 3
• Its recurrent structural causal model prior generates tokens from evolving latent state and reproduces natural text’s Zipfian frequencies, slow entropy-rate convergence, and multiscale mutual-information decay.The prior is constructed, not fit, and shares statistical signatures without sharing a specific script.Source: §1 · p. 3
• 0.9–2.4 bits per byte at one million bytes of context across six Wikipedia languages, down from the uniform rate of 8 bits per byte.Bits per byte falls steadily as the context grows.Source: §1 · p. 3
• With numerals in context, PFLM counts, compares quantities, and adds approximately, matching the number-sense pattern described for children.Source: §1 · p. 3
• PFLM predicts deterministic sequences such as Rudin–Shapiro, Kolakoski, and the prime indicator, whose terms follow generating rules rather than frequencies.Source: §1 · p. 3

2.1 In-context inference over languages

• Next-token prediction is Bayesian inference: the model averages language-specific predictions weighted by how well each language explains the prefix, with noise giving every finite sequence positive prior probability.Source: §2.1 · pp. 3–4, §2.2 · p. 5

2.2 A synthetic language as a recurrent SCM

• Each synthetic language is a sparse recurrent graph with freshly drawn wiring and update rules; quantized observed coordinates emit tokens while unobserved state coordinates act as hidden memory.Source: §2.2 · pp. 4–5

2.3 The prior over languages

• Each language independently samples graph rules, activations, noise, depth, periodic node rates, and an initialization law; power-of-two rates span multiple timescales.Source: §2.3 · p. 5, §2.2 · p. 5

2.4 Model and training

• 300M-parameter PFLM uses a 3:1 stack of gated linear-attention and sliding-window attention layers, trained on 150 billion synthetic tokens across six increasing-context stages.Source: §2.4 · pp. 5–6
• Across byte entropy, gzip compression ratio, bigram Zipf exponent, and active byte count, only the SCM prior’s range overlaps natural-language values on every panel.Source: Figure 3 · p. 6

3.1 Scalar fingerprints

• Only the SCM prior lies within the natural-language band on all four tested statistics, while each data-free baseline misses at least one signature.The comparison uses six Wikipedia languages and four data-free baselines.Source: §3.1 · p. 6, §3 · p. 6

3.2 Trajectory statistics

• The SCM prior’s 10–90% envelope contains every natural-language curve in conditional entropy from n-gram orders 1–8.Its median falls from about 5 bits at n = 1 to under one bit at n = 8.Source: §3.2 · p. 7, Figure 4 · p. 7
• The SCM prior’s median follows natural language’s slow MI decay, and its 10–90% envelope brackets every natural-language curve.The comparison spans distances from 1 to 512 bytes.Source: §3.2 · p. 7, Figure 4 · p. 7
• Fixing every node to update each step leaves scalar fingerprints unchanged but reduces integrated long-range MI by 18%, implicating the slow-node tail.The multi-rate schedule carries dependencies across many steps.Source: §3.2 · p. 7

4 In-Context Learning on Real Data

• The reported evaluations use one model trained only on synthetic-prior samples; they report bits per byte, number-task accuracy, and bits per symbol for digit sequences.Source: §4 · p. 7

4.1 Natural language

• On six Wikipedia languages, bits per byte falls from 8 to between 0.9 and 2.4 at one million bytes of context.The evaluated languages are English, Chinese, Hindi, Arabic, Japanese, and Korean.Source: §4.1 · p. 7
• Bits per byte differs across scripts partly because UTF-8 characters use different byte counts; remaining differences reflect language and corpus predictability.Latin-script characters use one byte, Arabic two, and Devanagari, Chinese, Japanese, and Korean three.Source: §4.1 · p. 7

4.2 Beyond language

• PFLM has the lowest bits per byte in every tested real-data domain against an online bigram, gzip, and PPMd.Table 2 reports means over 524,288-byte windows, with up to 32 windows per domain across its categories.Source: Table 2 · p. 8
• At 8192 examples, magnitude-comparison accuracy falls from 0.97 at a 1/2 ratio to 0.68 at 14/15, while approximate-addition accuracy falls from 0.93 to 0.62.For addition, the compared numbers have the same digit count, so length is not a cue.Source: §4.2 · p. 8, Figure 5 · p. 8
• PFLM reaches 0.04 bits per symbol on Rudin–Shapiro, 0.11 by 100,000 digits on Kolakoski, and 0.28 on the prime indicator after one million digits.On binary digits of π, the cost stays at one bit throughout.Source: §4.2 · p. 8

5 Related Work

• Prior-Fitted Networks apply synthetic-prior training to unseen real data, while natural-language in-context learning remained untested.The recipe had been extended to regression, survival analysis, and tabular prediction.Source: §5 · p. 9
• Unlike a prior distilled from formal languages and followed by English training, PFLM uses generic dynamics and never trains on real text.The cited cognitive-science approach is McCoy and Griffiths.Source: §5 · p. 9
• PFLM’s closest sibling uses a universal-Turing-machine prior over arbitrary programs; PFLM instead targets transfer to a specific unseen natural domain.The comparison is with Grau-Moya et al., which approximates the Solomonoff prior.Source: §5 · p. 9
• Earlier synthetic-text work trains on generated stories or textbooks, while PFLM’s prior shares text statistics rather than text content or syntax.Structured non-linguistic sequences had also transferred to real text through syntactic structure alone.Source: §5 · p. 9
• PFLM provides a test of accounts treating in-context learning as implicit Bayesian inference, using a known prior and a domain absent from training data.In-context learning was formalized as implicit Bayesian inference over a latent task variable.Source: §5 · p. 9

6 Conclusion

• PFLM predicts unseen language, numeral, and deterministic-sequence data, with Wikipedia bits per byte falling as context grows while weights remain fixed.The model is trained on no real data.Source: §6 · p. 9
• PFLM learns to count through carries, compare magnitudes, and add approximately, and predicts Rudin–Shapiro, Kolakoski, and prime-indicator sequences far below chance.These tasks share the inference problem of identifying a structured source from a prefix.Source: §6 · p. 9
• 1 million bytes of context—about 200,000 English words—still yields Wikipedia predictions short of modern models trained on trillions of tokens.The authors identify context length as the current limit on in-context learning.Source: §6 · p. 9

A Prior configuration

• Table 3 lists sampled ranges for every component of the prior.The caption distinguishes per-language from per-node draws.Source: Table 3 · p. 13, A Prior configuration · p. 13
• A per-language range is drawn once for the whole synthetic language.The table caption says these draws fix a value across that language.Source: Table 3 · p. 13, A Prior configuration · p. 13
• A per-node range is drawn independently at each node rather than once for the language.The distinction applies to the sampled ranges listed in Table 3.Source: Table 3 · p. 13, A Prior configuration · p. 13

B Model and training

• The model uses blocks of three gated linear-attention layers followed by one sliding-window layer, SwiGLU MLPs, and a 256-byte vocabulary.The parameter count is measured.Source: Table 4 · p. 13, B Model and training · p. 13
• Training uses six increasing-context stages, each initialized from the previous stage, with fresh prior sequences at every length and 150 billion tokens total.The stages use 20, 60, 40, 15, 10, and 5 billion tokens, respectively.Source: B Model and training · p. 13
• Gated linear-attention layers use FlashQLA kernels, while sliding-window attention uses FlashAttention-3.The cited implementations build on flash-linear-attention and FlashAttention-3, respectively.Source: B Model and training · p. 13

• Each of six natural languages occupies a characteristic script-specific UTF-8 byte range, while prior samples spread across different byte-space regions without sharing a script.The comparison includes ten random prior samples and six natural-language corpora.Source: C Byte-frequency profiles · p. 13, Figure 6 · p. 14

• All Sections 4.1 and 4.2 evaluations use frozen PFLM weights; adaptation comes only through context, without gradient steps.Source: D Evaluation protocol · p. 14
• Corpora are byte streams with a 256-value vocabulary; UTF-8 text and binary symbols map to bytes, so bytes per character vary across scripts.English uses about one byte per character, Arabic about two, and Devanagari, Chinese, Japanese, and Korean about three.Source: D Evaluation protocol · pp. 14–15
• The six Wikipedia languages come from the 20231101 dump; acquisition curves use 32 held-out sequences of 1,048,576 bytes per language, starting at article boundaries.The statistical-signature analysis uses 500 sequences of 4096 bytes per language.Source: D Evaluation protocol · p. 15
• Bits per byte is next-byte surprisal given the prefix, plotted as context grows while model weights remain fixed.Uncertainty bands are defined in each figure caption.Source: D Evaluation protocol · p. 15
• Four deterministic binary sequences are tested—Rudin–Shapiro, Kolakoski, the prime indicator, and π digits—with chance at one bit per symbol.The prime-indicator floor is 0.37 bits at n = 106 for a predictor knowing only local prime density.Source: D Evaluation protocol · p. 15
• Numerical tasks test exact-match counting on comma-separated numerals, plus few-shot comparison and approximate-addition judgments from logits at the answer position.Comparison uses unseen three-digit query pairs; addition queries require the sum to exceed either addend, and each cell has 500 queries at k = 128, 1024, and 8192.Source: D Evaluation protocol · p. 15