Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks

1 Introduction

• CLM pre-training learns world knowledge and general capabilities but does not optimize for linguistic competence, the capacity to comprehend and interpret diverse linguistic phenomena.The introduction characterizes linguistic competence as distinct from general capabilities.Source: §1 · p. 1
• The introduction likens surface-pattern mimicry without underlying linguistic scaffolding to rote learning in humans.It describes LMs as reproducing patterns without grasping generative rules.Source: §1 · p. 1
• The framework is motivated by the view that structured input can support representations beyond surface-level co-occurrence.The authors target data-efficient acquisition of morphological, syntactic, and semantic knowledge.Source: §1 · p. 1
• L2T integrates language learning tasks with standard CLM, converting raw text into structured input-output pairs without external supervision.The framework is evaluated at 500M and 1B parameters.Source: §1 · p. 1
• L2T improves BLiMP performance by up to 11.3% (2.8% on average), retains general performance, and accelerates linguistic-competence acquisition.These are the introduction’s stated contributions.Source: §1 · p. 1

2 L2T: Language Learning Tasks

• L2T comprises 14 language learning tasks, each mapping a raw text segment to a structured input-output pair.The input is linguistically perturbed or queried; the output is restored or analyzed.Source: §2 · p. 2
• Tasks span character, word, sentence, and discourse levels, targeting subword features, structural inference, coherence, and ambiguity resolution.Examples include Typo, Shuffle, and Fill Middle.Source: §2 · p. 2
• Character-level tasks such as Typo target subword features and are intended to discourage surface-level matching while enhancing morphological awareness.The rationale draws on error-correction work in human learners.Source: §2 · p. 2
• Word- and sentence-level reordering tasks are intended to promote structural inference over sequential statistics.Discourse-level completion tasks use longer context to support global coherence and ambiguity resolution.Source: §2 · p. 2
• The task mixture is presented as structural scaffolding that complements world knowledge acquired through standard CLM.The paper contrasts multi-task learning with CLM’s single-objective training.Source: §2 · p. 2

3.1 Pre-training Data Scenarios

• Both scenarios use a 100B-token training budget, which the authors describe as exceeding compute-optimal thresholds for the model sizes studied.The corpora are derived from English FineWeb-Edu.Source: §3.1 · p. 2
• The Disjoint scenario uses approximately 36B raw tokens and 64B L2T tokens from mutually exclusive subsets of an initial 100B-token sample.The mixture is subsampled to meet the 100B-token limit and tests a setting with high document diversity.Source: §3.1 · p. 2
• The Shared scenario uses the same 42B source tokens for raw-text CLM and L2T generation, testing structural stimulation without adding unique documents.The combined training budget is 100B tokens.Source: §3.1 · p. 2

3.2 Training Configuration

• The study trains from scratch at 500M and 1B parameter scales and computes loss on all tokens.The setup uses a 2,048-token sequence length and batch size of 256.Source: §3.2 · pp. 2–3

3.3 Evaluation Framework

• BLiMP evaluates linguistic competence across 12 phenomena and 67 datasets; general benchmarks cover reading comprehension, commonsense reasoning, language modeling, and ReCoRD.BLiMP comparisons use zero-shot log-likelihoods of minimal pairs; general benchmarks test whether L2T is complementary to CLM on raw text.Source: §3.3 · p. 3

4.1 Linguistic Competence

• 80.2 (500M) and 80.8 (1B) versus Raw’s 78.6 and 79.0 in Disjoint; in Shared, L2T leads Raw by 2.8 and 2.3 points.These are the overall BLiMP scores for the two scales in Disjoint and the reported L2T-over-Raw margins in Shared.Source: §4.1 · p. 3
• L2T improves six phenomena across subfields, with Island effects gaining 6.9 points (1B; Shared) to 11.3 points (1B; Disjoint).The paper reports that nearly all L2T tasks contribute to the Island-effect gain and relates the result to tasks across granularity.Source: §4.1 · p. 3
• L2T offers no improvement for determiner-noun agreement or ellipsis, and individual tasks do not improve Filler-Gap performance.The authors suggest that Filler-Gap may require targeted discourse-level objectives.Source: §4.1 · pp. 3–4
• L2T outperforms Raw across linguistic subfields from 5B tokens; at that point, its average advantage ranges from 3.3 (Syntax) to 6.5 (Semantics) at 500M.At 1B, the reported 5B-token advantage ranges from 1.0 (Syntax) to 4.5 (Semantics). Gains continue and level off around 50B tokens.Source: §4.1 · p. 4, Figure 2 · p. 4

4.2 General Benchmarks

• In Disjoint general benchmarks, the average difference is a 0.87-point drop at 500M and a negligible 0.07-point difference at 1B.The authors describe performance as competitive when the model retains access to many unique documents.Source: §4.2 · p. 4, Table 3 · p. 4
• In Shared general benchmarks, L2T averages a 0.15-point improvement at 500M but a 1.38-point drop at 1B, including a 4.2-point drop on ARC.The authors attribute the scale-dependent pattern primarily to commonsense-reasoning tasks and describe a tension between structural induction and factual reinforcement.Source: §4.2 · p. 4, Table 3 · p. 4

• Knowledge-intensive benchmark performance is sensitive to the ratio of raw text.Source: §5 · p. 4
• +5.4% for 500M and +3.0% for 1B on Raven Progressive Matrices in Disjoint; Ratio Effect is 0.870 for 1B L2T vs. 0.797 for 1B Raw in Disjoint.Source: Table 4 · p. 5, §5 · p. 5
• The combined L2T framework surpasses most single-task models on both BLiMP and general benchmarks.Source: §5 · p. 5
• Nine single-task objectives outperform Raw, while Space and Masked Char underperform it; Space loses up to 33 points in Morphology.One largely preserves general performance but provides insufficient structural signal for linguistic gains.Source: §5 · p. 5, Figure 3 · p. 5
• L2T improves zero-shot alignment with human typicality gradients, with results suggesting semantic categorization rather than simple word-level associations.Source: §5 · p. 5
• L2T models outperform Raw baselines in linguistic competence as early as 5B tokens; direct comparison with supervised augmentation remains an open research direction.Source: §5 · p. 5

• L2T integrates language-learning tasks alongside standard next-token prediction.Source: §6 · p. 5
• By requiring information extraction and restructuring, the tasks stimulate faster and improved linguistic competence development beyond rote learning.Source: §6 · p. 5
• Future work will extend L2T to multilingual settings to investigate learning behavior across languages.Source: §6 · p. 5

Limitations

• Current tasks focus largely on sentence-level local dependencies and do not explicitly address broader phenomena; discourse-level and cross-sentence tasks could capture long-range dependencies.Source: Limitations · p. 6
• Evaluation stops at 500M and 1B; larger-scale effects may differ, and the Shared 1B reasoning drop suggests scaling needs careful raw-text/L2T balance.Source: Limitations · p. 6
• Each configuration has one training run; consistency across two scales and data regimes is offered as evidence of robustness.Source: Limitations · p. 6
• Individual-task analysis uses 25B tokens, covering a maximal-development window but not comprehensively profiling the full pre-training duration.Source: Limitations · p. 6

A.1 Pre-training and Linguistic Competence

• Prior analyses report failures on distant rhetorical relations (Waldis et al., 2024) and phrase-level noun annotation (Cheng and Amiri, 2025); the authors attribute these shortcomings to CLM's lack of structural scaffolding.Source: §A.1 · p. 12

A.2 Enhancing Linguistic Competence of LMs

• Research specifically targeting decoder-based LMs remains limited; L2T targets broad linguistic abilities from scratch without auxiliary models or external knowledge.Source: §A.2 · p. 12

A.3 Self-Supervised Objectives and Data Transformation for Pre-training

• Unlike approaches relying on external models or task-specific datasets, L2T applies predefined transformation rules to raw text to target linguistic competence during pre-training.Source: §A.3 · p. 13

B Details on L2T: Language Learning Tasks

• L2T introduces 14 tasks that convert raw text into structured input-output pairs, ranging from local character and word dependencies to sentence syntax and discourse coherence.Figure 4 groups the tasks by character, word, sentence, and discourse granularity.Source: B Details on L2T: Language Learning… · p. 13, Figure 4 · p. 14

B.1.1 Character Count (Char Count)

• Char Count asks for total characters to encourage fine-grained processing, while Masked Char reconstructs text after character masking to foster character-level and morphological processing.Masked Char examples replace a percentage of characters with mask tokens.Source: §B.1.1 · p. 13, §B.1.2 · p. 13

B.1.3 Whitespace Insertion (Space)

• Space restores whitespace to stripped text to train word boundaries, while Typo corrects synthetic character errors to target character and word recognition.The Typos are generated by replacing a random percentage of characters.Source: §B.1.3 · p. 13, §B.1.4 · p. 14

B.2.1 Last Phrase Prediction (Last)

• Last selects a concluding phrase using sentence structure and context, while Masked Word reconstructs masked words to support vocabulary knowledge and contextual inference.The Last task chooses the segment after the final stop word.Source: §B.2.1 · p. 14, §B.2.2 · p. 14

B.2.3 Random Word Replacement (Random)

• Random replaces words with random vocabulary tokens, Shuffle restores the order of shuffled words, and Token Type counts occurrences of word categories.These tasks target noise robustness, syntax and local dependencies, and classification by linguistic category, respectively.Source: §B.2.3 · p. 15, §B.2.4 · p. 15, §B.2.5 · p. 15

B.3.1 Sentence Deletion (Deletion)

• Deletion removes or identifies an anomalous sentence to support relevance discrimination and discourse coherence, while Reordering restores the order of randomly permuted sentences.Reordering targets discourse structure, logical flow, and sentence-level coherence.Source: §B.3.1 · p. 15, §B.3.2 · p. 15

B.4.1 Second Passage Completion (Fill Middle)

• Fill Middle generates a missing passage, Half generates a text’s latter half, and One continues from a single-word prefix to target long-range coherence and generation structure.The tasks respectively emphasize multi-passage structure, narrative flow, and generation from minimal context.Source: §B.4.1 · p. 16, §B.4.2 · p. 16, §B.4.3 · p. 16

C.1 Pre-training Data Construction

• The data pipeline segments documents into sentence-complete chunks, applies one of the 14 tasks, formats input-output pairs, then packs and shuffles them with raw-text samples.Chunks are approximately 512 tokens; the mixture preserves raw text alongside transformed samples.Source: §C.1 · p. 17

C.2 Implementation and Training Details

• Token Type labels words as stopwords, digits, or content words after punctuation cleaning and prioritized checks; punctuation counting uses a regular-expression pattern.Task-specific instructions are inserted at the beginning or end of the input to distinguish Token Type from Char Count.Source: §C.2 · p. 17

C.3 Evaluation Details: BLiMP Benchmark

• BLiMP evaluates linguistic competence with 67 datasets covering 12 phenomena, using minimally different sentence pairs that contrast in grammatical acceptability.The phenomena span semantics, morphology, and syntax.Source: §C.3 · pp. 17–18
• BLiMP’s semantic phenomena include quantifiers and negative-polarity licensing, while its morphology phenomena include agreement and irregular forms.Its syntax phenomena include argument structure, binding, control/raising, ellipsis, filler-gap dependencies, and island effects.Source: §C.3 · p. 18

D Analysis on Additional General Benchmarks

• L2T’s SuperGLUE results vary by data scenario: Disjoint performance falls on knowledge-intensive tasks, whereas structural-resolution tasks such as MultiRC benefit.The passage attributes the pattern to a tension between structural induction and factual reinforcement, with model size also modulating effects.Source: D Analysis on Additional General… · p. 19, Table 7 · p. 19

E Efficacy of Individual Tasks

• On BLiMP island effects, almost all individual tasks outperform Raw, but none succeed on filler-gap phenomena.The authors suggest current L2T data may lack targeted coverage for moved elements and long-distance hierarchical dependencies; Figure 5 presents the individual-task comparison.Source: E Efficacy of Individual Tasks · p. 19, Figure 5 · p. 19

F Mixing Ratio of Raw and L2T Data

• After training, L2T models perform similarly across linguistic subfields at all mixing ratios, with maximum deltas of 1.1 in semantics, 0.17 in morphology, and 0.73 in syntax.The mixing-ratio experiment uses 500M models trained for 100B tokens in the Disjoint setting.Source: F Mixing Ratio of Raw and L2T Data · p. 19, Figure 8 · p. 20
• Computational limits restrict the mixing-ratio experiments to the 500M model.The compared ratios are 100%, 75%, 50%, and 25% L2T data.Source: F Mixing Ratio of Raw and L2T Data · p. 19
• At 5B tokens, L2T 75% scores 91.8 versus 90.8 and 89.7 for L2T 50% and 25% in morphology, and 74.1 versus 72.7 and 66.8 in semantics.Early gains frequently correlate with increased L2T data, except in the L2T 100% setting.Source: F Mixing Ratio of Raw and L2T Data · p. 20
• L2T 100%, with no raw text, has a 23-point ARC decline versus Raw; L2T 75%, which includes 25% raw text, differs by only 4.7 points on ARC.General-benchmark performance generally improves as raw-text proportion rises, except on LogiQA; Kendall tau correlations range from 0.67 to 1.0.Source: F Mixing Ratio of Raw and L2T Data · p. 20, Figure 7 · p. 20
• Even 25% L2T data yields a 1.9 overall gain over Raw in BLiMP, while at least 25% raw text is essential for robust general capabilities, with an 18-point ARC gain over L2T 100%.Figures 7 and 8 present the general-benchmark and linguistic-subfield comparisons, respectively.Source: F Mixing Ratio of Raw and L2T Data · p. 20, Figure 7 · p. 20, Figure 8 · p. 20

G Qualitative Analysis

• 23.2 improvement on the coordinate structure constraint complex left branch task within Island effects in the 500M Disjoint setup.The task distinguishes minimally different sentences.Source: G Qualitative Analysis · p. 20
• The authors attribute improvements on subtle syntactic violations to diverse structural objectives that enhance sensitivity to complex dependencies beyond simple word probabilities.The described challenge involves detecting violations caused by constituent misplacement.Source: G Qualitative Analysis · p. 20
• A 7.9-point drop on the “wh vs that with gap long distance” task within Fill Gap highlights a weakness on long-distance dependencies.The task tests whether a long-distance dependency is correctly licensed.Source: G Qualitative Analysis · p. 20

H License

• The study uses publicly available datasets with different licenses, and reports that all permit academic research.The dataset sources and licenses are summarized in Table 8.Source: H License · p. 21, Table 8 · p. 21

I Use of Generative AI Tools

• The authors acknowledge using Gemini 3.0 Pro to find related work and improve draft grammar and clarity, and GPT-5 for implementation and debugging.Source: I Use of Generative AI Tools · p. 21