Source-linked AI summary
From Amateur to Master: Infusing Knowledge into LLMs via Automated Curriculum Learning
Nishit Neema, Srinjoy Mukherjee, Sapan Shah, Gokul Ramakrishnan, Ganesh Venkatesh
TL;DR
LLMs lag in specialized domains, where pretraining data may lack domain coverage and the structured progression of expert materials. ACER generates curriculum-aligned synthetic textbooks and QA pairs for continual pretraining. It improves target-domain and knowledge-benchmark performance while preserving general-task capabilities.
Problem
LLMs underperform in specialized domains, and pretraining corpora underrepresent specialized knowledge or lack methodical exposition and progressive scaffolding.
Source: §1 · p. 1Method
ACER generates textbook-style curricula and Bloom-guided QA pairs across learner levels, then continually pretrains models using curriculum schedules mixed with general-domain data.
Source: §1 · p. 2Results
ACER yields about 3 percentage points in target-domain MMLU macro-average gains, improves non-target performance by about 0.7 points, and improves ARC and GPQA by over 2 absolute points while maintaining stable general-task performance.
Source: §1 · pp. 2–3Takeaways & Limitations
The results establish a principled foundation for structured knowledge integration and a scalable pathway toward more capable, domain-aware language models.
Source: §7 · p. 9Content generation relies on Gemini 2.0 Flash without a model ablation, and evaluation covers only 1B and 3B models.
Source: §5 · p. 9
Abstract
from arXiv · showhide
Large Language Models (LLMs) excel at general tasks but underperform in specialized domains like economics and psychology, which require deep, principled understanding. To address this, we introduce ACER (Automated Curriculum-Enhanced Regimen) that transforms generalist models into domain experts without sacrificing their broad capabilities. ACER first synthesizes a comprehensive, textbook-style curriculum by generating a table of contents for a subject and then creating question-answer (QA) pairs guided by Bloom's taxonomy. This ensures systematic topic coverage and progressively increasing difficulty. The resulting synthetic corpus is used for continual pretraining with an interleaved curriculum schedule, aligning learning across both content and cognitive dimensions. Experiments with Llama 3.2 (1B and 3B) show significant gains in specialized MMLU subsets. In challenging domains like microeconomics, where baselines struggle, ACER boosts accuracy by 5 percentage points. Across all target domains, we observe a consistent macro-average improvement of 3 percentage points. Notably, ACER not only prevents catastrophic forgetting but also facilitates positive cross-domain knowledge transfer, improving performance on non-target domains by 0.7 points. Beyond MMLU, ACER enhances performance on knowledge-intensive benchmarks like ARC and GPQA by over 2 absolute points, while maintaining stable performance on general reasoning tasks. Our results demonstrate that ACER offers a scalable and effective recipe for closing critical domain gaps in LLMs.
1 INTRODUCTION
LLMs lag in specialized domains where pretraining data is sparse and lacks the structured progression of expert materials. ACER builds curriculum-aligned synthetic materials and uses them for continual pretraining, improving target-domain performance while retaining broader capabilities.
1 INTRODUCTION
- LLMs underperform in specialized domains requiring deep understanding, and niche-subdomain MMLU performance can lag broader fields.General-web-dominated pretraining underrepresents specialized knowledge, while domain-specific material often lacks textbooks’ methodical exposition and progressive scaffolding.Source: §1 · p. 1
- Existing approaches leave gaps: instruction tuning may not deepen knowledge, domain-specific pretraining risks data scarcity and forgetting, and synthetic methods can lack systematic concept coverage.The paper motivates a methodical approach that builds domain expertise without undermining general capabilities.Source: §1 · pp. 1–2
- ACER generates textbook-style curricula and Bloom-guided exam-style QA pairs for four learner levels, then continually pretrains models with interleaved cognitive- and content-based schedules.General-domain data is mixed into training to retain broad capabilities.Source: §1 · p. 2
- ACER gains about 3 percentage points in target-domain MMLU averages, nearly 5 in microeconomics, and about 0.7 on non-target domains.It also improves ARC and GPQA by more than 2 absolute points while maintaining stable performance on general reasoning, arithmetic, and commonsense tasks.Source: §1 · pp. 2–3
2 RELATED WORK
Prior work addresses domain adaptation, synthetic data generation, and curriculum learning, but the paper identifies a gap in scalable, curriculum-aligned infusion of domain knowledge. It investigates curriculum scheduling for domain-adaptive pretraining with synthetic textbook-style corpora.
2 RELATED WORK
- Domain-adaptive pretraining can improve specialization, but curated data scarcity and forgetting risks constrain it.The cited work includes continual pretraining and structured curriculum-style adaptation, but narrow-domain training can compromise general capabilities.Source: §2 · p. 3
- Synthetic corpora and instruction–response pairs offer scalable alternatives, but targeted domain-synthesis methods often lack systematic concept coverage and curriculum progression.Examples include open textbook-style generation and methods mining or synthesizing domain-related content.Source: §2 · p. 3
- Curriculum sequencing has improved training efficiency and downstream accuracy elsewhere, but remains largely unexplored for domain infusion with synthetic textbook-style corpora.This work studies curriculum scheduling as a mechanism for progressive knowledge infusion during domain-adaptive pretraining.Source: §2 · p. 3
3 AUTOMATED CURRICULUM-ENHANCED REGIMEN - ACER
ACER turns domain goals and audience context into a structured synthetic corpus through staged planning, textbook and QA generation, and curriculum scheduling. Its schedules vary cognitive difficulty, audience level, and cross-domain interleaving.
3 AUTOMATED CURRICULUM-ENHANCED REGIMEN - ACER
- ACER begins with domain intent and audience context, then expands them into an outline and textbook-style material with section-aware QA pairs.The pipeline is described as transforming high-level learning goals into a structured synthetic training corpus.Source: §3 · pp. 3–5
- Domain detailing captures content requirements and audience context, producing an editable JSON schema with a domain description, subtopics, and key questions.Inputs include the topic, corpus intent, and learner profile; the schema supports refinement before later generation stages.Source: §3 · p. 4
- Outline generation converts finalized topic metadata into a hierarchical Table of Contents that guides textbook creation and logical progression.The outline-generation inputs include genre and style parameters alongside the domain schema.Source: §3 · pp. 4–5
- Textbook content is generated section by section, and QA pairs are derived from that content to reinforce retrieval, explanation, and application.Questions progress from recall or definition toward interpretation, analysis, and application; answers are grounded in section content.Source: §3 · p. 5
3.1 ACER - CURRICULUM SCHEDULING
- The four schedules range from flat mixing to cognitive ordering of books, easy QA, and hard QA, with Cog+Con additionally ordering audience levels.Cog+Con progresses from high school through undergraduate and graduate to researcher material.Source: §3.1 · pp. 5–6
- Interleaving alternates chapter-section chunks across domains while preserving persona order, testing structured sequencing across both cognitive difficulty and audience level.Figure 2 illustrates the schedules; its personas and chunk labels are illustrative rather than the full set.Source: §3.1 · p. 6, Figure 2 · p. 6
4 EXPERIMENTS
ACER was evaluated on five MMLU domains selected by student–teacher gaps, using synthetic curricula and continual pretraining. Cog+Con improved target-domain results while non-target performance also rose, and gains extended to knowledge-focused benchmarks without losses on general-capability tasks.
4 EXPERIMENTS
- The five largest 3B-student/8B-teacher MMLU gaps—microeconomics, statistics, econometrics, mathematics, and psychology—defined targets for both student sizes.Synthetic books and exam-style QA were generated across high-school, undergraduate, graduate, and researcher audience levels.Source: §4 · p. 6, Figure 3 · p. 7
4.1 TRAINING DETAILS
- Continual pretraining compared Flat, Cognitive, Cog+Con, and chapter-section Interleaved schedules, tracking five target and 51 non-target MMLU subsets.In-domain synthetic data was mixed 1:1 with general replay data to mitigate catastrophic forgetting.Source: §4.1 · p. 7
4.2 RESULTS: IMPACT OF ACER WITH LLAMA 3.2 3B
- 3.0 percentage points on 3B target domains and 0.7 percentage points on non-target Macront: Cog+Con improved over the pretrained baseline.The Table 1 caption reports a 3.0-point target-domain improvement for 3B; the prose reports approximately 0.7 points on non-target domains.Source: Table 1 · p. 8, §4.2 · p. 8
- +2.5 points in Macrot over baseline came from the Flat schedule; Cog+Con performed best, while Interleaved underperformed the other schedules.Interleaved mixes book sections and QA across domains within training stages; its largest drops were in mathematics and statistics.Source: §4.2 · p. 8
4.2.1 RESULTS: SCALING TO SMALLER MODELS- LLAMA 3.2 1B
- More than 2.5 percentage points on 1B target domains and 0.4 points on non-target domains: the smaller student also benefited from Cog+Con.These gains are compared with the pretrained 1B baseline.Source: §4.2.1 · p. 8
4.3 IMPACT BEYOND MMLU: GENERALIZATION TO BROADER CAPABILITIES
- More than 2 absolute percentage points on ARC, GPQA, and MMLU: both model sizes improved on knowledge-intensive benchmarks while general-task scores remained stable.The general-capability comparison covers AGIEval, GSM8K, and HellaSwag; 3B differences were within 0.3 absolute points on HellaSwag and smaller on other tasks.Source: Table 2 · p. 9, §4.3 · p. 9
5 LIMITATIONS OF THE WORK
The evaluation and content-generation setup leave two scope limits: generator choice was not ablated, and only 1B and 3B models were evaluated.
5 LIMITATIONS OF THE WORK
- The study did not compare content-generation models and, due to time and resource constraints, evaluated only 1B and 3B models.The authors note that more powerful and generalizable models may yield greater benefits.Source: §5 · p. 9
6 REPRODUCIBILITY DETAILS
The paper identifies where to find the training and generation settings needed to reproduce its experiments.
6 REPRODUCIBILITY DETAILS
- Section 4.1 reports training parameters, hardware, seeds, prompts, and de-duplication thresholds; Appendix A.6 lists generation prompts.Source: §6 · p. 9
7 CONCLUSION
The conclusion presents ACER as a curriculum-based approach to domain infusion that improves target-domain performance while preserving existing capabilities and enabling cross-domain transfer.
7 CONCLUSION
- ACER combines progressive curriculum-aligned learning sequences with targeted assessment to infuse domain knowledge while maintaining existing-task capabilities and supporting cross-domain transfer.Source: §7 · p. 9
A.1 TOKENS GENERATED USING ACER SYNTHESIS FLOW
ACER’s synthetic book corpus contains 31.97 million tokens, with token counts organized across five domains and four personas.
A.1 TOKENS GENERATED USING ACER SYNTHESIS FLOW
- Gemini 2.0 Flash APIs were used to generate the synthetic data.Source: §A.1 · p. 14
- 31.97 million tokens make up the total ACER synthetic book corpus.The counts use the Llama tokenizer.Source: Table 3 · p. 14
- The corpus token counts are reported across five domains and four personas.Source: Table 3 · p. 14
A.2.1 DOMAIN DETAILING
Domain detailing explicitly encodes domain structure before corpus synthesis, supporting traceability and curriculum-aware scaling.
A.2.1 DOMAIN DETAILING
- The section presents domain detailing as a step before corpus synthesis.Source: §A.2.1 · p. 14
- Domain detailing provides a transparent knowledge map linking input requirements to generated content.Source: §A.2.1 · p. 14
- It enables later synthetic data creation, including textbooks and assessments, to follow a structured hierarchy rather than ad-hoc sampling.Source: §A.2.1 · p. 14
A.2.2 HIERARCHICAL STRUCTURE GENERATION
This section describes how ACER uses a hierarchical textbook outline and prompts to generate textbook-grounded instructional content, including question sequences that progress from recall to deeper cognitive tasks. It also presents microeconomics examples and discusses data deduplication and the balance between synthetic data and replay.
A.2.2 HIERARCHICAL STRUCTURE GENERATION
- The outline-generation process specifies 4–6 parts, 4–6 chapters per part, and 3–6 sections per chapter, with optional subsections.A structured JSON outline supports consistency, automated downstream generation, and progression from foundational to advanced material.Source: §A.2.2 · p. 14
A.2.3 SYNTHETIC CONTENT CREATION
- Section content is generated from a tree whose nodes retain titles, descriptions, content, and node types, with prompts carrying surrounding outline context and style guidelines.The model is instructed to produce polished instructional prose without meta-commentary or explanations of its reasoning.Source: §A.2.3 · pp. 14–15
A.3 ABLATIONS: WHAT IS THE RIGHT DATA MIXTURE?
- About 3 points higher Macrot and 0.7 points higher Macront than baseline, the 1:1 synthetic-to-replay mixture outperforms the tested skewed ratios.Replay supports non-target performance, while in-domain synthetic data drives target-domain gains.Source: §A.3 · p. 15, Table 4 · p. 15
A.4 EXAMPLE DOMAIN: MICROECONOMICS / A.5 EXAMPLE OF PERFORMANCE IMPROVEMENT AFTER ACER
- In a microeconomics example, the appendix shows generated instructional content and easy/hard QA pairs, while ACER answers an elasticity question correctly that base LLaMA3.2-3B misses.The passage attributes the improvement to knowledge injected during continual pretraining, including a book section on supply curves and elasticity.Source: §A.4 · pp. 15–16, §A.5 · p. 17
A.6.1 TOPIC DETAILING PROMPT / A.6.2 TOC GENERATION PROMPT
- The topic-detailing prompt asks for a factual, high-level topic description suited to the target audience, 6–8 core themes or subtopics for well-rounded coverage, and 6–8 questions the book should answer from that audience’s perspective.Source: §A.6.1 · p. 18
A.6.3 SECTION GENERATION PROMPT
- The section-generation prompt asks for detailed, cohesive, elaborated prose that follows style guidelines and avoids meta-commentary or structural notes.It directs the model to produce polished content ready for publication.Source: §A.6.3 · p. 20
A.6.4 QA PAIR GENERATION PROMPTS
- The QA-pair prompt casts the model as a professor specializing in the subject.The supplied passage only introduces the question-generation prompt and its role framing.Source: §A.6.4 · p. 20
- Question sequences start with simple factual-recall or definition questions, then become slightly more challenging through understanding, interpretation, analysis, and application.Later questions should differ in focus or angle from the previous question while remaining relevant to the section and audience.Source: §A.6.4 · p. 21
- Answer-generation prompts require detailed, educational, self-contained answers grounded in the section content and prohibit inventing information.The answer should use clear language and examples when helpful, and suit the specified audience.Source: §A.6.4 · p. 22
A.7 DECONTAMINATION- DATA ANALYSIS
- Semantic deduplication found little to no benchmark contamination, while generated samples with similarity above 0.9 covered topics commonly studied when learning the domain.The analysis embedded generated and benchmark samples, compared each generated sample with its closest benchmark neighbor using cosine similarity, and inspected samples above a threshold; Table 5 reports benchmark samples with similarity ≥0.8, and Table 6 shows examples.Source: §A.7 · p. 22, Table 5 · p. 22, Table 6 · p. 23