Source-linked AI summary
CLASS-IT: Conversational and Lecture-Aligned Small-Scale Instruction Tuning for BabyLMs
Luca Capone, Alessandro Bondielli, Alessandro Lenci
TL;DR
The paper asks whether instruction tuning can benefit BabyLM-scale models trained on child-comparable exposure, and compares conversational and question-answering data in merged and sequential curricula. It finds modest gains in fine-tuning settings, especially for sequential data, but those benefits do not consistently carry over to zero-shot evaluation.
Problem
The study asks whether instruction tuning benefits BabyLMs trained on child-comparable linguistic exposure and whether instruction-data type or curriculum makes a difference.
Source: §1 · p. 1Method
Results
Instruction tuning yields modest gains in fine-tuning scenarios, particularly with sequential rather than merged data, but benefits do not transfer consistently to zero-shot evaluations.
Source: §7 · pp. 6–7Takeaways & Limitations
At this scale, instruction-tuning gains are context-dependent, and the authors identify hybrid curriculum approaches as a direction for improving generalization under ecological training limits.
Source: §7 · p. 8Instruction-tuning datasets were small relative to pre-training, conversational data were underrepresented, and the Simple Wikipedia augmentation was only partially validated.
Source: Limitations · p. 8
Abstract
from arXiv · showhide
This work investigates whether small-scale LMs can benefit from instruction tuning. We compare conversational and question-answering instruction tuning datasets, applied either in a merged or sequential curriculum, using decoder-only models with 100M and 140M parameters. Evaluation spans both fine-tuning (SuperGLUE) and zero-shot (BLiMP, EWoK, WUGs, entity tracking, and psycholinguistic correlation) settings. Results show that instruction tuning yields small but consistent gains in fine-tuning scenarios, with sequential curricula outperforming merged data; however, improvements do not consistently transfer to zero-shot tasks, suggesting a trade-off between interaction-focused adaptation and broad linguistic generalization. These results highlight both the potential and the constraints of adapting human-inspired learning strategies to low-resource LMs, and point toward hybrid, curriculum-based approaches for enhancing generalization under ecological training limits.
1 Introduction
The paper asks whether targeted instruction can benefit BabyLMs trained on child-comparable linguistic exposure, and whether data type and curriculum matter. It focuses on interaction as a shared feature of human and model learning despite the much greater resources used by LMs.
1 Introduction
- LM abilities reflect training, but whether some abilities emerge abruptly or increase gradually remains uncertain.Training-data type and order can affect task performance, while instruction tuning and RLHF can enhance interaction and reasoning abilities.Source: §1 · p. 1
- Human learning and LM training both respond to instruction and interaction, but LMs require far more data and computation than children.The study focuses on interaction as a component of childhood learning and tests models trained on child-comparable exposure.Source: §1 · p. 1
- The study tests whether a BabyLM benefits from instruction tuning, which instruction data works better, and whether a curriculum offers gains.The comparison is between conversational and open-ended question-answering data, and between curriculum approaches.Source: §1 · pp. 1–2
2 Related Works
The section qualifies claims about children’s early language abilities and distinguishes this study’s formal instruction tuning from earlier interactive BabyLM approaches. It motivates testing whether explicit question-answer and conversational instruction benefits models trained on limited exposure.
2 Related Works
- Children’s fluent speech can coexist with difficulty using complex connectives and producing coherent narratives, with mastery developing gradually.The passage describes these challenges in children up to about age nine.Source: §2 · p. 2
- Earlier BabyLM work used encoder-only masked language modeling or decoder PPO reward preferences, whereas this study explicitly trains answers to questions and conversational prompts.The authors frame this instruction tuning as closer to formal schooling than reward-based preference optimization.Source: §2 · p. 2
- The study asks whether formal instruction-like input benefits limited-exposure LMs more than conversational data.It investigates interactive instruction as a way to enhance BabyLM training.Source: §2 · p. 2
3 Dataset
The study uses roughly 91 million pre-training words and an instruction-tuning set combining Switchboard conversation with Simple Wikipedia question-answer pairs. Only a representative instructional subset is used to stay within the challenge’s word limit.
3.1 Pretraining Dataset
- The pre-training corpus contains approximately 91 million words, including organizer data and material overlapping with instruction tuning.Roughly 75 million words from CHILDES, Gutenberg, BNC, and OpenSubtitles were used exclusively for pre-training; another 16 million came from Switchboard and Simple Wikipedia.Source: §3 · p. 2, §3.1 · p. 3
3.2 Instruction-Tuning Dataset
- Instruction tuning combines Switchboard prompt–reply dialogue items with Simple Wikipedia question-answer pairs generated using LLaMA-3.2-3B-Instruct.The Switchboard component contains 38,802 items and approximately 1.3 million words; three question-answer pairs were generated per article.Source: §3.2 · p. 3
- The study uses an 8.7-million-word representative instructional subset to keep the combined corpus within the challenge’s 100-million-word limit.The augmented Simple Wikipedia dataset totals 97,697 items and 18 million words; the full dataset is to be released after validation.Source: §3.2 · p. 3
4 Models and training
The authors train two decoder-only LLaMA-style models and apply a shared pre-training procedure followed by target-token instruction tuning. They compare mixing conversational and instructional data with presenting the datasets sequentially in either order.
4 Models and training
- The two decoder-only LLaMA-style models have approximately 140 million and 100 million parameters, with the larger model using larger hidden and vocabulary sizes.The smaller model uses a reduced hidden size and vocabulary comparable to baseline models.Source: §4 · p. 3
- Both models were pre-trained for 8 epochs and instruction-tuned for 10 epochs, with instruction-tuning loss applied only to target tokens.The models processed approximately 908 million words in total, of which about 817 million influenced weight updates.Source: §4 · pp. 3–4
- The merged strategy shuffles conversational and instructional items together, while sequential variants present the datasets in either order.The order variants are it_switch_wiki and it_wiki_switch.Source: §4 · p. 4
5 Evaluation and Results
Instruction tuning's effects vary by evaluation: fine-tuning gains are modest and inconsistent, while zero-shot results show no clear overall advantage. Across combined evaluations, sequential curricula outperform merged data, but model size also matters.
5 Evaluation and Results
- To compare heterogeneous tasks, the authors standardized each model's task score as a z-score, averaged scores into a global relative index, and plotted medians, IQRs, and outliers.The index measures relative standing against other models rather than absolute task performance.Source: §5 · pp. 4–5, Figure 3 · p. 5
- Instruction-tuned models beat their pretrained counterparts in at least one configuration on every task, but gains were small and inconsistent; baseline wins occurred only for some WSC configurations.No instruction-tuning configuration systematically led to better results.Source: §5 · p. 4, Figure 2 · p. 5
- All 140M fine-tuning variants had positive median z-scores versus negative medians for 100M; the two sequential variants alone had all scores above zero and much smaller IQRs.The merged instruction-tuned model had the highest median, while single-dataset variants performed worse and were on par with the pretrained-only model.Source: Figure 3 · p. 5, §5 · p. 5
- On zero-shot tasks, accuracy differences were small, WUGs performance was strong and entity tracking difficult; 100M variants led on R²-based tasks.The 100M variants scored vastly above both 140M models and baselines on the Change in R² tasks.Source: §5 · pp. 5–6
- Only two models had positive median zero-shot z-scores; the Simple Wikipedia-only tuned variant ranked best, but no clear instruction-tuning trend appeared.The authors caution that the 100M variants' strong standing could reflect their advantage on R²-based tasks.Source: §5 · p. 6
- Across fine-tuning and zero-shot evaluations, larger models generally performed better and sequential instruction tuning beat merged data, but no clear overall instruction-tuning trend emerged.The Simple Wikipedia-only model outperformed the model tuned first on Wikipedia and then on conversations, which the authors note may reflect dataset size.Source: §5 · p. 6
6 Discussion
Instruction tuning appears most beneficial after task-specific fine-tuning, while gains are less apparent on zero-shot evaluations. Smaller models correlate better with human data, but performance varies substantially across tasks and evaluation methods.
6 Discussion
- Instruction tuning appears somewhat beneficial, especially after task-specific fine-tuning, but improvements are less apparent on zero-shot evaluations.The authors hypothesize that instruction tuning may shift models toward conversational or factual tasks at the expense of generalization on language tasks.Source: §6 · p. 6
- Smaller models consistently correlate better with human data than larger models, and the pretrained model is slightly better among the authors’ models.Only one baseline of the same size but a different architecture achieves comparable results.Source: §6 · p. 6
- Models perform relatively consistently within individual tasks but inconsistently across tasks and evaluation methods, suggesting challenge data and compute constraints may limit generalization.The authors note that the constraints are those on data and training compute allowed for a run.Source: §6 · p. 6
7 Conclusion and future work
Instruction tuning yields modest gains in fine-tuning, particularly when conversational and question-answering datasets are presented sequentially rather than merged. These benefits do not consistently transfer to zero-shot evaluation, and the authors identify evaluation and data limitations while recommending hybrid approaches for future work.
7 Conclusion and future work
- Instruction tuning yields modest but measurable fine-tuning gains, particularly when conversational and question-answering datasets are presented sequentially rather than merged.The benefits do not consistently transfer to zero-shot evaluations and may come at the expense of broader linguistic generalization at this scale.Source: §7 · pp. 6–7
- Figure 4 separates zero-shot accuracy tasks in the left chart from tasks measured by change in R2 in the right chart.The supplied caption identifies the charts’ metrics but does not report their values.Source: Figure 4 · p. 7
- Figure 5 shows medians, inter-quartile ranges, and outliers for model z-scores in zero-shot evaluation.The caption describes the distribution summary, not which models perform better.Source: Figure 5 · p. 7
- Challenge evaluation may undermine measured performance, particularly for conversational instruction tuning, because most fine-tuning tasks are classification and most zero-shot tasks use log-likelihood proxies.These choices accommodate more architectures and simplify evaluation, according to the authors.Source: §7 · p. 7
- Figure 6 shows medians, inter-quartile ranges, and outliers for model z-scores across zero-shot and fine-tuning evaluations.The authors say larger models perform better overall in this combined view.Source: Figure 6 · p. 7
- Switchboard represents a restricted spoken-English register and lacks the contextual diversity of everyday interaction; the authors propose hybrid training, alternative architectures, and interactive benchmarks for future work.They also report stronger human psycholinguistic correlations for smaller models.Source: §7 · pp. 7–8
Limitations
The authors identify limits in instruction-data scale and balance, as well as partial validation of the Simple Wikipedia augmentation.
Limitations
- Instruction datasets were small relative to pretraining, conversational data were underrepresented, and Simple Wikipedia augmentation was only partially validated.The authors note that these factors may reduce instruction-specific effects, bias results toward factual rather than interactive skills, and constrain robustness and generalization.Source: Limitations · p. 8