Source-linked AI summary
Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Training
Lukas Borggren, Jenny Kunz, Marco Kuhlmann
TL;DR
Large language models may perform modestly in understudied domains, motivating adaptation to Swedish journalism and better in-domain evaluation. The paper curates Swedish news data, develops BonEval, and tests continued pre-training with full and parameter-efficient methods. Continued pre-training improves generative and knowledge-intensive journalism tasks when paired with experience replay, while discriminative gains are not consistent; instruct vectors further help LoRA models but not full fine-tuning.
Problem
Journalism and Swedish remain understudied targets for large language model adaptation, while existing Swedish evaluation resources do not comprehensively assess journalism.
Method
The paper curates BonCorpus from millions of Swedish news articles, constructs the six-task BonEval benchmark, and adapts models through continued pre-training with experience replay and parameter-efficient methods.
Results
Continued pre-training consistently improves generation quality and factual knowledge across model sizes when paired with experience replay, but not discriminative performance; instruct vectors further help LoRA models but impair full fine-tuning.
Takeaways & Limitations
Targeted evaluation is essential because the existing Swedish benchmark largely fails to capture the models' in-domain performance gains.
Takeaways & Limitations
The study's generalisability is limited by experiments using a single model family, and its unreleased artefacts reduce reproducibility.
Abstract
from arXiv · showhide
Large language models are increasingly capable in general, but their utility can remain modest in niche or understudied areas. One approach to address this limitation is to specialise existing models through additional training on target-domain corpora. In this work, we investigate such continued pre-training for adapting large language models to Swedish journalism, using a high-quality dataset that we curate from millions of news articles. To evaluate the adaptation efficacy, we also construct a novel domain-specific benchmark that covers six editorial tasks. Through full and parameter-efficient fine-tuning across two model sizes, we find that continued pre-training yields benefits in the target domain, but only when paired with experience replay to mitigate forgetting. We observe consistent enhancements in the models' generation quality and factual knowledge, but not their proficiency in discriminative tasks. Exploring a training-free method to facilitate instruction following, we see further improvements, but exclusively for models trained with low-rank adaptation. Crucially, we demonstrate the importance of targeted evaluation in the adaptation process, as an existing Swedish benchmark largely fails to capture the models' in-domain performance gains.
1 Introduction
Journalism and Swedish remain comparatively understudied targets for adapting large language models, while training models from scratch requires substantial data and compute. This paper addresses both gaps by curating Swedish news data, applying continued pre-training, and introducing targeted evaluation.
- Journalism has rarely been treated as a target domain for large language model adaptation, with only one prior study identified.
- From-scratch training for journalism and Swedish-language models demands vast amounts of data and compute, creating barriers for lower-resource languages.
- The paper curates a high-quality corpus from millions of Swedish news articles to specialise models through continued pre-training.
- The authors introduce a domain-specific benchmark covering six editorial tasks to evaluate adaptation effectiveness.
- Continued pre-training is presented as a viable route for adapting language models to Swedish journalism and potentially retaining publisher control over editorial tools.
2 Background and Related Work
The related work frames continued pre-training as a way to specialise models while managing catastrophic forgetting. It also motivates replay, parameter-efficient adaptation, instruction-following remedies, and targeted journalism evaluation.
- Continued pre-training adapts general-purpose language models to target domains, but must avoid largely losing their original abilities through catastrophic forgetting.
- Experience replay mitigates forgetting by mixing previously seen data into continued-pre-training corpora and may also improve target-domain performance.
- Parameter-efficient methods such as LoRA restrict updates to reduce forgetting, potentially trading off adaptation learning.
- Instruction-like data, model merging, and task-vector-style editing have been explored to preserve or restore instruction-following after continued pre-training.
- Swedish journalism lacks a comprehensive evaluation resource, with only three related tasks and general benchmark collections available.
3 Method
The paper builds BonCorpus from Swedish journalism and BonEval from editorial content, then uses replay mixtures to adapt models while addressing distribution shift and forgetting.
- BonCorpus: The corpus contains 21.7M Swedish articles published by Bonnier News from 1991 through 2026 across newspapers and magazines.
- BonCorpus: BonCorpus preprocessing standardises formats, normalises text, filters low-quality articles, and removes exact and fuzzy duplicates.
- BonEval: BonEval comprises six editorial tasks spanning generation, information extraction, and factual-knowledge evaluation.
- BonEval: The six tasks are Headline, Lead, Summary, Topic, Entity, and Quiz, covering generation, classification, named-entity recognition, and question answering.
- BonEval: Evaluation examples are mainly drawn from post-2022 BonCorpus articles and excluded from training, with 8,192 stratified examples for several tasks.
- BonLM: BonLM uses BonCorpus as primary data and adds code, English, or Swedish replay data to mitigate distribution shift and forgetting.
4 Experimental Setup
The experiments derive a continued-pre-training recipe on 3B models, apply selected methods to 8B models, and compare them across targeted and existing Swedish evaluations.
- Models and training: Initial continued-pre-training experiments use Ministral 3B and 8B base models, with smaller-model experiments guiding the training recipe.
- Evaluation: Models are evaluated on BonEval, supplementary editorial tasks, and seven Swedish EuroEval tasks using few-shot prompting and repeated evaluation rounds.
- Training procedure: Each 3B data mixture is trained for 8,000 steps, and the mixture with the best average BonEval score is selected for subsequent experiments.
- Training procedure: The selected recipe is applied with LoRA and LLaMA Pro, alongside full fine-tuning and experiments adding unweighted instruct vectors.
- Final training: BonLM 8B is trained for the equivalent of four epochs and compared with similarly sized Ministral, GPT-SW3, Llama SW3, and Apertus baselines.
- Evaluation tables: Table 1 reports average BonEval scores, while Table 2 reports task-specific metrics including F1, CHRF3++, and MCC.
5 Results and Analysis
CPT improves Swedish journalism performance when paired with suitable replay mixtures, but its effects vary by adaptation method, task type, and evaluation benchmark. Gains are strongest for generation and factual knowledge, while discriminative and general benchmarks can reveal regressions.
- 5.1 Impact of Data-Mixture Composition: BCES achieves the highest average BonEval score among the initial 3B mixtures, narrowly outperforming BCS.All mixtures improve over the base model except B and BES; B shows the greatest degradation, likely from catastrophic forgetting.
- 5.2 PEFT and Instruct-Vector Compatibility: LoRA + IV yields the highest average BonEval score, while IV improves LoRA but impairs FFT.LoRA has a mean relative parameter distance of 0.08 from the base model, compared with 0.18 for FFT.
- 5.4 Improved Performance on BonEval: BonLM FFT and LoRA outperform all baselines on average and on most individual BonEval and supplementary tasks.LoRA is generally superior to FFT for the 8B models, and LoRA + IV again achieves the highest average score.
- 5.4 Improved Performance on BonEval: CPT consistently improves generative Headline, Lead, and Summary tasks but slightly degrades discriminative Topic and Entity tasks.Quiz shows the largest absolute gains, suggesting that news articles provide culturally rich factual knowledge.
- 5.4 Improved Performance on BonEval: Supplementary tasks support BonEval’s validity: CPT improves generation quality and factual knowledge, while instruction tuning increases generation quality across five generative tasks.LoRA + IV surpasses LoRA, whereas prolonged post-training of Ministral reduces Topic, Entity, and Quiz scores.
- 5.3 Mixed Results on EuroEval: CPT improves BonEval performance but degrades EuroEval performance, partly reflecting domain-specific gains at the expense of general capabilities.Regressions are especially pronounced on translated MMLU and HellaSwag, whereas native Swedish Quiz and SwedishFacts show substantial knowledge gains.
6 Conclusion and Future Work
The paper presents a comprehensive study of adapting LLMs to Swedish journalism through BonCorpus, BonEval, and experience-replay-based CPT. It concludes that targeted evaluation is essential because EuroEval fails to reflect broader in-domain gains, while future work will examine editorial applications and human evaluation.
- 6 Conclusion and Future Work: CPT improves generative and knowledge-intensive tasks across two model sizes when paired with carefully constructed experience-replay mixtures.Its benefits do not extend consistently to discriminative tasks, and IV generally helps LoRA but not FFT.
- 6 Conclusion and Future Work: BonEval reveals in-domain trends that EuroEval’s Swedish tasks fail to capture, underscoring the importance of targeted evaluation during adaptation.The study introduces BonCorpus and BonEval as resources for Swedish journalism and broader domain specialization.
- 6 Conclusion and Future Work: Future work will fine-tune BonLM for editorial use cases and collaborate with journalists on human-evaluation protocols.The authors identify headline generation, stylistic correction, and real-world newsroom utility as intended directions.
Limitations
The study’s reproducibility and generalisability are constrained by unavailable resources, a single model family, undisclosed pre-training data, limited downstream testing, and exclusive reliance on automatic evaluation.
- Limitations: Proprietary data, code, and models cannot be released, reducing the study’s reproducibility.The restriction is attributed to business and legal reasons.
- Limitations: Experiments use a single model family, limiting the generalisability of the findings.Because Ministral already has relatively strong Swedish capabilities, another model family could yield different conclusions.
- Limitations: Undisclosed original pre-training data prevent assessment of overlap with BonCorpus or BonEval and its effects on results.This leaves potential data contamination and its consequences unresolved.
- Limitations: The study provides limited evidence about downstream editorial effects because BonLM is not fine-tuned for specific applications.The authors also rely exclusively on automatic evaluation, preventing assertive claims about real-world utility.
Ethical Considerations
The paper limits external misuse risk because its research artefacts will not be released, while intended internal deployment could still affect journalism and public news. It also documents extensive corpus cleaning, filtering, language identification, and deduplication procedures.
- Ethical Considerations: Unreleased research artefacts limit the general risk of misuse by external actors.
- Ethical Considerations: BonLM’s intended internal deployment could affect reporters, editors, and the news presented to the public.The system would be subject to strict usage policies.
- Corpus Processing: The corpus pipeline removes peripheral fields, advertisements, notices, bylines, JavaScript, and HTML while preserving fact boxes, tables, and lists.
- Corpus Processing: Articles are filtered using text and metadata criteria, including minimum length, lexical diversity, and fastText Swedish classification above 0.9.
- Corpus Processing: The pipeline removes non-Swedish, automatically generated, unsuitable, and duplicate articles using language filtering and document-level similarity checks.Deduplication uses MinHash and locality-sensitive hashing for candidate generation, followed by normalized Levenshtein distance.
B Evaluation Data
BonEval evaluation data are selected from BonCorpus through task-specific filters designed to produce suitable and representative examples. The setup also specifies task labels, computational resources, sequence packing, and parameter matching for adaptation methods.
- Evaluation Data: BonEval excludes articles longer than 1,024 tokens and applies additional criteria tailored to each editorial task.
- Evaluation Data: Headline, lead, and summary examples are filtered by length and structure, including headline, lead, and bullet-point requirements.The supplied criteria cover headline lengths of 10–120 characters, lead lengths of 30–500 characters, and summaries of 200–800 characters.
- Evaluation Data: Topic classification is simplified from thousands of hierarchical labels to 19 top-level topics.
- Evaluation Data: Entity examples retain labels mentioned within the first 90% of an article and prefer specific over geographically nested general labels.
- Evaluation Data: Quiz examples require four answer choices and exclude English-language, image- or audio-dependent, and strongly time-dependent content.
- Training Setup: Experiments use DGX H200 nodes and efficiency methods, while packed sequences and comparable trainable-parameter counts support computationally controlled training.LoRA and LLaMA Pro use 395M and 466M trainable parameters, respectively, for the 3B model.
D Evaluation Setup
Models are evaluated with BonEval implementations and EuroEval for tasks outside BonEval, using repeated few-shot evaluation. The number of examples is chosen to fit a 4,096-token context limit and capped at 12.
- Evaluation Setup: BonEval is implemented directly, while EuroEval supplies evaluations and existing implementations for tasks outside BonEval.Model outputs use log probabilities for Topic and Quiz and structured generation for Entity.
- Evaluation Setup: Each model is evaluated over 10 runs with randomly sampled few-shot examples from the training split.
- Evaluation Setup: Few-shot counts are selected so inputs remain within 4,096 tokens and are capped at 12 examples.The resulting task-specific counts are reported in Table 5.
E Complete Evaluation Results
The complete evaluation materials include task prompts and metric definitions for BonEval and EuroEval across the 3B and 8B experiments. Reported 8B results show that CPT models outperform baselines on average and most individual BonEval tasks, with LoRA + IV achieving the highest average score.
- Complete Results: Complete 3B results are reported in Tables 10 and 11, while BonLM 8B and baseline results are reported in Tables 12 and 13.
- Prompts: BonEval base-model prompts cover headline, lead, summary, topic, and quiz-style inputs in Swedish and translated forms.
- Metrics: BonEval uses task-specific metrics, including CHRF3++, micro-averaged F1, and MCC, with supplementary metrics and confidence intervals reported for 8B results.