PaperlayerView exact revision on arXiv

Source-linked AI summary

FrenchNews-7: Benchmarking Cross-Publisher French News Editorial Desk Classification

Amr Sobhy

arXiv:2608.18097v1cs.CL

TL;DR

French news research lacks a benchmark for reliably classifying editorial desks across publishers. FrenchNews-7 builds such a benchmark and evaluates trained classifiers across publisher settings, finding that CamemBERT-base generalizes to unseen outlets and sustains strong seven-class performance.

  • Problem

    French news research lacks a directly targeted cross-publisher benchmark for reliable editorial-desk classification across outlets.

  • Method

    FrenchNews-7 combines a multi-outlet corpus, a seven-class URL-derived taxonomy, hybrid URL/LLM labeling, and in-distribution and held-out-publisher classifier evaluations.

  • Results

    CamemBERT-base sustains macro-F1 ≈0.80 on unseen outlets across all seven classes, with full-text input outperforming headline-only input.

  • Takeaways & Limitations

    URL-derived labels support a reusable cross-publisher editorial-desk classifier, while category transfer varies across editorial boundaries.

  • Takeaways & Limitations

    Most ambiguous-URL training labels are single-LLM outputs without per-article human verification, and broader multi-annotator consensus remains future work.

Abstract

from arXiv · show

We present FrenchNews-7, a cross-publisher France-based French-language news editorial desk classification benchmark combining a large multi-outlet corpus, a URL-derived seven-class taxonomy, and a fine-tuned CamemBERT classifier. Labels are assigned via a hybrid pipeline combining publisher URL slugs with LLM annotation for structurally ambiguous cases, audited through an inter-rater study (2 humans + 2 LLMs; pairwise $κ\geq 0.766$, human--human $κ= 0.806$). We evaluate lexical, multilingual, and French-specific trained classifiers under both in-distribution and held-out-publisher settings, with additional comparison against zero-shot LLM baselines (GPT-OSS-120B, Mistral Small 3.2, Llama-3.3-70B) on the held-out pool. The strongest model, CamemBERT-base on full article text, outperforms headline-only input, generalizes to unseen outlets, and exceeds all three zero-shot LLM baselines on overall recall (0.799), with the gap concentrated in the ambiguous editorial-boundary categories Economie and Societe. Cross-publisher evaluation reveals uneven boundary stability: Sport, Culture & Loisirs, and International transfer cleanly, while Economie (recall = 0.517) is close to blinded human agreement (0.55), and Societe (precision = 0.577) absorbs boundary ambiguity, both suggesting editorial conventions rather than recoverable classifier headroom. The fine-tuned CamemBERT-base model, labeled manifest, reference collection scripts, and a reliability-tier guidance table are available at https://huggingface.co/LeFrenchNewsLab/camembert-base-frenchnews7 (model) and https://huggingface.co/datasets/LeFrenchNewsLab/frenchnews-7 (dataset).

1 Introduction

FrenchNews-7 addresses the lack of a directly cross-publisher French news benchmark by targeting publisher editorial-desk assignments rather than latent content topics. It provides a harmonized seven-category corpus and CamemBERT classifier evaluated both in-distribution and on held-out publishers.

  • Motivation: FrenchNews-7 fills a gap in French comparative media research, where existing corpora are single-outlet or too narrow to support cross-publisher generalization.The benchmark is designed for agenda-setting, media-diversity, and longitudinal topic-coverage studies requiring reliable classification across publishers.
  • Target definition: Editorial-desk assignments are the target variable by construction, distinguishing the benchmark from content-based topic classification.The label represents publisher routing decisions rather than a noisy proxy for latent topics.
  • Labeling: 72.2% of labels come from publisher URL slugs, while 27.8% are content-labeled by an LLM when URLs lack a desk signal.The hybrid labeling pipeline combines deterministic URL-slug rules with LLM annotation for ambiguous cases.
  • Taxonomy: The taxonomy includes editorial desks spanning society and domestic affairs, Culture & Loisirs, International, Politique, Sport, Économie, and Sciences & Technologies.These categories form the benchmark’s harmonized editorial-desk label space.
  • Contributions: The benchmark contributes a French editorial-desk corpus and a CamemBERT classifier evaluated under both in-distribution and held-out-publisher conditions.The evaluation explicitly tests transfer beyond the publishers represented during training.

2 Related Work

Prior work spans classical and neural news-topic classification, French-language encoders, LLM-assisted annotation, and cross-publisher taxonomy harmonization. FrenchNews-7 builds on these lines while distinguishing taxonomy mismatch from cross-lingual domain shift.

  • News topic classification: News-topic classification progressed from TF-IDF + SVM and RCV1 to large-scale datasets and BERT-family encoders targeting harder classification settings.
  • French NLP: CamemBERT is a RoBERTa-based encoder pretrained on ~138GB of French text, while CamemBERTav2 uses DeBERTaV3, 275B tokens, and a 1,024-token sequence limit.FlauBERT introduced the FLUE benchmark suite; the closest identified French news classifier is a FlauBERT model fine-tuned on MLSUM.
  • LLM-based annotation: LLM annotation research reports strong agreement with crowd or expert annotators and lower costs, while FrenchNews-7 applies LLM labeling only to structurally ambiguous articles (27.8%).The cited prior work includes zero-shot ChatGPT, GPT-4, and GPT-4-labeled fine-tuned classifiers across multiple annotation settings.
  • Cross-publisher harmonization: Cross-publisher harmonization has been studied with EMMediaTopic, which labels 21,000 texts in four non-French languages across 17 IPTC categories and trains XLM-RoBERTa-large with macro-F1 = 0.746.FrenchNews-7 extends URL-slug-derived taxonomy signals from a single-publisher precedent to 13 publishers.
  • Taxonomy comparability: Taxonomy misalignment explains less than half of the performance gap between EMMediaTopic and CamemBERT on FrenchNews-7, with cross-lingual domain shift identified as the dominant factor.The probe uses the EMMediaTopic classifier on the FrenchNews-7 test split without retraining, under French text and an unseen label space; structural mismatch remains distinct from representational limitations.

3 Dataset Construction

FrenchNews-7 combines 87,769 articles from 13 politically heterogeneous France-based French-language outlets with a seven-category taxonomy derived from publisher URL structures. Labels use deterministic slug rules when possible and GPT-OSS-120B for ambiguous slugs, with audit evidence supporting label quality and residual boundary ambiguity.

  • Taxonomy: The taxonomy contains seven editorial categories selected from URL-path families using presence in ≥10 of 13 publishers and corpus share ≥3.5%.The categories are Société, Culture & Loisirs, International, Politique, Sport, Économie, and Sciences & Technologies; smaller families are absorbed or deferred to LLM labeling.
  • Corpus: 87,769 articles were collected from 13 France-based French-language outlets spanning national, weekly, regional, digital-native, and broadcast news coverage.The corpus primarily covers 2018–2026, with 4% of articles predating 2018.
  • Annotation: 72.2% of articles received deterministic Bucket A labels through 74 slug rules, while 27.8% entered Bucket B because their slugs lacked an unambiguous category signal.Bucket B articles were labeled by GPT-OSS-120B using category definitions, headlines, and full body text with temperature = 0.
  • Annotation: 2,215 Bucket B articles versus 1,982 Bucket A articles contributed to Sciences & Technologies, the category with the largest relative Bucket B contribution.This reflects less consistent representation of science slugs across publishers.
  • Quality assurance: A Bucket A-only ablation changed macro-F1 by −0.004 (95% CI [−0.024, +0.024]; McNemar p = 0.63), indicating no measurable impact from Bucket B labels on reported results.The audit also found 84.0% blinded human recovery of author labels (Cohen’s κ = 0.806), while disagreements concentrated in Société, Culture & Loisirs, and Économie.

4 Modeling and Experimental Setup

The study compares four trained classifiers and a prompting-based LLM comparator under full-text and headline-only conditions. Evaluation uses stratified in-distribution splits, strictly disjoint held-out publishers, macro-F1, per-class metrics, confidence intervals, and corrected significance tests.

  • Models: Four trained models—TF-IDF + Logistic Regression, mBERT, CamemBERT-base, and CamemBERTav2-base—are compared with a prompting-based LLM comparator.The lexical baseline uses character and word n-grams with L2 regularization; the neural models are multilingual or French-pretrained encoders.
  • Input conditions: Both headline + body and headline-only inputs are evaluated for all four trained models.Full-text inputs use 512-token truncation for models with that limit.
  • Evaluation: 13,146 test articles support evaluation after a 70/15/15 stratified split across 13 outlets, with four additional outlets reserved as a strictly disjoint held-out set.Macro-averaged F1 is the primary metric, alongside per-class results; main and pooled held-out results use 95% bootstrap confidence intervals over 1,000 replicates, while held-out recall slices use 95% Wilson intervals.
  • Evaluation: CamemBERT-base cross-publisher macro-F1 ranges from 0.786 to 0.816 across five seeds, with CV = 1.6%, while in-distribution variance has CV = 0.36%.The five seeds are 42, 123, 456, 789, and 101112.
  • LLM comparison: GPT-OSS-120B is additionally evaluated with balanced 5-shot and 7-shot prompts using only deterministic Bucket A validation examples.Examples come strictly from the in-distribution validation split, excluding LLM-annotated exemplars to prevent unseen-publisher leakage and circular label-distribution reinforcement.

5 Results

CamemBERT-base achieved the strongest overall in-distribution performance, with full-text input consistently outperforming headline-only input. It also transferred to unseen publishers, where Sport remained robust but Économie and Société exposed persistent editorial-boundary ambiguity.

  • In-distribution results: CamemBERT-base reached macro-F1 0.847 and accuracy 0.860 on the held-out test split, making it the recommended deployment model.The 95% bootstrap CIs were [0.841, 0.854] for macro-F1 and [0.854, 0.866] for accuracy.
  • In-distribution results: The 1.4pp mBERT–CamemBERT-base gap quantified the benefit of French-specific pretraining, while CamemBERTav2 matched CamemBERT-base at native 1,024-token context.Under 512-token truncation, the 0.4pp CamemBERT-base–CamemBERTav2 gap was not statistically reliable (McNemar p = 0.147).
  • Per-class performance: Sport achieved the highest in-distribution F1 at 0.964, while Économie was weakest at 0.744 and Société was the dominant error destination.Culture & Loisirs reached 0.924, International 0.902, Sciences & Technologies 0.753, and Société 0.796.
  • Input ablation: Full-text input outperformed headline-only input across all four model families, with the largest gap for TF-IDF and the smallest for CamemBERTav2.Headline-only input remained a deployment-constrained fallback for sources such as RSS feeds and paywalled archives.
  • Cross-publisher transfer: On the 2,100-article unseen-outlet pool, Économie remained weakest with precision 0.807, recall 0.517, and F1 0.630, whereas Sport reached F1 0.915.Société absorbed boundary-ambiguous texts, combining recall 0.810 with precision 0.577; six of seven categories achieved recall ≥0.810.
  • Zero-shot comparison: CamemBERT-base exceeded GPT-OSS-120B across zero-, five-, and seven-shot settings, with its advantage concentrated in Société and Économie.GPT-OSS-120B zero-shot scored mean accuracy 0.753 and macro-F1 0.758 on the unseen-outlet pool.

6 Discussion

FrenchNews-7 combines a reproducible, empirically derived seven-category taxonomy with publisher-native URL labels, LLM annotation for ambiguous cases, and held-out cross-publisher evaluation. Under distribution shift, Société absorbs boundary-ambiguous text while Économie reflects cross-publisher editorial discretion, motivating emphasis on robustness and reproducibility over boundary precision.

  • Contributions: FrenchNews-7 is presented as the first French benchmark combining a multi-publisher taxonomy, publisher-native URL labels, LLM annotation, and held-out evaluation across seven categories.The taxonomy can be reproduced for other media ecosystems using the same URL-slug derivation.
  • Cross-publisher robustness: ≈0.80 macro-F1: CamemBERT-base sustains this score under pooled mixed-class evaluation on unseen outlets.The evaluation discriminates among all seven categories under distribution shift.
  • Limitations: Société absorbs boundary-ambiguous text, whereas Économie’s difficulty reflects cross-publisher editorial discretion rather than recoverable classifier headroom.The release therefore prioritizes robustness and reproducibility over boundary precision.

Limitations

The study uses a deliberately coarse taxonomy and publisher-assigned URL slugs as reference labels, limiting category granularity and introducing label-construction constraints. Generalization beyond the observed time period, publishers, and francophone regions remains unverified, while classifier utility varies by downstream task.

  • Taxonomy and labels: The seven-class taxonomy collapses finer distinctions, including Environnement and Santé, into broader categories.This deliberately coarse design limits editorial-desk granularity.
  • Generalization scope: Three Appendix B analyses argue against publisher memorisation, while cross-publisher and temporal checks remain bounded to specified outlets and evaluation settings.These include per-publisher F1 breakdowns, Integrated Gradients attribution, publisher-token masking, a balanced 7-way test, category recall slices, and later articles from the same 13 publishers.
  • Taxonomy and labels: The 2,100-article held-out test uses publisher URL slugs as reference labels by construction rather than corrected human-audit labels.The blinded human audit shows recoverable semantic signal but does not revise the reference labels.
  • Generalization scope: Temporal generalization beyond 2022–2025, coverage outside 13 known France-based publishers, and validation on francophone media from Belgium, Switzerland, Quebec, or Africa remain unverified.Reported bootstrap and Wilson intervals quantify test-sample uncertainty only.
  • Downstream use: Classifier utility varies substantially by downstream research task, with Appendix E mapping three reliability tiers to concrete task types.The tiers are based on cross-publisher evaluation results.

Ethics Statement

The benchmark releases its model, labels, metadata, and evaluation artifacts while withholding verbatim article text and full headlines for copyright reasons. It is intended for editorial-desk classification, with acknowledged risks from correlated LLM labeling errors and residual outlet-style bias.

  • Open release and copyright: The CamemBERT-base model and labeled manifest are publicly released on Hugging Face, with article metadata including URLs, dates, publishers, categories, provenance, and SHA-256 text hashes.The manifest does not redistribute verbatim article text or full headlines.
  • Open release and copyright: Researchers can inspect labels and evaluation artifacts, while authorized users may re-fetch source pages under their own legal and institutional frameworks using reference collection scripts.The repository also includes the 74-rule slug lookup table and seven-category prompt template.
  • Scope and limitations: The benchmark targets cross-publisher editorial desk classification rather than person-level inference or fully general claims about French media structure.Annotation procedures are documented in §3.3–3.4.
  • Scope and limitations: Remaining risks include correlated LLM labeling errors and residual outlet-style bias.These limitations qualify interpretation of the benchmark beyond its intended classification task.

A Case Study: Validating Editorial Fingerprints at Scale

Applying CamemBERT-base to 1,940,734 held-out articles produced outlet-specific editorial fingerprints and rising topical differentiation from 2022 to 2025. A temporal-generalization check quantified prediction limits, motivating conservative error assumptions for longitudinal analysis.

  • Editorial fingerprints: 1,940,734 held-out articles across 13 outlets revealed distinct predicted topic fingerprints, with Société ranking first or second at every outlet.The 87,637 manifest articles were excluded by exact URL match before aggregation.
  • Dynamic agenda setting: 0.098 [95% CI: 0.094, 0.104] to 0.128 [0.122, 0.134] marked the increase in mean pairwise JS divergence between the first and final 24 months.The 48-month trend is consistent with growing topical differentiation, although major contemporaneous events could also drive realignment.
  • Temporal generalization: 57.2% to 96.9% recall spanned classes in the temporal-generalization check, from Économie to Sport.Six of seven classes had Wilson half-widths of at most 1.3 percentage points; Sciences & Technologies was wider at ±2.7 points because it was comparatively rare.
  • Temporal generalization: 0.598 recall on the temporal sample was 0.235 lower than the held-out pool’s 0.833, reflecting the latter’s balanced, structurally cleaner Sciences & Technologies slice.The temporal check used a stratified 100,000-article sample from the same 13 training outlets, while the held-out pool deliberately oversampled 300 Sciences & Technologies articles.
  • Uncertainty calibration: 17.9% empirical error on the parseable subset led to retaining ϵ = 0.20 as the rounded central estimate and reporting ϵ = 0.25 as a stress test.Errors were disproportionately absorbed by Société; non-parseable articles were enriched in Société (+13.8 points) and Économie (+5.5).

B Publisher-Level Analysis and Feature Attribution · C IPTC Media Topics Crosswalk

Publisher performance varied substantially without a significant training-size relationship, while attribution and masking indicate topical rather than publisher-specific decision cues. The IPTC crosswalk shows partial structural overlap and systematic category collisions in the observational FrenchNews-7 taxonomy.

  • B Publisher-Level Analysis and Feature Attribution: Macro-F1 ranges from 0.626 (TF1 INFO) to 0.906 (Ouest-France), spanning 0.280 points across 13 publishers.Le Monde, the largest training contributor at 37.8% (Ntest = 5,010), achieves macro-F1 = 0.849.
  • B Publisher-Level Analysis and Feature Attribution: Spearman ρ = 0.088 (p = 0.775, N = 13) indicates no statistically significant relationship between publisher training-set size and test macro-F1.The small N makes this result indicative rather than inferential.
  • B Publisher-Level Analysis and Feature Attribution: Integrated Gradients identified unambiguously topical leading tokens across all seven classes, including festival for Culture & Loisirs and football for Sport.The analysis used a stratified 199-article in-distribution test sample, a zero-embedding baseline, and 50 integration steps.
  • B Publisher-Level Analysis and Feature Attribution: No publisher domain stem, section header, or byline fragment appeared in any class top-10 attribution list.This supports the interpretation that the model attends to topical vocabulary rather than publisher-identifying surface cues.
  • B Publisher-Level Analysis and Feature Attribution: Aggregate ΔF1 = 0.000 after masking publisher tokens, with no per-class change in the 199-article sample.Only 4 tokens were masked, representing 0.004% of the body-token budget.
  • C IPTC Media Topics Crosswalk: 10 of 17 top-level IPTC Media Topics categories map to at least one FrenchNews-7 class, yielding 58.8% structural overlap.Seven categories lack a straightforward equivalent, including Environment, Labour, Religion, and Weather.
  • C IPTC Media Topics Crosswalk: Société absorbs four IPTC categories—Society, Crime/Law/Justice, Education, and Health—making it the largest crosswalk collision point.FrenchNews-7 is derived from newsroom routing URLs and deliberately collapses distinctions such as Environnement into Société and Sciences & Technologies.
  • C IPTC Media Topics Crosswalk: International has no IPTC equivalent because it represents geographic scope—foreign versus domestic routing—rather than an IPTC hierarchical topic.This illustrates that FrenchNews-7 includes an editorial-routing dimension absent from the IPTC hierarchy.

D Exploratory Cross-Taxonomy Evaluation · E Reliability Tiers for Downstream Use

The paper probes a non-French IPTC classifier against FrenchNews-7 without retraining, using mapped IPTC predictions and two macro-F1 accounting schemes. It also translates cross-publisher findings into three reliability tiers for downstream task selection.

  • D Exploratory Cross-Taxonomy Evaluation: EMMediaTopic, an XLM-RoBERTa-large model trained on 21,000 non-French articles across 17 IPTC categories, is evaluated on FrenchNews-7.
  • D Exploratory Cross-Taxonomy Evaluation: The cross-taxonomy probe uses the FrenchNews-7 test split without retraining the EMMediaTopic classifier.
  • D Exploratory Cross-Taxonomy Evaluation: Top-1 IPTC predictions are collapsed into FrenchNews-7 classes through the mapping specified in Table 9.
  • D Exploratory Cross-Taxonomy Evaluation: International is excluded because IPTC has no equivalent category for that FrenchNews-7 class.
  • D Exploratory Cross-Taxonomy Evaluation: Table 10 evaluates 13,146 test-split articles using macro-F1 with and without an unmapped-prediction penalty.
  • E Reliability Tiers for Downstream Use: Three reliability tiers map cross-publisher evaluation results to concrete downstream task types.

F Full-Context CamemBERTav2 Evaluation · G Pairwise Significance Tests

Full-context CamemBERTav2 matches CamemBERT-base across in-distribution and held-out evaluation, indicating the earlier 0.4pp gap arose from truncation rather than architecture. Pairwise significance testing compares model performance using corrected bootstrap and McNemar procedures across both evaluation settings.

  • F Full-Context CamemBERTav2 Evaluation: CamemBERTav2 was retrained with its native 1,024-token context using six epochs, learning rate 2 × 10−5, effective batch 32, and seed 42.Evaluation covered the in-distribution test split and held-out cross-publisher pool.
  • F Full-Context CamemBERTav2 Evaluation: CamemBERTav2 matches CamemBERT-base at full context on both evaluation settings, while CamemBERT-base remains the recommended deployment model.The comparison uses CamemBERT-base at 512 tokens, CamemBERTav2 at 512-token truncation, and CamemBERTav2 at native 1,024-token context.
  • F Full-Context CamemBERTav2 Evaluation: 0.4pp is the in-distribution gap reported under truncation, attributed to context-window limitation rather than architectural difference.The full-context comparison removes this discrepancy between CamemBERTav2 and CamemBERT-base.
  • G Pairwise Significance Tests: 10,000 bootstrap replicates with fixed seed 42 test macro-F1, while continuity-corrected McNemar statistics test per-article correctness.Holm–Bonferroni correction is applied independently within each evaluation family, and macro-F1 averages over seven gold label classes.
  • G Pairwise Significance Tests: 13,146 articles define the in-distribution pairwise significance evaluation, using paired bootstrap tests on macro-F1 and McNemar’s tests on correctness.Table 13 reports seed 42 results with Holm correction over all 12 raw p-values.
  • G Pairwise Significance Tests: 2,100 unseen-outlet articles define the held-out pairwise significance evaluation, with Holm correction applied within the test family.GPT-OSS is tested per seed against CamemBERT, while seed 11 is representative for cross-model pairs.
  • G Pairwise Significance Tests: GPT-OSS-s11 versus Llama shows bootstrap confidence intervals including 0 but McNemar p = 0.009, reflecting differing sensitivity to correlated per-article errors.Bootstrap macro-F1 averaging smooths correlated errors, whereas McNemar’s test remains sensitive to them.
Loading 2608.18097v1…