Source-linked AI summary
Framing by Wording, Framing by Selection: A Large-Scale Two-Dimensional Audit of French News Headlines, 2022-2025
Amr Sobhy
TL;DR
Computational framing measures often blur how headlines word issues with which story forms outlets repeatedly select. This paper separates those dimensions, validates a French headline classifier, and audits outlets at scale. The results show that the dimensions overlap without becoming interchangeable, while recalibrated thresholds yield lower corpus-level salience estimates than default thresholds.
Problem
Computational framing measures often combine wording and topic selection, limiting the ability to distinguish salience from selection and their interaction.
Source: §1 · p. 1Method
Results
TF-IDF replication yields r=.667 rather than .736, while outlet profiles show distinct salience-led and selection-led patterns; recalibrated corpus rates are 34.6% any-salience and 31.3% high-charge.
Source: §4 · p. 6, §4.3 · p. 8, §5 · p. 9Takeaways & Limitations
Outlet profiles that look similar on one axis can reflect different editorial mechanisms, so the two-dimensional framework preserves distinctions a single score would miss.
Source: §5 · p. 9This is a descriptive research audit, not a regulatory compliance instrument, and its group labels identify explicit lexical mentions rather than sentiment, endorsement, or hostility.
Source: Ethics, Reproducibility, and Data… · p. 11
Abstract
from arXiv · showhide
News headlines frame public issues both by what they select and by how they word it, yet computational framing work typically collapses these operations into a single score. We introduce a two-dimensional framework that separates salience framing, measured through four wording devices (loaded vocabulary, blame attribution, threat framing, rhetorical question), from selection framing, measured through outlet-level story-form and high-charge distributions. We build a 10,000-headline French supervision set using three LLM annotators with majority-vote resolution and human arbitration, validate the labels against two annotator-independent blind human studies, and apply the strongest classifier to 902,111 deduplicated headlines from 25 French outlets (2022-2025). Three main findings emerge. First, salience and selection divergence are positively correlated yet leave nearly half of outlet-level variance unexplained, populating interpretively distinct off-diagonal cells in a four-cell outlet typology. Second, default classification thresholds systematically inflate corpus-level salience estimates; a precision-floor recalibration protocol corrects this distortion. Third, group-mention analysis reveals sharply unequal salience contexts: headlines mentioning Jews, the Far-right, and Muslims carry the highest detected salience rates, which broad event-context composition does not fully explain (residuals are descriptive, not same-event causal estimates; per-group lexicon precision is reported alongside). To our knowledge, this is the largest framing-focused French headline audit to date; we release the supervision set, lexicons, and analysis code.
1 Introduction
Headlines frame issues both through wording and through repeated editorial selection, but computational measures often blur these distinct operations. This study separates the two in a large French headline audit and asks how they vary across outlets and group-mention contexts.
Why headlines
- Headlines matter because many shared news URLs are never clicked, and experimental work finds they can shape memory and inference even when readers later read the article.Their compactness also makes editorial choice unusually visible.Source: §1 · p. 1
Two editorial operations
- A headline can frame an event through charged wording and through repeated selection of particular event types for particular groups.The examples contrast “Migrants are invading” with “Migrants are arriving” and describe a far-right aggregator filling four in five headlines with crime stories about identifiable groups.Source: §1 · p. 1
Research gap
- Computational framing measures often capture wording or topic selection separately, leaving unclear whether outlet differences arise from salience, selection, or their interaction.The study tests whether salience and selection are dissociable rather than one construct measured twice.Source: §1 · p. 1
Contributions
- The paper contributes a 10,000-headline supervision set, a precision-floor recalibration protocol, and an audit of 902,111 headlines showing overlap without collapsing the two axes.Source: §1 · p. 2
2 Framework and Related Work
The framework treats salience as wording and selection as outlet-level distributions of story forms and high-charge coverage. This measurement split preserves distinct operations that prior framing and media-bias measures often combine or leave unmeasured.
Operational framework
- Salience framing measures wording devices, while selection framing measures repeated story-form choices and the share of those forms that are high-charge.The four primary wording devices are loaded vocabulary, blame attribution, threat framing, and rhetorical question; us-vs-them is supplementary.Source: §2 · p. 2, Table 1 · p. 3
Measurement boundary
- Published-headline distributions cannot reveal which events an outlet did not select, so selection framing is conditional on coverage.Outlets with identical story-form distributions may still differ in the events they chose to cover.Source: §2 · p. 2
Measurement schema and prior work
- The schema links the four primary devices to frame functions, while prior work often uses a single frame typology, one bias dimension, or issue filtering rather than wording-versus-selection measurement.The table distinguishes non-mutually-exclusive wording devices from selection-level story form and high-charge.Source: §2 · p. 2, Table 1 · p. 3
Prior work
- Existing English-language auditing resources address related questions at scale but do not separate wording from selection mechanisms.Source: §2 · p. 2
French-language setting
- French media research motivates the group inventory, while earlier French framing studies were smaller or issue-bound than this headline-only, multi-outlet audit.The paper characterizes its corpus as unusually large for framing-focused work in French.Source: §2 · pp. 2–3
Why the distinction matters
- A single framing-intensity score cannot preserve the distinction between repeated coverage, which affects accessibility, and charged wording, which shapes interpretive applicability.Source: §2 · p. 3
3 Data and Method
The study constructs a French headline supervision set, validates its labels independently, and applies a mixed-encoder classifier to a large outlet corpus. Its method combines calibrated thresholds, separate salience and selection labels, and explicit limits on what headline-only analysis can observe.
Corpora
- The corpus contains 902,111 deduplicated headlines from 25 outlets, while the 10,000-headline supervision set is split into 6,999 training, 1,500 validation, and 1,501 test items.The analysis corpus samples Politics, Economy, and Society headlines across outlet, section, and year for 2022–2025.Source: §3 · p. 3
Annotation
- Three schema-constrained LLM annotators label the supervision set, with binary labels resolved by 2/3 vote and 642 three-way story-form conflicts assigned to a human arbitrator.Binary conflicts are not arbitrated because the models may share errors; blind human studies provide the control for binary heads.Source: §3 · p. 3
Labels
- Salience is multi-label, while selection uses a binary high-charge label and a 10-class story-form taxonomy; target-group mentions are analyzed separately with a 219-term lexicon.Story form classifies the journalistic action reported rather than only the subject domain.Source: §3 · p. 4
Thresholding
- Validation-only thresholds maximize F1 subject to precision floors assigned by annotator-agreement tier before validation metrics are observed.Floors are .85 for rhetorical question, .70 for loaded vocabulary, blame attribution, and charge band, and .60 for threat framing and us-vs-them.Source: §3 · p. 4, Table 2 · p. 4
Models
- .742 mean salience+charge F1 (XLM-R) vs .726 (CamemBERT) and .476 (TF-IDF); XLM-R story-form macro-F1 is .731, while CamemBERT scores .900 vs .841 on rhetorical questions.The production ensemble uses CamemBERT only for rhetorical-question prediction.Source: §3 · p. 4, Table 3 · p. 5
Independent validation
- Two unaffiliated annotators independently re-annotate a stratified 499-headline held-out sample, with majority-vote human–LLM κ ranging .608–.869 across the four primary devices.Any-salience human–LLM agreement declines only modestly for the period most exposed to possible pretraining overlap, and human–human κ is .680 on the composite.Source: §3 · pp. 4–5, Table 4 · p. 5
Group analysis
- The 219-term group lexicon detects 134,046 headlines, and its precision is at least .83 for 9 of 12 groups; Jews, LFI, and Unions have lower reported precision.Indirect or paraphrastic references are missed by design; the two-human audit reports κ=.94–.98 on a representative tier and κ=.83–.85 on a harder tier.Source: §3 · p. 5
Analysis plan and scope
- The outlet analysis uses Jensen–Shannon divergence from corpus baselines and median splits as a cartographic typology, not as evidence of categorical differences.The measures use different bin counts, so their magnitudes are not directly comparable; headline-only classification also strips context that may disambiguate blame or threat attribution.Source: §3 · pp. 4–5
4 Results
Across French outlets, salience and selection divergence are positively associated but remain distinct, while group-mention contexts and temporal patterns show further variation. The results also report calibrated corpus-level rates and robustness checks that qualify these findings.
4 Results
- 34.6% of headlines contain at least one salience device and 31.3% are high-charge, with rates lower than default-threshold estimates after precision-floor recalibration.The 34.6% figure is best read against the French corpus baseline, not as a universal prevalence benchmark.Source: §4 · p. 6
4.1 Outlet Profiles and Dissociability
- Outlet profiles distinguish threat-blame, interrogative-evaluative, and institutional-blame signatures; double-distinctive outlets range from 7.2% to 72.6% any-salience.Ouest-France has the panel’s lowest any-salience and high-charge rates; Blast’s profile is illustrative because it has 646 headlines, and Causeur’s ELITE-heavy profile warrants caution.Source: §4.1 · p. 6
- Salience distinctiveness explains 54.2% of outlet-level selection variance (r=0.7361), leaving 45.8% unexplained; stable off-diagonal profiles include JDD and TF1 INFO.Only seven near-boundary outlets shift under ±20% median changes, and the reported 95% bootstrap interval is wide ([.503, .936], N=25).Source: §4.1 · p. 6, Table 5 · p. 6, Figure 1 · p. 7
4.2 Group-Mention Salience Contexts
- 78.7% of headlines mentioning Jews contain detected salience and 79.8% are high-charge; year×story-form reweighting leaves a descriptive +29.1 pp residual.The residual indicates broad story-form composition does not fully explain the pattern, not same-event causal targeting; the group lexicon has .70 precision and .97 recall.Source: §4.2 · p. 7, Table 6 · p. 8
- Far-right mentions show 77.7% raw salience and a +27.7 pp residual, while Muslims show 66.4% and +19.8 pp; these patterns reflect event contexts, not uniform targeting.Device profiles also differ: Jews show the strongest multi-device signal, Police are blame-dominated, and Far-right mentions are elevated across three adversarial devices.Source: §4.2 · p. 8
4.3 Robustness
- Jews, the Far-right, and Muslims remain the three highest-salience groups in every year from 2022 to 2025, with annual lifts of ×1.78–×2.35.The pattern is temporally persistent, while group differences are reported as significant under BH-FDR-corrected tests whose intervals do not propagate classifier uncertainty.Source: §4.3 · p. 8, §4.2 · p. 8
- With a TF-IDF story-form model, the salience-selection correlation falls from r=0.736 to 0.667, while Spearman correlation remains stable at 0.741 versus 0.747.The within-outlet panel coupling is positive but treated as confirmatory (β=0.758, 95% CI [−.11, 1.63], p=.084); a leave-one-out correction gives r=0.721.Source: §4.1 · p. 7, §4.3 · p. 8
- Any-salience rises from 31.7% in 2022 to 38.1% in 2025; +4.86 pp of the +6.34 pp change is assigned to within-story-form wording and +1.20 pp to composition.The pattern holds in nine of ten story forms and under joint outlet × section × story-form standardisation; standardisation controls event category, not intensity.Source: §4.3 · pp. 8–9
5 Discussion
The two-dimensional audit distinguishes outlets with similar aggregate scores but different wording and selection profiles. It also shows that the present headline-only analysis cannot determine whether observed patterns reflect stable editorial stance, event context, or both.
5 Discussion
- TF1 INFO and JDD illustrate opposing profiles: TF1 INFO has elevated rhetorical divergence but near-baseline selection divergence, while JDD has clearer selection divergence but almost none in wording.Fdesouche is distinctive on both dimensions; the authors state that separating stable editorial stance from event-context concentration requires richer annotation.Source: §5 · p. 9
- The two-dimensional framework resolves ambiguity in single-axis audits because similarly scored outlets may differ sharply in their underlying mechanism.The authors connect this distinction to separating wording-device load from story-form concentration.Source: §5 · p. 9
- The precision-floor recalibration protocol, with its released threshold log, extends to longitudinal multi-label framing audits, though cumulative default-threshold distortion remains untested.Source: §5 · p. 9
Limitations
The audit measures published headlines and aggregate editorial patterns, not audience effects or first-level event selection. Its estimates also have specific limits from annotation uncertainty, event-context confounds, baseline construction, and outlet support.
Limitations
- The headline-only audit cannot establish whether editorial patterns produce audience accessibility or applicability effects, which require experimental work.It also does not capture article-body framing or cross-platform dynamics.Source: Limitations · p. 9
- Selection framing measures published-headline story-form concentration, so outlets with similar scores may still differ in which real-world events they cover.The two-dimensional typology may also leave distinct mobilisation-register and adversarial-register outlets grouped together.Source: Limitations · p. 9
- The v6 ensemble is recall-oriented, so corpus-level device rates should be read as upper-bound prevalence estimates; loaded vocabulary also has the lowest human-human agreement among the primary heads.The rhetorical-question head indexes headline form rather than rhetorical intent, and headline-level interpretation remains difficult for implicit causality and rhetorical register.Source: Limitations · p. 9
- The balanced development set and proportionally stratified production corpus create estimated precision drift; smaller-outlet estimates, especially Blast’s, are case profiles rather than equally precise population estimates.Source: Limitations · p. 10
- Group detection captures explicit lexical mentions only, and its lexicon is a reproducible lower bound rather than an exhaustive account of group references.Year-by-story-form reweighting is coarse, not a same-event causal design; separating event intensity from outlet framing would require matched same-event corpora or event clustering.Source: Limitations · p. 10
- Period-stratified agreement shows no aggregate evidence of contamination-inflated annotations, but cannot rule out residual priors for specific high-salience events.For Israel/Gaza or election coverage requiring contamination-free labels, the authors direct users to the human-validated N=499 subset.Source: Limitations · p. 10
- Merging ELITE into OTHER leaves Causeur double-distinctive, while a separate taxonomy perturbation shifts BFMTV and Marianne but no focal outlet.Causeur’s original assignment partly relies on an ELITE-heavy profile with lower arbitration agreement.Source: Limitations · pp. 10–11
- Loaded-vocabulary and us-vs-them estimates fall below their production-precision floors, while the corpus baseline partly reflects the weight of three high-volume outlets.Leave-one-out rankings remain robust, with only two near-boundary outlets changing typology cells.Source: Limitations · p. 11
Ethics, Reproducibility, and Data Availability
The study audits public editorial output, not private individuals, and its group labels describe explicit mentions rather than sentiment or intent. It documents misuse risks and releases reproducibility materials while withholding headline text and model-training assets.
Ethics, Reproducibility, and Data Availability
- The audit concerns public editorial output and does not infer protected attributes of private individuals or support individual profiling or content-moderation decisions.Source: Ethics, Reproducibility, and Data… · p. 11
- Group-mention rates describe the rhetorical and event contexts around groups, not editorial sentiment, intent, or uniform targeting.High salience for Jews, the Far-right, and Muslims reflects distinct coverage contexts; the Fdesouche exclusion check finds these elevated rates distributed across the broader outlet panel.Source: Ethics, Reproducibility, and Data… · p. 11
- The authors identify political misuse risks in which group salience could be misrepresented as over-representation, legitimization, or persecution rather than event-driven coverage context.LLM-assisted annotations are not treated as ground truth; the final supervision set uses majority-vote resolution and selective human arbitration.Source: Ethics, Reproducibility, and Data… · p. 11
- The 10,000-headline supervision set is released as outlet, date, section, and URL pointers, with 516 human-validation headlines disclosed verbatim.The 902,111-headline inference layer contains predictions and metadata but no headline text; group lexicons, schema, panel materials, and analysis scripts are also released.Source: Ethics, Reproducibility, and Data… · pp. 11–12
- Model-training and threshold-calibration code is not released because it is bound to the licensed corpus and non-redistributable trained checkpoints.Source: Ethics, Reproducibility, and Data… · p. 12
(a) Sparse-cell exclusion sensitivity
The temporal-panel robustness table reports stable within-outlet coupling across sparse-cell exclusion thresholds, while cluster-robust estimates are positive but not individually significant.
(a) Sparse-cell exclusion sensitivity
- Within-outlet coupling remains stable across sparse-cell exclusion thresholds, while cluster-robust estimates are positive but not individually significant.The table characterizes the panel as confirmatory rather than a stand-alone test.Source: Table 29 · p. 20