Source-linked AI summary
Recognizing Is Not Reversing: A Controlled Inversion Test of Fact-Preserving News Framing
Yi Liu
TL;DR
Existing framing studies assess generation, detection, or apparent neutralization, but not whether LLMs can reverse known presentation changes while preserving facts. This paper introduces controlled inversion over evaluative lexis, agency realization, and discourse salience, finding that factual fidelity and framing recovery remain distinct capabilities.
Problem
Existing framing studies measure whether framing is produced, recognized, or reduced, but not whether a model can reverse a known presentation change while preserving facts.
Method
The paper constructs fact-preserving paired news variants with logged framing edits and separately evaluates framing recognition, factual preservation, and intervention reversal.
Results
Factual fidelity remains substantially easier than presentation-structure recovery, with model rankings differing across recognition and intervention reversal.
Takeaways & Limitations
Controlled inversion makes framing recovery directly measurable against recorded edits, supporting separate reporting and stage-specific model selection rather than a single quality score.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly used to analyze and rewrite news, yet current framing studies mainly evaluate generation, detection, or whether rewritten text appears more neutral. They do not directly show whether a model can undo a known framing transformation while keeping the facts fixed. We introduce a controlled inversion test over three established textual realizations of framing: evaluative lexis, agency realization, and information salience. Across 60 news articles and three intervention strengths, this yields 540 paired variants with preserved atomic facts and recorded edits. Across Qwen, DeepSeek, and Kimi, factual preservation remains near 0.84, whereas intervention reversal is 0.044--0.068. Even when both framing type and direction are recognized correctly, pooled reversal reaches 0.071. These results reveal a clear separation between factual fidelity, framing recognition, and framing inversion: recognizing how an article is framed does not imply that the framing can be undone.
1 Introduction
The paper reframes news-framing analysis as a controlled inversion problem: can models identify a known presentation change and undo it while preserving atomic facts? It separates framing recognition, factual fidelity, and intervention reversal as distinct capabilities.
- 1 Introduction: Three realization levels ground the interventions: evaluative lexis, agency realization, and information salience.These correspond to lexical, clause-level, and discourse resources in news framing.
- 1 Introduction: The benchmark makes framing transformations observable by constructing fact-preserving article pairs with recorded presentation edits.A transformed article y is generated from reference x while preserving its atomic factual inventory.
- 1 Introduction: Figure 1 summarizes the central question of whether a fact-preserving frame can be recognized and then reversed.The figure presents controlled inversion as the paper’s motivating test.
- 1 Introduction: The test distinguishes recognizing a framing intervention from successfully reversing it.This separation addresses what aggregate bias scores are not designed to isolate.
2 Related Work
Prior work studies framing, bias, detection, generation, neutralization, and rewriting, but the paper positions controlled inversion as a distinct evaluation of whether known presentation changes can be undone.
- 2 Related Work: News-framing research links interpretation to selection, salience, responsibility, and recurring structural, thematic, and rhetorical patterns.Linguistic accounts additionally examine evaluative stance, actor representation, and discourse organization.
- 2 Related Work: LLM studies evaluate generated bias, framing detection, summary fairness, neutralization, and reframing across several complementary benchmarks.Related work also examines prompt sensitivity and the effects of explicit reasoning on model behavior or faithfulness.
- 2 Related Work: Figure 2 connects the controlled benchmark and fixed D0-to-R0 interface with a real case illustrating recognition without reversal.Its right panel shows DeepSeek recognizing a favorable salience intervention while reconstruction leaves the injected ordering unchanged, with IRR = 0.
3 Controlled Framing Inversion
The controlled benchmark generates validated, fact-preserving framed news variants across three operators and intervention strengths, then evaluates detection and reconstruction through a fixed interface. Metrics separately assess factual preservation, framing recognition, and reversal of logged edits.
- 3 Controlled Framing Inversion: The generator records the operator, strength, direction, and injected edit set while preserving the reference article’s atomic factual inventory.Operators target evaluative wording, agency realization, or information salience without changing the underlying facts.
- 3 Controlled Framing Inversion: Detection predicts framing properties first, and reconstruction then receives the model’s own detection output rather than the ground-truth intervention.This fixed handoff tests whether recognized framing can be recovered through the model’s complete pipeline.
- 3 Controlled Framing Inversion: The benchmark contains 60 news sources and 540 validated framed variants produced by three operators at three strengths, plus 60 clean controls.Sources contain 6–10 atomic facts, and each is canonicalized to 180–260 words.
- 3 Controlled Framing Inversion: The protocol fixes task wording, field order, and output schema across model families before evaluation.Detection and reconstruction use the fixed D0 and R0 interfaces.
- 3 Controlled Framing Inversion: Reconstruction is scored with factual preservation and intervention reversal, where reversal counts injected edits removed from the reconstruction.Conditional analysis requires the predicted operator and direction to match ground truth, with uncertainty estimated using clustered bootstrap replicates.
4 Experiments and Results
Across three model families, factual preservation substantially exceeds framing reversal, and recognizing framing does not ensure its removal. Reversal varies by mechanism and intervention strength, while native thinking leaves the central separation intact.
- 4.2 Facts survive while frames persist: FactF1 remains near 0.84, whereas IRR ranges from 0.044–0.068 across model families.The benchmark separates factual fidelity from framing recovery; Qwen leads overall IRR, Kimi leads Macro-F1 and exact recognition, and DeepSeek has the lowest clean false positive rate.
- 4.3 Recognizing is not reversing: Pooled IRR rises from 0.049 to 0.071 when exact type and direction recognition is correct, but 92.9% of logged interventions remain unreversed.Only 75 of 1,052 injected edits are reversed in the exact-recognition subset.
- 4.4 Where framing is encoded matters: Lexical framing has the highest class F1, whereas salience has the lowest reversal rate across all three model families.Agency has higher IRR than lexical framing for DeepSeek and Kimi and ties lexical framing for Qwen, showing that cue visibility does not determine edit recovery.
- 4.5 Intervention strength: Increasing intervention strength raises direction accuracy and IRR across models while FactF1 changes much less.At high strength, IRR remains below .09 for every model.
- 4.3 Recognizing is not reversing: Native thinking changes IRR heterogeneously—+.025 for DeepSeek, -.021 for Qwen, and +.000 for Kimi—while FactF1 changes by less than .01.The recognition–inversion separation remains visible under the second inference configuration.
- 4.6 Regularities across models: Across models, exact recognition raises IRR, salience is least reversible, and stronger interventions improve recovery while factual scores remain comparatively stable.The benchmark also separates model leadership across axes: Kimi leads recognition, Qwen reversal, and DeepSeek clean calibration.
5 Discussion
The discussion frames controlled inversion as a multi-axis measurement and auditing framework rather than a single quality score. Separating recognition, reconstruction, and factual preservation reveals stage-specific model strengths and supports verification of whether intended representational changes were actually reversed.
- 5 Discussion: Separating recognition, factual preservation, and intervention reversal exposes distinct capabilities that aggregate quality scores would conceal.The framework measures each stage under fixed facts and a logged edit map.
- 5 Discussion: Kimi’s exact recognition (.312) exceeds Qwen’s (.130), while Qwen’s IRR (.068) exceeds Kimi’s (.054).The reversal in model ordering shows that detection and recovery should remain separate stages.
- 5 Discussion: The decomposition yields a verifiable editing workflow in which detection identifies the mechanism, reconstruction performs the edit, and IRR checks the intended change against the known intervention.This evaluates correction by recorded edits rather than surface fluency alone.
- 5 Discussion: Lexical evaluation is more visible than agency and salience, which require tracking actor foregrounding and information placement.The logged presentation changes provide supervision targets for each framing mechanism.
6 Conclusion
The conclusion presents controlled inversion as a fact-preserving benchmark that directly measures whether known framing edits are recovered. Across model families, stronger interventions are easier to recover and salience is least reversible, while factual fidelity remains stable.
- 6 Conclusion: Controlled inversion separates factual fidelity, framing recognition, and framing inversion across evaluative lexis, agency realization, and discourse salience.Its objective scores reconstruction against recorded forward interventions rather than apparent neutrality.
- 6 Conclusion: Salience is the least reversible operator, while stronger interventions improve recoverability and factual fidelity remains stable.The benchmark therefore distinguishes recovery difficulty from preservation of the underlying news facts.