Source-linked AI summary
MANTIS: Multi-Axis News Transformations for Instance-Wise Stress Testing and Worst-Case Training in Fake News Detection
Dexin Liu, Fan Yang, Zimu Wang, Xunfei Zhu, Bolin Ren, Bingyan Zhang, Wei Wang, Qi Chen
TL;DR
Fake-news detection can rely on surface cues that make authoritative presentation a useful vehicle for deceptive claims. MANTIS-Bench tests claim-preserving rewrites across multiple presentation dimensions, and MANTIS trains on each instance’s worst-case view. The approach reports strong performance across original and transformed news, with lower UASR, including a reduction from 8.61% to 5.06% on GossipCop.
Problem
Human credibility assessments can mistake fluent, professional, neutral presentation for truth, while LLMs make authoritative packaging of deceptive claims inexpensive to produce.
Source: §1 · p. 1Method
MANTIS-Bench pairs original news with claim-preserving views across presentation dimensions, while MANTIS jointly trains on each original and its instance-specific worst-case transformed view.
Source: §1 · p. 2, §3.3 · p. 4Results
MANTIS showed strong performance and lower UASR across PolitiFact and GossipCop; on GossipCop, it reduced UASR from 8.61% to 5.06%.
Source: §5.1 · p. 6, §6 · p. 8Takeaways & Limitations
The findings support multi-axis evaluation and case-specific robust training as important for reliable fake-news detection.
Source: §6 · p. 9The evaluation covers three benchmark datasets, and six offline views cannot represent every presentation variation encountered in practice.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Automated fake news detection is vital on social media, where fake news is easily rewritten to fool detectors. However, existing approaches typically rely on superficial writing cues or single-style text rewrites, resulting in poor robustness under diverse cosmetic modifications. We construct MANTIS-Bench, a multi-axis presentation-attack benchmark that pairs original news with rewritten views across writing style, emotional expression, and dissemination framing. We further propose MANTIS, an instance-adaptive adversarial framework enabling dynamic worst-case training tailored to each news article with zero inference-time overhead. This reframes fake news defense from heuristic format smoothing into targeted, fact-invariant credibility assessment. Experiments on MANTIS-Bench show that MANTIS reduces the Union Attack Success Rate (UASR) by 3.55 percentage points relative to SheepDog on GossipCop and consistently outperforms prior single-axis defenses on Macro-F1 while maintaining competitive clean accuracy, demonstrating the effectiveness of multi-axis and instance-adaptive worst-case training.
1 Introduction
Fake-news detection can be vulnerable to presentation cues even when underlying claims remain unchanged. The paper addresses this gap with a multi-axis benchmark and instance-wise worst-case training.
1 Introduction
- Surface cues such as fluent phrasing, professional syntax, and neutral tone can be mistaken for truth, while LLMs make deceptive claims in authoritative packaging inexpensive to produce.The passage links these tendencies to cognitive biases and undermined social trust.Source: §1 · p. 1
- Prior defenses target style or sentiment individually, but robustness to one perturbation family does not guarantee robustness to others.The paper motivates a unified multi-axis approach to address these cross-axis blind spots.Source: §1 · pp. 1–2
- Presentation changes can alter detector predictions despite preserving the core claim, while uniform rewrites may be suboptimal because instances can be vulnerable to different transformations.The cited transformations include changes to style, emotion, and dissemination framing.Source: §1 · p. 2
- MANTIS-Bench pairs original news with claim-preserving rewrites across writing style, emotional expression, and dissemination framing, and MANTIS selects each instance’s most challenging view for training.Training jointly optimizes the original and selected views, without inference-time overhead.Source: §1 · p. 2
2 Related Work
Prior fake-news robustness work has addressed domain, temporal, and spurious-correlation challenges, while presentation defenses have generally targeted individual axes. The paper situates MANTIS as a multi-axis, instance-wise alternative that confines transformations to offline training.
2 Related Work
- Earlier robust fake-news methods addressed domain heterogeneity, temporal drift, or entity shortcuts, but did not explicitly address presentation-level shortcuts under claim-preserving changes.Examples include MDFEND and M3FEND for domain heterogeneity, FTT for temporal trends, and ENDEF for entity shortcuts.Source: §2 · p. 2
- SheepDog and AdStyle target style, while AdSent targets sentiment; these single-axis defenses do not explicitly consider instance-specific worst-case variants in a unified multi-axis space.This leaves complementary presentation vulnerabilities as the motivation for MANTIS.Source: §2 · p. 2
- LLM-assisted detectors include rationale generation and multi-step verification, while MANTIS keeps multi-axis transformations and instance-wise worst-case optimization in offline training for single-pass inference.The overview describes six offline views and training on the original news plus each instance’s worst-case view.Source: §2 · p. 2, Figure 2 · p. 3
3 Methodology
This section defines claim-preserving transformations, introduces MANTIS-Bench across three presentation dimensions, and describes instance-wise worst-case training. GossipCop diagnostics show transformation-specific failures and Macro-F1 degradation across detectors.
3.1 Preliminaries
- A transformation is claim-preserving when it retains the original core claim, so the transformed view receives the original news's veracity label.Source: §3.1 · p. 3
3.2 MANTIS-Bench
- MANTIS-Bench pairs original news with rewritten views under core-claim preservation across writing style, emotional expression, and dissemination framing.The three presentation factors are brought into one transformation space.Source: §3.2 · p. 3
- 41 style-specific, 42 emotion-specific, and 82 dissemination-specific single-transformation failures among originally correct Llama-3.1-8B-Instruct cases reveal complementary vulnerabilities; all four detectors lose Macro-F1 on at least one dimension.The most damaging dimension varies by detector.Source: Figure 4 · p. 4, §3.2 · p. 4, Figure 3 · p. 4
3.3 The MANTIS Framework
- MANTIS evaluates six fixed offline views, selects the highest-loss view separately for each case under the current detector, and jointly trains on original and selected views.The selected view depends on both the news instance and detector parameters.Source: §3.3 · pp. 4–5
- At inference, the learned detector classifies unseen news directly with a single forward pass and no transformed-view construction.Source: §3.3 · p. 5
4 Experimental Setup
MANTIS-Bench pairs original news with transformed views across three datasets and presentation dimensions, and its construction is evaluated for claim preservation and target realization. Experiments compare detector families using per-view performance and a union vulnerability metric.
4.1 Benchmark Construction and Quality Evaluation
- MANTIS-Bench pairs original news with transformed views across PolitiFact, GossipCop, and Weibo21, which is used only for cross-lingual evaluation.The English datasets use balanced temporal splits; Weibo21 has a topic-balanced evaluation subset of 1,350 cases across nine topics.Source: §4.1 · p. 5, Table 1 · p. 5
- 58 of 60 transformed pairs (96.7%) preserved factual claims, and 56 of 60 (93.3%) met both claim-preservation and target-realization criteria.Two annotators evaluated 60 pairs; agreement was 95.0% (κ = 0.79) for claim preservation and 93.3% (κ = 0.76) for target realization.Source: §4.1 · p. 5
4.2 Baselines
- Baselines include supervised encoders, zero-shot LLMs, and robust detectors SheepDog and AdSent, evaluated on the same original and transformed views.SheepDog targets style variation, while AdSent targets sentiment-related cues.Source: §4.2 · pp. 5–6
4.3 Evaluation Metrics
- UASR measures transformation-induced failures among cases correctly classified originally, while Accuracy and Macro-F1 measure performance on each view.Higher Accuracy and Macro-F1 and lower UASR indicate better performance.Source: §4.3 · p. 6
4.4 Implementation Details
- Supervised models train for five epochs, while MANTIS uses LoRA on Llama-3.1-8B-Instruct and its results average three runs.Supervised results average ten seeds; MANTIS uses two GossipCop epochs and the third-epoch PolitiFact checkpoint.Source: §4.4 · p. 6
5 Experimental Results
MANTIS improves robustness across the evaluated datasets, transformations, and cross-lingual setting, with ablations supporting the combined training objective. Results also show that vulnerabilities vary by transformation and news topic.
5.1 Main Results
- 7.51% from 8.64% (PolitiFact) and 5.06% from 8.61% (GossipCop) UASR; MANTIS leads Macro-F1 across GossipCop views and Accuracy and Macro-F1 across PolitiFact views.On GossipCop, AdSent retains slightly higher original-view Accuracy; Table 2 reports the main comparisons.Source: §5.1 · p. 6, Table 2 · p. 7
5.2 Ablation Study
- Removing original-view loss lowers original-view performance, removing adversarial-view loss produces the highest UASR, and random selection raises UASR to 9.83% and 8.95%.The respective random-selection UASR values are for PolitiFact and GossipCop; the complete objective achieves the best result in every column.Source: §5.2 · p. 7
5.3 Generalization to SheepDog Style Attacks
- MANTIS outperforms SheepDog on all four publisher-style attack sets, while its advantage over AdSent varies across sets.MANTIS performs better on Sets C and D; AdSent remains stronger on Sets A and B.Source: §5.3 · pp. 7–8, Table 4 · p. 8
5.4 Cross-Lingual Generalization on Weibo21
- 18.80% from 29.42% UASR on Weibo21; English-trained MANTIS also exceeds AdSent in Accuracy and Macro-F1 across original and transformed views.The evaluation applies models trained on English GossipCop to Chinese Weibo21 without further training.Source: Table 5 · p. 8, §5.4 · p. 8
- Across nine Weibo21 topics, transformations reduce zero-shot LLM performance, with dissemination framing generally causing the largest degradation.Topic difficulty varies, so vulnerability depends on both transformation type and the underlying news content.Source: Figure 6 · p. 8, §5.4 · p. 8
6 Conclusion
The paper introduces MANTIS-Bench for claim-preserving presentation changes and MANTIS for instance-wise worst-case training. Results span original and transformed news, with lower UASR, transfer to unseen style attacks, and improved but challenging cross-lingual generalization.
6 Conclusion
- MANTIS-Bench pairs original news with complementary writing-style, emotional-expression, and dissemination-framing views, while MANTIS trains on each case’s most challenging transformed view alongside the original.The conclusion frames both contributions as addressing robust detection under claim-preserving presentation changes.Source: §6 · p. 8
- Experiments on PolitiFact and GossipCop show strong performance across original and transformed news, together with consistently lower UASR.Source: §6 · p. 8
- SheepDog publisher-style attacks show transfer to unseen style transformations, while Weibo21 results indicate improved but still challenging cross-lingual generalization.Source: §6 · pp. 8–9
- The paper concludes that multi-axis evaluation and case-specific robust training are important for reliable fake news detection.Source: §6 · p. 9
Limitations
The authors bound MANTIS to three presentation dimensions, six offline-generated views, and three benchmark datasets, noting limited human review and remaining Weibo21 challenges. The appendices specify the detection, rewriting, and preservation-judging prompts used in evaluation.
Limitations
- MANTIS covers three presentation dimensions with six offline LLM-generated views, but cannot represent every presentation variation; human review is limited, generation artifacts may remain, and evaluation spans three datasets.Moderate absolute Weibo21 performance indicates simultaneous language, domain, and topic shifts remain challenging.Source: Limitations · p. 9
A Prompts
- The appendix provides prompts for fake-news detection, claim-preserving rewriting, and transformation-quality assessment.Bracketed uppercase text marks runtime placeholders, and the class names remain real and fake throughout.Source: A Prompts · p. 11
A.1 Fake News Detection Prompt
- The same binary detection prompt is used for zero-shot LLMs and MANTIS; for Weibo21, the English prompt is retained while the article is Chinese.Source: §A.1 · p. 11
A.2 English Transformation Prompts
- English rewriting combines shared claim-preservation instructions with a target-specific instruction and the original article, without giving the generator the label.Labels are used outside the prompt to select diagnostic transformation direction; the shared instructions prohibit changing the central claim or material factual details.Source: §A.2 · p. 11
- English writing-style and emotion prompts change only their target dimension while preserving other presentation dimensions and prohibiting unsupported factual or evaluative additions.Targets include mainstream-newsroom or tabloid style, neutral emotion, and stronger event-congruent emotion.Source: §A.2 · p. 11
- English dissemination prompts target public-information or attention-and-engagement framing while preserving the other dimensions and disallowing unsupported additions.Source: §A.2 · pp. 11–12
A.3 Chinese Transformation Prompts for Weibo21
- Weibo21 uses the same six operational targets in Chinese, while the English detection prompt retains a shared label vocabulary and decision format.The Chinese rewriting instructions likewise preserve core factual claims and prohibit adding or altering factual content.Source: §A.3 · p. 12
A.4 LLM-as-a-Judge Prompt
- The preservation judge returns a binary factual-preservation label, and Cohen’s κ is computed against human labels rather than generated by the LLM.Source: §A.4 · p. 12
B.1 Dataset Composition and Topic-Balanced Sampling
The evaluation uses a topic-balanced Weibo21 subset drawn from the official release. It contains equal numbers of real and fake cases across nine named topics, totaling 1,350 cases.
B.1 Dataset Composition and Topic-Balanced Sampling
- The official Weibo21 release has 9,592 cases across training, validation, and test splits, plus nine named topics and an undetermined category.The split counts are 5,751 training, 1,918 validation, and 1,923 test cases.Source: §B.1 · pp. 12–13
- Figure 7’s word clouds show the nine named evaluation topics, with larger terms indicating greater frequency within each topic.Source: Figure 7 · p. 13
- Table 6 reports category distributions for the original Weibo21 data and the topic-balanced evaluation subset across the official splits.Val. denotes the official validation split.Source: Table 6 · p. 13
- The topic-balanced Weibo21 evaluation subset contains 150 cases per named topic and 1,350 cases overall, with 75 real and 75 fake cases sampled from each of nine topics.The evaluation-only subset pools the official splits, excludes undetermined cases and specified duplicates, and uses sampling seed 2027.Source: §B.1 · p. 13
- The same sampled cases are used to construct style, emotion, and dissemination views for every evaluated detector.Source: §B.1 · p. 13
B.2 Detailed Zero-Shot Results
On topic-balanced Weibo21, each of the four zero-shot LLMs has substantially lower Macro-F1 on at least one transformed view than on original news. Qwen2.5-7B-Instruct leads on original, emotion, and dissemination views, while Qwen3-8B leads on style.
B.2 Detailed Zero-Shot Results
- Qwen2.5-7B-Instruct performs best on original, emotion, and dissemination views, whereas Qwen3-8B performs best under style transformation.Source: §B.2 · pp. 13–14
- 62.61% UASR is reported for Llama-3.1-8B-Instruct, the most sensitive model overall; GLM-4-9B has the lowest UASR but comparatively low original- and transformed-view performance.UASR is conditioned on cases the model classified correctly in the original view.Source: §B.2 · p. 14
- All four zero-shot models have substantially lower Macro-F1 on at least one transformed view than on original news.Table 7 reports overall results on all 1,350 topic-balanced cases without separating them by topic.Source: §B.2 · pp. 13–14
- The original three-by-three topic visualization is rearranged into one column to keep the Chinese terms readable.Source: §B.2 · p. 14
B.3 Topic-Level Analysis
Across nine Weibo21 topics, zero-shot LLM Macro-F1 falls under all three presentation transformations; dissemination framing is consistently the most challenging. Topic performance differs significantly, but transformation effects do not significantly interact with topic.
B.3 Topic-Level Analysis
- Figure 6 presents Accuracy and Macro-F1 across nine topics for original news and the three presentation transformations.The analysis averages four zero-shot LLMs to characterize topic- and view-level performance variation.Source: §B.3 · p. 14
- Averaged across nine topics, zero-shot LLMs achieve 69.45% Macro-F1 on original news, falling to 51.94%, 50.76%, and 45.71% under style, emotion, and dissemination transformations, respectively.Source: §B.3 · p. 14
- Dissemination framing has the lowest Accuracy and Macro-F1 among the three transformations in every topic, while the Topic × Transformation interaction is not significant for either metric.Permutation tests report pperm = 0.0617 for Accuracy and pperm = 0.0781 for Macro-F1; overall performance differs across topics (pperm = 0.0061 and 0.0053, respectively).Source: §B.3 · p. 14
B.4 Topic Word Clouds
- Figure 7 provides a qualitative overview of recurring terms associated with the nine Weibo21 topics.Source: §B.4 · p. 14