PaperlayerView exact revision on arXiv

Source-linked AI summary

All That Glisters Is Not Gold: A Benchmark for Reference-Free Counterfactual Financial Misinformation Detection

Yuechen Jiang, Zhiwei Liu, Yupeng Cao, Yueru He, Ziyang Xu, Chen Xu, Zhiyang Deng, Prayag Tiwari, Xi Chen, Alejandro Lopez-Lira, Jimin Huang, Junichi Tsujii, Sophia Ananiadou

arXiv:2601.04160v3cs.CLcs.CEq-fin.CP

TL;DR

Financial misinformation benchmarks commonly rely on external evidence, leaving unclear whether LLMs can detect internally misleading, commitment-shifting edits in isolation. RFC-BENCH addresses this gap with paragraph-level reference-free detection and paired comparative diagnosis across realistic financial-news perturbations. Across 14 models, performance remains near chance without comparison but becomes strong when explicit contrast is available, while the benchmark’s scope is limited to English-language U.S. stock news and controlled rewrites.

  • Problem

    Existing evaluations commonly validate claims against external evidence, leaving limited evidence about whether LLMs detect internally commitment-shifting financial misinformation without grounding.

  • Method

    RFC-BENCH uses minimally perturbed financial-news paragraphs and evaluates single-paragraph Reference-free Detection alongside paired Comparative Diagnosis across four manipulation types.

  • Results

    Across 14 open- and closed-source LLMs, reference-free performance remains near chance while accuracy rises to 0.85–0.97 when an original paragraph enables Comparative Diagnosis.

  • Takeaways & Limitations

    Current LLMs can often identify changes with explicit contrast but do not reliably detect locally plausible counterfactual distortions in isolation.

  • Takeaways & Limitations

    RFC-BENCH covers only English-language financial news focused on U.S. stocks, limiting applicability to other languages, regions, and financial systems.

Abstract

from arXiv · show

We introduce RFC Bench, a benchmark for evaluating large language models on financial misinformation under realistic news. RFC Bench operates at the paragraph level and captures the contextual complexity of financial news where meaning emerges from dispersed cues. The benchmark defines two complementary tasks: reference free misinformation detection and comparison based diagnosis using paired original perturbed inputs. Experiments reveal a consistent pattern: performance is substantially stronger when comparative context is available, while reference free settings expose significant weaknesses, including unstable predictions and elevated invalid outputs. These results indicate that current models struggle to maintain coherent belief states without external grounding. By highlighting this gap, RFC Bench provides a structured testbed for studying reference free reasoning and advancing more reliable financial misinformation detection in real world settings.

1 Introduction

RFC-BENCH targets a gap in financial misinformation evaluation: models must detect commitment-shifting edits from paragraph-internal cues, without external evidence. It introduces paired reference-free and comparative tasks and finds that explicit contrast substantially improves performance.

  • LLMs are usually evaluated on interpreting fluent text, but rarely on whether that text is admissible as an object of interpretation.
  • Minimal edits can preserve fluency while shifting warranted commitments, such as changing possibility into certainty or sequence into causation.
  • RFC-BENCH contains 1,826 original–perturbed financial-news paragraph pairs spanning directional, numerical, sentiment, and causal manipulations.The dataset retains perturbations only after minimality constraints and domain-expert validation, with category correctness agreement of 98.9% and rewrite validity agreement of 93.7%.
  • Reference-free Detection peaks at 53.6% accuracy, whereas Comparative Diagnosis reaches 0.85–0.97 accuracy when an original paragraph is provided.The reference-free peak is reported for DeepSeek-reasoner; GPT-4.1 reaches 52.7%, and the best few-shot result reaches 56.7% with LLaMA-3.3-70B.
  • The benchmark defines Reference-free Detection on single paragraphs and Comparative Diagnosis on original–perturbed pairs.
  • Across 14 open- and closed-source LLMs, models perform well with explicit comparison but remain unreliable when manipulated paragraphs must be flagged in isolation.This pattern is described as consistent with an accommodation-first failure mode.

2 RFC-BENCH

RFC-BENCH evaluates paragraph-level financial misinformation using single-paragraph detection and paired comparative diagnosis. It constructs minimally perturbed financial-news paragraphs across four manipulation categories and applies expert-guided rewriting and quality assurance.

  • Evaluation tasks: RFC-BENCH evaluates misinformation through reference-free detection of a single paragraph and comparative diagnosis of paired factual and manipulated paragraphs.Reference-free detection uses only the input document, whereas comparative diagnosis provides the original and perturbed versions.
  • Dataset construction: The benchmark is curated from original–perturbed financial-news paragraph pairs, with perturbations designed to be minimal yet sufficient to instantiate predefined misinformation categories.The source collection includes 1,404 unique financial news articles covering 223 publicly traded stocks.
  • Manipulation categories: Four manipulation categories are used: Numerical Perturbation, Directional Flipping, Sentiment Amplification, and Causal Distortion.A rule-based classifier first identifies explicit category cases before category-specific rewriting and validation.
  • Category-specific rewriting: GPT-4.1 rewrites categorized paragraphs under category-specific constraints controlling the direction and magnitude of semantic distortions.Prompts are iteratively refined through expert validation, with automatic quality-control mechanisms checking adherence to manipulation constraints.
  • Directional Flipping: Directional Flipping reverses implied market outlook while preserving events, entities, and numerical values.Expert-validated rewrites and token-length constraints are used to limit incomplete inversions and factual drift.
  • Sentiment Amplification: Sentiment Amplification intensifies evaluative tone while preserving factual content and directional meaning.Expert-validated rewrites use a token-length ratio of 0.90–1.30 to constrain semantic changes.
  • Causal Distortion: Causal Distortion changes explanatory relations between events while preserving entities and observable outcomes.Prompt refinement targets unintended outcome or entity changes, with a token-length ratio of 0.90–1.30 enforced.
  • Human quality assurance: Human quality assurance combines expert review, independent auditing, dual-annotator evaluation, and adjudication of disagreements.Category correctness reached accuracy ≥0.965, while severe label imbalance motivated reporting Gwet’s AC1 alongside Cohen’s κ and Macro-F1.

3 Experiments

Across 14 models, performance is near chance for standalone reference-free detection but rises sharply when original–perturbed paragraphs enable comparative diagnosis. Few-shot prompting yields limited gains, while error analysis indicates reliance on surface cues and difficulty isolating manipulation mechanisms.

  • Main Results: Models struggle on standalone detection but become highly accurate when original and perturbed paragraphs are provided together.The results reveal a consistent asymmetry between reference-free judgment and pairwise diagnosis.
  • Reference-free Detection: Macro-F1 remains below 0.53 and MCC near zero in Task 1, with best zero-shot accuracy of 53.0% from Qwen3-8B.GPT-4.1 reaches 52.7% accuracy, while few-shot prompting improves results only modestly.
  • Comparative Diagnosis: Task 2 accuracy rises to 0.85–0.97 when models compare original and perturbed paragraphs.The explicit contrast supports discrepancy localization and manipulation attribution.
  • Few-shot Ablation: Few-shot prompting provides limited gains and remains far below Task 2 performance.The highest reported accuracy is 58.4% for GPT-4.1 with 2-shot prompting, while GPT-5 Mini improves from 45.2% zero-shot to 57.5% with eight shots.
  • Artifact Check: A surface-feature logistic regression underperforms the median LLM on Task 2, so formatting and lexical artifacts are not the primary source of the performance gap.The baseline uses length, punctuation, numeric-token differences, and lexical overlap.
  • Error Analysis: Errors show that models may reject speculative statements, accept authoritative-looking fabricated narratives, and confuse overlapping manipulation cues.Examples include polarity reversals mislabeled as numerical changes and evaluative reversals expressed through causal phrasing.

4 Conclusion and Future Work

RFC-BENCH evaluates commitment-shifting financial misinformation through minimally perturbed news paragraphs in standalone and comparative settings. Models remain near chance without contrast but perform strongly with explicit alternatives, leaving substantial room to improve reference-free admissibility for reliable financial use.

  • Benchmark and Findings: RFC-BENCH uses minimally perturbed financial news paragraphs to evaluate reference-free detection and comparative diagnosis.The benchmark covers single-paragraph judgments and pairwise attribution with the original provided.
  • Benchmark and Findings: Across 14 LLMs, models remain near chance without contrast but achieve strong accuracy when explicit contrast is available.A surface-feature baseline suggests this gap is not driven by trivial lexical or formatting artifacts.
  • Future Work: The results suggest that current LLMs struggle to detect commitment shifts from discourse-internal cues alone and default to accommodation unless contrast is provided.The conclusion identifies reference-free admissibility as a prerequisite for reliable use in finance and other high-stakes domains.

Limitations

RFC-BENCH is intentionally controlled and narrow, limiting how broadly its findings generalize and what aspects of real-world misinformation use it evaluates. Its results should complement, not replace, evidence-based and multimodal evaluation.

  • Scope: The benchmark covers only English-language financial news focused on U.S. stocks, limiting applicability to other languages, regions, and financial systems.Different reporting conventions, regulatory regimes, and discourse styles may constrain transferability.
  • Scope: The text-only design excludes tables, figures, earnings slides, audio, and video, so it does not test multimodal or cross-document evidence integration.Such information often accompanies real-world financial disclosures.
  • Data realism: Controlled, minimal perturbations may not capture real-world adversarial misinformation involving longer-range inconsistencies, cross-paragraph contradictions, or coordinated narratives.The rewriting pipeline preserves plausibility but constrains the distribution of misinformation represented.
  • Evaluation setting: Paragraph-level isolation does not measure how models should combine internal discourse cues with external verification in broader-context settings.The isolation is intentional for studying reference-free admissibility but narrows practical coverage.
  • Downstream impact: The evaluation measures detection and diagnosis accuracy, not downstream effects on decision-making, trading behavior, or human trust.Integration into end-to-end financial analysis pipelines remains open.
  • Interpretation: RFC-BENCH is a controlled diagnostic testbed whose findings are complementary to, rather than a replacement for, evidence-based and multimodal evaluations.Its scope is necessarily limited.

Ethical Considerations

The study uses publicly accessible, corporate-level financial narratives and applies safeguards for responsible annotation, synthetic rewriting, dataset release, and research use. Its materials and procedures are designed to avoid personal or confidential information and to constrain harmful content.

  • Data handling: All source materials come from publicly accessible Yahoo Finance articles, while annotators and models did not process personal, confidential, proprietary, or non-public information.The dataset concerns corporate-level financial narratives rather than private individuals.
  • Harm minimization: Annotators were instructed not to introduce defamatory content, legal accusations, fabricated events, or misleading claims involving identifiable individuals.Synthetic misinformation is confined to corporate-level financial performance, numerical statements, market outlooks, and related narratives.
  • Responsible use: The released dataset is for academic research and must not be used as real market information, investment advice, or guidance for trading decisions.Synthetic articles are marked as artificial and released only in controlled research settings.
  • Annotation practice: Annotators and experts received briefings on responsible data handling, research integrity, and harm minimization, with guidelines emphasizing caution and neutrality.These procedures support transparent and ethically grounded dataset construction and release.
  • Release constraints: The dataset redistributes metadata and synthetic rewritten text rather than original Yahoo Finance content, while requiring compliance with copyright, platform, and ethical standards.Commercial use, investment decision-making, and real-world financial communication are prohibited.
  • Quality assurance: Human quality assurance uses a system showing the ticker, manipulation category, original paragraph, and rewritten instance, with pass, fail, and mis-category decisions.The process combines domain experts and trained annotators with finance and auditing backgrounds.

E Dataset Statistics

The appendix reports temporal distributions for the financial-news subsets before deduplication and for the merged dataset after global deduplication, alongside sample counts before and after adjudication.

  • Temporal coverage: The collected financial-news articles span April 25, 2025 to December 15, 2025.This range is reported for the supplementary temporal-distribution statistics.
  • Before deduplication: Figure 8 shows temporal distributions for four data subsets before deduplication.The figure describes the subsets prior to merging and global deduplication.
  • After deduplication: Figure 9 shows the temporal distribution of the merged dataset after global deduplication.It complements the pre-deduplication distributions shown for the four subsets.
  • Adjudication counts: Figure 10 compares pre-adjudication samples, final retained samples, and hard-case subsets across misinformation categories.The figure presents sample counts before and after adjudication.

F Annotation Consistency Analysis

Annotation consistency is assessed across both annotation stages using confusion matrices, with disagreements concentrated in subtle or borderline cases while overall consistency supports dataset reliability.

  • Step 1: Step 1 measures agreement on category correctness between mis-category and truth-category labels.Disagreements mainly arise from subtle contextual ambiguity.
  • Step 2: Step 2 measures rewrite validity using fail and pass labels, with most samples reliably validated and remaining discrepancies reflecting borderline cases.The reported pattern supports stable annotation consistency.
  • Overall assessment: Overall, the consistency results support the reliability of the RFC-BENCH dataset.This conclusion follows the two-stage confusion-matrix analysis.
  • Consistency analysis: Figure 11 reports annotation consistency for Step 1 and Step 2.The analysis covers category correctness and rewrite validity.

G RFC-BENCH dataset construction workflow

RFC-BENCH construction combines staged data review with reliability-aware evaluation, using expert and annotator checks before final benchmarking.

  • Dataset construction workflow: The workflow covers data acquisition, LLM-based category-specific rewriting, human quality assurance, and final adjudication.Human QA includes expert review and dual annotation.
  • Evaluation metrics: Confusion matrices generalize from binary TP, FN, FP, and TN counts to a K × K matrix in multi-class settings.Each matrix entry records ground-truth class i predicted as class j.
  • Annotation reliability: Gwet’s AC1 is additionally reported because severe class imbalance can make Cohen’s κ and Macro-F1 unstable.The benchmark also reports invalid predictions separately from valid metric calculations.

J Removed and Hard-case Examples

The examples illustrate directional, numerical, sentiment, and causal distortions by changing financial claims while preserving much of their surrounding form.

  • Flipping: A directional flip can recast a price-target increase as a decrease and reverse the analyst rating, violating numerical and semantic coherence.The 3M example changes $164 to $170 from raised to lowered and Overweight to Underweight.
  • Numerical inconsistency: Numerical perturbations alter reported values or introduce fabricated quantitative details, as shown in the ChargePoint and AMD examples.The examples change revenue, expectation, price-target, and percentage figures.
  • Sentiment distortion: Sentiment distortion injects emotionally loaded language that exaggerates a balanced financial assessment without necessarily changing its numerical facts.The Zoetis example intensifies cautious language into alarming uncertainty while retaining the price-target figures.
  • Causal distortion: Causal distortion adds unsupported explanations, such as investor positioning or shifting market liquidity, to otherwise similar market claims.These rewrites introduce spurious attribution or an unsubstantiated mechanism.

K Dataset Release and Access

RFC-BENCH is released with structured metadata and documentation, while excluding original article text and retaining a separate hard-case subset for robustness analysis.

  • Final retained dataset: The final retained dataset contains 1,845 paragraph-level financial misinformation instances that passed expert review, dual annotation, and adjudication.These samples constitute the cleaned benchmark used in primary experiments.
  • Release scope: The released data exclude original Yahoo Finance article text and provide metadata, public source URLs, and controlled rewritten misinformation text instead.Metadata include stock ticker and publication date.
  • Hard-case subset: The hard-case subset contains 122 ambiguous or disagreement cases excluded from the main benchmark but retained for robustness analysis and error characterization.These cases represent elevated ambiguity or borderline semantic shifts identified during adjudication.
  • Access and documentation: Both releases use a unified schema covering tickers, dates, source links, manipulation categories, and rewritten counterfactual text.Documentation describes the schema, category definitions, annotation process, and evaluation protocols.

L Experiment Result Visualization

The experiment visualizations present confusion matrices for both tasks, separating open-source model results from the broader open- and closed-source comparison.

  • Open-source models: The open-source visualizations separately label Task 1 and Task 2 confusion-matrix panels.The panels are identified as (a) Task 1 and (b) Task 2 for open-source models.
  • Confusion matrices: Figure 13 presents confusion matrices for Task 1 and Task 2 across open-source and closed-source models.The figure is organized by task and model-source grouping.

M Few-shot Ablation Results

The ablation materials compare Task 1 performance across few-shot settings and inspect whether original–perturbed pairs differ through trivial surface cues.

  • Few-shot comparison: Figure 14 presents Task 1 confusion matrices for open-source models under two-shot and eight-shot settings.
  • Few-shot comparison: Table 8 reports Task 1 performance under 2-shot and 8-shot settings, including invalid outputs and classification metrics.The table defines accuracy, precision, recall, macro-F1, and Matthews Correlation Coefficient.
  • Surface-level diagnostics: The diagnostic analysis tests whether perturbations depend on trivial artifacts such as length, punctuation, or numeric changes.The analysis reports distributional comparisons across misinformation categories.
  • Surface-level diagnostics: Figure 15 compares original and perturbed paragraphs across multiple surface-level dimensions.

O Error Analysis and Case Studies

The case studies identify several failure modes in reference-free judgments and manipulation classification, including temporal anchoring, surface plausibility, numeric-cue reliance, and confusion between polarity and causality.

  • Reference-free judgment failures: Reference-free reasoning can reject coherent predictive journalism when models treat analyst expectations as requiring externally verifiable evidence.This conflates contextual plausibility with factual certainty.
  • Reference-free judgment failures: A model accepted a false-looking financial narrative because its Reuters-like style, precise figures, and plausible explanations created surface plausibility.
  • Reference-free judgment failures: Temporal anchoring can produce a correct label for an incorrect reason when models treat post-cutoff information as inherently unverifiable.The analysis contrasts this behavior with evaluating plausibility under the document’s implied timeframe.
  • Overall implication: Correct accuracy in Task 1 does not necessarily indicate coherent paragraph-level belief formation under reference-free settings.
  • Manipulation-classification errors: Polarity reversals with unchanged numerical magnitudes can be misclassified as Numerical when models prioritize percentages and directional keywords.The reported case preserves values such as 6.5%, 28.6%, and multi-year percentages while reversing their interpretation.
  • Manipulation-classification errors: Polarity-driven rewrites can also be misclassified as Causal when rhetorical causal language supports an already flipped conclusion.The example changes a positive analyst endorsement into a negative one while adding causal connectors.
  • Data filtering and causal criteria: The benchmark’s pre-filtering uses category-specific keyword patterns, including digits, dollar amounts, percentages, directional signals, and performance-versus-expectation phrases.
  • Data filtering and causal criteria: Causal candidates require an explicit reported cause–effect relation, and rewrites should preserve the outcome and actors while replacing only the stated cause.
Loading 2601.04160v3…