PaperlayerView exact revision on arXiv

Source-linked AI summary

Temporal Leakage in Financial News NLP: A Multi-Architecture Audit with a Regime-Specific M&A Signal

Chenhao Xue, Raslen Guesmi, Siwei Feng, Yucheng Gong, Jacob Xavier Sundram, Jordan Pang, Lan Wang, Julian Kaljuvee

arXiv:2608.17223v1cs.CLcs.LG

TL;DR

Financial-news NLP benchmarks may overstate predictive performance because random splits expose models to temporally shared market structure. This paper audits modern architectures chronologically and finds substantial leakage, while a small surviving signal is localized to M&A articles within the studied regime.

  • Problem

    Financial-news NLP lacks systematic evidence on leakage across modern architectures and on whether news-to-return signals survive strict temporal validation.

  • Method

    The paper audits feature-model combinations under random and chronological splits, then conditions locked-test evaluation on event type and examines the surviving M&A signal.

  • Results

    Random-split MCC exceeds temporal-split MCC by ∼1×–6.5×, while M&A is the only audited event type with consistently positive chronological signal.

  • Takeaways & Limitations

    Chronological splitting leaves a small, regime-specific M&A-concentrated residual rather than a universal financial-news predictor.

  • Takeaways & Limitations

    The study is largely confined to a single 2025 regime and near-temporal test horizon, leaving generalization across structurally different regimes untested.

Abstract

from arXiv · show

Financial-news direction prediction has become a popular NLP benchmark, yet reported gains depend critically on whether the train-test split is chronological or random, i.e., on temporal leakage. We audit this dependence on a 49,799-article corpus across 16 feature-model combinations spanning TF-IDF, MiniLM, FinBERT, and fine-tuned RoBERTa-large / DeBERTa-v3-large, plus separate zero/few-shot and LoRA probes of Llama-3 and Qwen2.5 LLMs: random splits inflate MCC by $1.1\times$ to $6.5\times$, tracking model capacity and feature richness, and end-to-end FinBERT fine-tuning re-amplifies rather than closes the gap (size-matched ratio $1.75\times$). Conditioning on event type, mergers and acquisitions (M&A) is the only audited category with a positive locked-test signal under near-temporal chronological evaluation (TF-IDF MCC $= 0.138$ train-only, $0.068$ under train$\cup$val refit; 10,000-permutation $p < 10^{-3}$); the signal does not transfer to FNSPID's 2009-2020 U.S. corpus, localising the headline to our 2024-2025 European-tilted M&A semantics rather than a universal predictor. Three independent role labellers converge on acquirer-tagged articles as the signal locus, a power-limited qualitative convergence rather than a hypothesis-tested asymmetry. Chronological splitting plays for financial NLP the role characteristics-purging plays for asset pricing: it strips the predictable, stale component of news and leaves a residual that is small, event-localized, and lexically shallow. We advocate leakage audits as a required disclosure for financial-NLP benchmarks.

1 Introduction

The introduction identifies temporal leakage as a structural vulnerability in financial-news NLP and presents a multi-architecture audit showing inflated random-split performance. Under chronological validation, general prediction is near-random, while M&A retains the only audited event-conditioned signal.

  • Motivation: Financial-news NLP is vulnerable because articles, firms, market regimes, and return labels are jointly temporally autocorrelated.Random article splits can expose models to same-cycle vocabulary, entity, and return correlations.
  • Leakage audit: 1.1× to 6.5× random-split MCC inflation grows with model capacity and is largest for high-capacity nonlinear models on rich features.The comparison averages 10 seeds across 16 feature–model combinations.
  • Surviving signal: 0.138 TF-IDF MCC with p < 10^-3 marks M&A as the only audited event-conditioned subset retaining a near-temporal locked-test signal.Across three independent role labellers, the signal is qualitatively associated with acquirer-side localization rather than a hypothesis-tested asymmetry.
  • Study scope: The study audits leakage, temporal signal survival, event-conditioned recovery, and mechanisms across modern architectures and feature stacks.The audit covers 16 feature–model combinations on 49,799 articles, from TF-IDF through fine-tuned DeBERTa-v3-large, plus LLM probes.
  • Temporal validation: 0.060 test MCC is achieved by the strongest temporal model, FinBERT+LR, indicating near-random general prediction under temporal validation.The introduction characterizes domain-adapted FinBERT embeddings as providing only minor recovery.

2 Related Work

Prior work spans event-driven stock prediction, financial sentiment and domain-specific language models, and text-based asset-pricing research. This paper extends that literature by quantifying chronological-evaluation gaps for fine-tuned financial models on short-window press releases.

  • Event-driven stock prediction: Event-driven stock prediction evolved from neural news embeddings and tweet-price models to news-price integration, attention, inter-firm graphs, hierarchical transformers, and self-supervised augmentation.
  • Financial sentiment, domain LMs, and financial LLMs: Financial-text research progressed from dictionaries and rhetorical features to FinBERT, curated FOMC corpora, domain LLMs, and headline-based return prediction.
  • Financial sentiment, domain LMs, and financial LLMs: The paper quantifies the chronological-evaluation gap for GPU-fine-tuned FinBERT, RoBERTa-large, and DeBERTa-v3-large on short-window press releases.
  • Media, information, and asset prices: Prior information and asset-pricing studies linked media content to price reversion, disaster risk, text-based factors, and text-as-data methods, while M&A research documented target-favouring wealth effects.

3 Task, Data, and Label Construction

The study frames next-day stock-return direction as a binary financial-news classification task using 49,799 non-neutral articles from a timestamped 2020–2025 corpus. It uses chronological train/validation/test splits, market-adjusted labels, event-conditioned analysis, and MCC-centered evaluation with locked-test permutation testing.

  • Data and task: 49,799 articles remain after removing neutral or near-zero returns from 56,409 timestamped financial-news articles spanning 2020–2025.The corpus covers 64 stock exchanges, 203 event types, and 81% 2025 articles.
  • Data and task: UP/DOWN labels encode subsequent one-trading-day stock-return direction, with MCC as the primary metric and balanced accuracy as a complement.The horizon is close-to-close, or close-to-next-open for after-hours releases.
  • Label construction: 85–90% of raw and market-adjusted labels agree, while the adjusted variant compares returns with the primary index of the listing exchange.The market-adjusted label is UP for positive abnormal return and DOWN otherwise; M&A results use both schemes.
  • Chronological evaluation: 21,654 training, 10,866 validation, and 17,279 test articles define a strictly chronological, near-temporal split, with test consultation exactly once.Training precedes 2025-04-01, validation covers 2025-04–2025-05, and testing begins 2025-06-01.
  • Event-conditioned analysis: 12 frequent, economically interpretable event categories receive detailed analysis, with M&A selected for locked-test evaluation because deals affect valuation and convey rich semantic content.The stated mechanisms include documented wealth effects and information about acquirer, target, deal type, premium, and financing.
  • Statistical testing: 10,000 random test-label permutations define the locked M&A null test, while per-event analyses apply Benjamini–Hochberg correction at q = 0.05 across 12 events.Confidence intervals use weekly block bootstrap resampling of calendar weeks.

4 Multi-Architecture Temporal Leakage Audit

The audit shows that random splits systematically overstate financial-news direction-prediction performance, with inflation increasing with model capacity and feature richness. Chronological evaluation exposes small temporal MCCs, while fine-tuning can re-amplify leakage and ratio claims require denominator-aware interpretation.

  • Leakage patterns: 1.1×–6.5× random/temporal MCC inflation varies by architecture, with tree ensembles reaching 6.5× and FinBERT[CLS]+LR just 1.1×.The audit compares size-matched temporal and random splits while holding features and hyperparameters fixed.
  • Leakage patterns: Adding content and numerical features raises random-split MCC while leaving temporal MCC near zero, indicating reliance on memorizable time-local patterns.Feature richness systematically widens the random-versus-temporal performance gap.
  • Fine-tuning: 2.7× random/temporal inflation appears for end-to-end FinBERT fine-tuning, versus 1.1× for FinBERT[CLS]+LR; the size-matched fine-tuned ratio is 1.75×.Fine-tuned FinBERT records random MCC 0.174 ± 0.009 and temporal MCC 0.064 ± 0.007 across 5 seeds.
  • Cross-corpus audit: 28.9× EDT-RF inflation is driven by temporal MCC = 0.001, while the audit’s directional pattern replicates across the EDT corpus.The EDT audit covers 106,619 articles from 2020–2021; linear models inflate approximately 1.7× and RF reaches 28.9×.
  • Ratio interpretation: 0.048 ΔMCC for FinBERT-FT is more interpretable than its ratio, with nonoverlapping 95% bootstrap CIs of [0.090, 0.134] and [0.054, 0.074].The passage recommends reporting ΔMCC = MCCrandom − MCCtemporal with paired confidence intervals because near-zero temporal denominators destabilize ratios.

5 Models and Conditions

The study benchmarks four supervised feature-model baselines and separately evaluates LLMs under multiple prompting configurations. Its chronological split spans the COVID transition, so extreme RandomForest collapse may combine leakage with regime-shift generalisation failure.

  • Supervised baselines: Four supervised baselines span TF-IDF, metadata-enhanced RandomForest, FinBERT, and MiniLM representations.FinBERT [CLS]+LR achieves the strongest single-model temporal result, with general MCC = 0.060.
  • Supervised baselines: M&A TF-IDF+LR reaches MCC = 0.138 with validation-selected hyperparameters from a 360-cell grid.The selected configuration uses nfeat = 100, C = 5.0, ngram=(1,1), sublinear_tf=False, and min_df=2.
  • Chronological-split condition: The EDT chronological split spans the COVID transition, confounding extreme RandomForest temporal collapse with regime-shift generalisation failure.This limitation means the collapse cannot be attributed solely to leakage.
  • LLM evaluation: LLM zero-shot tests compare Claude Sonnet 4.5, Claude Opus 4.7, and GPT-5.4 across five input or prompting configurations.Outputs are parsed into UP/DOWN using a fixed regex determined before test; locked M&A analyses use 10,000-permutation tests and weekly block bootstrap.

6 General News Prediction Results

Under proper temporal validation, general news prediction is near-random, with FinBERT+LR the strongest temporal model. Zero-shot LLMs perform somewhat better, but general prediction remains limited by heterogeneity, staleness, label noise, and publication lag.

  • MCC = 0.060: FinBERT+LR is the strongest model under proper temporal validation, while general news prediction remains near-random.
  • 203-event-type heterogeneity, temporal staleness, label noise, and publication lag are identified as drivers of weak general-news performance.
  • MCC = 0.108: multi-LLM consensus using title plus event is the best zero-shot general-news configuration.

7 Event-Conditioned Results

M&A is the only event type with consistently positive predictive signal, and locked-test analyses support a text-driven, event-specific effect that remains robust across statistical and cutoff checks. The result is suggestive in rolling evaluation but survives corrected locked-test and cross-event comparisons.

  • Rolling-window screening: M&A achieved mean MCC = 0.081, was positive in 7/8 rolling-window months, and had sign-test p = 0.035, but failed BH correction at q = 0.05.The rolling-window result is therefore characterized as suggestive rather than conclusive.
  • Locked-test M&A signal: MCC = 0.138 with balanced accuracy = 0.569 on the locked M&A test, with monthly MCCs of +0.126, +0.135, and +0.171.The train-only protocol fits on M&A train data, selects hyperparameters on validation, and evaluates once on the locked test.
  • Statistical validation: Observed MCC = 0.1378 produced z-score = 3.81 and two-sided p < 10^-3 under 10,000 permutations, with weekly bootstrap 95% CI = [+0.066, +0.205].The confidence interval excludes zero, and Bonferroni-12 adjustment remains below 0.05.
  • Robustness: Test MCC ∈[+0.115, +0.156] under ±7- and ±14-day cutoff perturbations, while the six-month horizon yields MCC = +0.133 and ∆= 0.005 from the headline.These checks indicate the result is not a cutoff-specific artifact.
  • Event specificity and controls: Only M&A produced a substantial positive test MCC (+0.123) among seven next-largest event categories, while text achieved val MCC 0.228 versus 0.037 for exchange tokens and −0.020 for metadata.Across the four-event panel, M&A is the unique signal-positive cell, supporting event specificity and a text-driven effect.

8 What Drives the M&A Signal? Acquirer-Side Localisation (Power-Limited)

The M&A signal localizes to acquirer-tagged articles across independent role-labelling approaches, while target-side results are near zero. However, acquirer–target asymmetry remains power-limited and qualitative because bootstrap confidence intervals span zero.

  • Power limitation: 95% bootstrap confidence intervals for ∆MCC = MCC_ACQ − MCC_TGT span zero under both labellers, so the asymmetry is reported as qualitative rather than hypothesis-tested.The intervals are [−0.075, +0.331] for regex and [−0.277, +0.586] for NER+dependency parsing.
  • M&A model performance: 0.138 test MCC: TF-IDF LogReg dominates M&A, exceeding fine-tuned FinBERT-tone 0.050 ± 0.030 and balanced DeBERTa-v3-large 0.061 ± 0.040.It also outperforms the other reported supervised and heuristic baselines.
  • Acquirer-side localisation: 0.160 regex-ACQUIRER MCC and 0.221 NER-ACQUIRER MCC contrast with symmetric TARGET MCC values of 0.000, 0.017, and 0.098.A regex-ACQUIRER FinBERT specialist also reaches MCC = 0.195, supporting convergent acquirer-side localisation.

9 Conclusion

Chronological splitting removes predictable news components, leaving a small, regime-specific M&A residual rather than a universal financial-news predictor. The conclusion also bounds reported trading and LLM findings by their evaluation and reproducibility conditions.

  • Conclusion: Chronological splitting parallels characteristics-purging by removing predictable components and leaving a regime-specific M&A residual.The residual achieves MCC = 0.138 train→val→test and +0.068 train+val→test merge, with p < 10^-3.
  • Conclusion: The M&A residual reproduces on the proprietary corpus, partially replicates on EDT, but does not transfer to FNSPID 2009–2020 US M&A.Within-FNSPID MCC = −0.011 at n = 4,235; the signal is bounded to the 2024–2025 European-tilted regime.
  • Conclusion: +2.62 is the top-quartile cost-aware Sharpe at 10 bps/side and is an upper bound, while all-trade frictionless +0.52 is the baseline.Closed-source LLM cells use prerelease identifiers, but the audit-ratio and supervised M&A headlines do not depend on them.

Limitations · A Reproducibility, Compute, and Statistical Methodology · A.1 Reproducibility Details

The paper’s main limitations are proprietary data, near-temporal evaluation in a largely 2025 regime, limited statistical power, and noncausal interpretation. Reproducibility is supported by released audit artifacts, explicit return-window rules, software and hardware specifications, and documented hyperparameter grids.

  • Limitations: The proprietary primary dataset cannot be publicly released, so only the M&A claim receives partial external mitigation through EDT replication.Other event-type findings remain dataset-specific.
  • Limitations: 81% of data is from 2025, and the three-month near-temporal test leaves generalization across structurally different regimes untested.A six-month 2025 sensitivity test yielded pooled MCC = +0.133 (p < 10^-4, but remains within-regime evidence).
  • Limitations: n = 786 with 11 weekly clusters; the locked M&A test reports one-sided permutation p < 10^-4, two-sided p < 10^-3, and a block-bootstrap 95% CI of [+0.066, +0.205].The acquirer asymmetry remains power-limited, with two-sided p = 0.141 under the global-specialist regex protocol and p = 0.086 under the role-specific specialist.
  • Limitations: The FinBERT-FT random/temporal ratio is 1.75× under size matching, compared with 2.7× in the default unequal-size protocol.The matched protocol reports random MCC = 0.112 ± 0.011, showing inflation is real but smaller than the unsized comparison suggests.
  • Limitations: The M&A signal is predictive correlation rather than causation and may reflect underreaction, vendor-corpus selection, microstructure, or label-construction artifacts.The paper therefore does not assign a causal explanation to the observed signal.
  • A.1 Reproducibility Details: All audits use seed 42 unless otherwise noted, with released Python implementations, chronological split definitions, LLM prompts, and figure-generation scripts.Figures can be regenerated from the released JSON result files.
  • A Reproducibility, Compute, and Statistical Methodology: CPU baselines require approximately 22 min per architecture cell over 10 seeds, while six GPU fine-tuning scripts require approximately 80 min on one RTX 3090.The software stack includes Python 3.10, scikit-learn 1.6, NumPy 2.4, SciPy 1.17, sentence-transformers 2.7, transformers 4.42, and PyTorch 2.3.
  • A.1 Reproducibility Details: Return construction uses one-trading-day open-to-open returns after hours and close-to-close otherwise, with local-time normalization and deferral of nontrading-day or near-close releases.The benchmark is the listing exchange’s primary index, including S&P 500 for U.S. venues and broad national or regional European indices.

A.2 Statistical Methodology Details … C.1.6 B6: Audit Robustness Across Extended HP Cells

The paper’s M&A result remains positive across permutation, bootstrap, horizon, cutoff, hyperparameter, and protocol checks, but is regime-specific and lexically shallow rather than a universal predictor. Auxiliary audits localize the signal to deal semantics, show limited acquirer-only power, and confirm that temporal leakage patterns persist across extended model cells.

  • A.2 Statistical Methodology Details; B.1 Per-Event Negative Control; B.1.1 Rolling-Window M&A Figure: None of 12 rolling events survives BH correction at q = 0.05, so the headline relies on the locked-test permutation result rather than M&A’s uncorrected rolling p = 0.035.M&A’s corrected q* is 0.42.
  • B.1.1 Rolling-Window M&A Figure: M&A is the only rolling series positive in every month, while the full-corpus and event-conditioned averages remain near zero.This temporal pattern motivates treating M&A as an event-localized signal rather than a corpus-wide effect.
  • A.2 Statistical Methodology Details; B.1.2 M&A Permutation Null Figure: 10,000 permutations give M&A test MCC z = 3.81 and two-sided p < 10^-3, while the 1,000-block bootstrap gives mean +0.139 with 95% CI [+0.066, +0.205].The permutation test uses shuffled labels; the bootstrap resamples 11 ISO-week blocks from June–August 2025.
  • B.2 Extended HP Grid Stability: ∆val-MCC ≤0.03 across the top-15 cells, and leave-one-axis-out selection yields test MCCs {0.131, 0.140, 0.138, 0.143, 0.138}, mean 0.138± 0.006.The extended grid therefore forms a broad plateau, and the locked-test result is robust to removing any single hyperparameter axis.
  • B.2 Extended HP Grid Stability: 0.132 is the mean test MCC across cutoff offsets of −14, −7, 0, +7, and +14 days, with std = 0.017, so the result is not a one-date artifact.The corresponding test MCCs are {0.115, 0.156, 0.138, 0.134, 0.118}.
  • B.4 M&A Locked-Test Protocol Map; B.3 Extended-Window Sensitivity Test on the M&A Specialist: MCC = +0.068 under train+val refit, matching the stress-test value to three decimals, while train-only is the lowest-leakage headline protocol.The extended-window and train+val protocols serve as horizon- and deployment-style checks rather than replacements for the headline evaluation.
  • B.3 Extended-Window Sensitivity Test on the M&A Specialist: +0.133 at n = 1275 remains statistically indistinguishable from +0.138 at n = 786, with p < 10^-4 and 95% bootstrap CI [+0.040, +0.220].Five of six extended-window months are positive, while March 2025 is −0.067; the test remains within 2025 and does not resolve broader regime shift.
  • C.1.1 B1: Calibration of the M&A Specialist; C.1.2 B2: Per-Firm Fairness Audit: 0.138 is not concentrated in frequent acquirers: the head bucket has negative MCC, the unseen-at-train-time bucket retains 0.043, and TF-IDF achieves ROC-AUC = 0.567 with average precision = 0.655.The modest MCC of 0.138 accompanies well-separated extreme scores but uncertainty in the middle.

C.2 Why Small-Data LoRA Fine-Tuning Fails on M&A Headlines … D.2 Deep M&A Specialist Multi-Seed Variance

Across robustness and multi-seed audits, the M&A signal is threshold-stable but temporally variable, lexically shallow, and concentrated beyond dominant publishers, while small-data LoRA can collapse and deep specialists remain inferior to TF-IDF. Fine-tuning also amplifies FinBERT’s random-split leakage gap rather than resolving it.

  • C.2 Why Small-Data LoRA Fine-Tuning Fails on M&A Headlines: MCC = 0.000 is Llama-3-8B LoRA’s only complete open-LLM collapse, producing an almost-all-UP predictor on 731 training headlines.The model converges after roughly 137 optimizer steps despite training loss falling to approximately 0.66.
  • C.2 Why Small-Data LoRA Fine-Tuning Fails on M&A Headlines: 0.066 MCC is Qwen2.5-7B-LoRA’s six-epoch result, still below Qwen zero-shot at 0.115 after its predict-UP rate falls from 0.986 to 0.758.The evidence supports using the pretrained checkpoint directly for ntr ≤1k short headlines, while full-parameter SFT and longer LoRA schedules remain untested.
  • C.3.2 B8: Per-Month Rolling MCC on the Locked Test: Two of three locked-test months are positive, but August 2025 turns slightly negative, making pooled MCC the honest three-month summary.The decomposition indicates short-horizon variability that can flip month by month and discourages extrapolation across other horizons.
  • C.3.3 B9: True spaCy NER Entity Blanking: 0.014 absolute MCC is the erosion from masking six spaCy NER entity types, while masking ORG spans alone costs 0.003 MCC.The result indicates that the signal is overwhelmingly non-entity vocabulary rather than firm-identity leakage.
  • C.3.4 B10: Per-Headline Coefficient Attribution on Acquirer Articles: No firm name appears among the top-three TF-IDF contributors in any of ten highest-confidence acquirer-side predictions.The predictive lexicon instead consists of shared deal verbs and structural nouns such as enters, agreement, company, software, limited, holding, and board.
  • C.3.5 B11: Publisher-Level Fairness Audit: MCC = +0.028 in the two dominant-publisher mid bucket contrasts with signal concentration in tail and unseen-at-train publisher buckets.The evidence suggests high-volume publishers contain more routine or repetitive listings, whereas smaller and more topical publishers carry denser informative patterns.

D.3 Pooled 10-Seed, Multilingual, Full-SFT, and CoT Controls (DeBERTa-v3-large, XLM-R, FinBERT-tone SFT, Qwen CoT) · E LLM Benchmarks and Template Sensitivity · E.3 LLM Zero-Shot Detailed Results

Across pooled deep-transformer, full-SFT, multilingual, and chain-of-thought controls, no model recovers the locked-test TF-IDF MCC of 0.138. LLM performance is template-sensitive: structured prompts and M&A consensus generally underperform simpler baselines, while richer content helps only modestly beyond the low-feature-budget audit setting.

  • D.3 Pooled 10-Seed, Multilingual, Full-SFT, and CoT Controls (DeBERTa-v3-large, XLM-R, FinBERT-tone SFT, Qwen CoT): 0.061 ± 0.040 pooled-10 DeBERTa-v3-large MCC falls below the original 5-seed estimate of 0.085 ± 0.044, indicating the apparent lift was seed noise.The new five-seed mean was 0.038 ± 0.021.
  • D.3 Pooled 10-Seed, Multilingual, Full-SFT, and CoT Controls (DeBERTa-v3-large, XLM-R, FinBERT-tone SFT, Qwen CoT): 0.045 ± 0.030 mean test MCC for XLM-RoBERTa-large shows multilingual encoding does not improve the M&A result.One seed was degenerate, predicting UP for every article and obtaining MCC = 0.000.
  • D.3 Pooled 10-Seed, Multilingual, Full-SFT, and CoT Controls (DeBERTa-v3-large, XLM-R, FinBERT-tone SFT, Qwen CoT): 0.034 ± 0.035 mean test MCC for full-parameter FinBERT-tone SFT remains below the sampler-balanced LoRA-style baseline of 0.050 ± 0.030.Unfreezing all parameters and using class-weighted loss therefore does not rescue FinBERT-tone performance.
  • D.3 Pooled 10-Seed, Multilingual, Full-SFT, and CoT Controls (DeBERTa-v3-large, XLM-R, FinBERT-tone SFT, Qwen CoT): 0.042 CoT MCC for Qwen2.5-7B-Instruct trails the plain zero-shot M&A result of 0.115 and the TF-IDF specialist’s 0.138.CoT also shifts the predicted-UP rate to 0.954; the multi-seed CoT-v2 strict mean was +0.011±0.030.
  • E LLM Benchmarks and Template Sensitivity: 0.4 abs-MCC is the largest title-versus-title+content difference at max_features=50, whereas richer max_features=2000 budgets make content add genuine but small signal.The audit retains the low-feature-budget protocol because feature budget interacts with the cross-architecture comparison.
  • E LLM Benchmarks and Template Sensitivity: 0.138 locked-test MCC is achieved only by the TF-IDF specialist among the large-model audits; every transformer and LLM variant performs below MCC = 0.10.All comparisons use the locked M&A test with n = 786.
  • E.3 LLM Zero-Shot Detailed Results: 0.065 M&A MCC for zero-shot Claude Sonnet 4.5 is below supervised TF-IDF, while role-prompt and title+event variants fall to 0.057 and 0.035.Adding structured prompt information therefore does not monotonically improve the single-model M&A result.
  • E.3 LLM Zero-Shot Detailed Results: 0.058 M&A MCC for Sonnet/Opus/GPT consensus is worse than the best single model because GPT-5.4 is anti-correlated with the text-based signal.On general news, consensus instead reaches MCC = 0.108 versus the best single model’s 0.077.

E.4 Open-LLM M&A Benchmark … I.4.2 LLMs (zero-shot and CoT-v2, multi-seed)

The paper finds that the M&A signal is strongest in acquirer-tagged articles, economically meaningful when confidence-ranked, and distinct from other events, while open-LLM results are weaker and highly prompt-sensitive. Cross-event and model-family audits support a regime-specific M&A effect rather than a generalizable benchmark advantage.

  • E.4 Open-LLM M&A Benchmark: Qwen2.5-7B zero-shot is the strongest open-LLM result but trails the supervised TF-IDF specialist by 17% relative, while LoRA fine-tuning degenerates.Its predicted-UP rate is 0.355 versus the test set’s true UP rate of 0.594.
  • E.4 Open-LLM M&A Benchmark: ΔMCC ≈ 0.14 separates two reasonable zero-shot prompts: M&A-specific Qwen-zs reaches +0.115, whereas the cross-event uniform prompt yields −0.022 ± 0.010.Both evaluations use the same locked test set of n = 786, making prompt sensitivity a central limitation.
  • F.1 NER + Dependency Role-Attribution Pipeline: Acquirer-tagged articles carry the stronger M&A signal under independent regex and NER labellers, with MCC gaps of +0.154 and +0.123, respectively.The NER pipeline independently assigns roles using organization entities and dependency-linked deal verbs.
  • F.2 ACQUIRER/TARGET FinBERT Specialists: MCC = 0.195 for the ACQUIRER FinBERT specialist versus 0.017 for TARGET, triangulating the role asymmetry across three model- and label-source-independent pipelines.The comparison is power-limited because the test subsets contain 125 acquirer and 85 target articles.
  • G.1 General-News Detail / G.2 Non-Text Control Detail: M&A’s text specialist outperforms general-news and non-text controls: FinBERT [CLS]+LR reaches MCC = 0.060, while the M&A validation MCC is 0.228.General-news results use a locked test set of 17,279 articles; the M&A text specialist’s validation score exceeds structured and aggregate metadata baselines.
  • H.1 Economic Significance Backtest / I.3 Cross-Event Full CPU Pack: Per-Event Replication of B-tests: Sharpe +2.62 at 10 bps/side is achieved by the top-quartile-confidence strategy, whereas indiscriminate all-trade M&A prediction does not survive realistic frictions.The confidence-filtered subset contains n = 197 trades; the frictionless all-trade Sharpe is +0.52.
  • I.1 Per-Event Specialist Table / I.2 Cross-Event Audit Pipeline: M&A vs. Clinical Trials vs. Legal Issues vs. Earnings: M&A test MCC = +0.123 is 7× the next-best event and uniquely transfers from validation to held-out test, while clinical-study leakage produces a 4.20× audit ratio and near-zero locked-test MCC.The cross-event audit covers M&A, clinical studies, legal issues, and earnings under event-specific validation and locked-test protocols.
  • I.3.1 Calibration and Discrimination (B1) / I.3.2 ORG-Token Role (B5; train-only protocol) / I.3.3 NER-Based Entity Blanking (B9) / I.3.4 Decision-Threshold Sweep (B7) / I.3.5 Per-Month Rolling MCC (B8; train+val combined) / I.3.6 Publisher Fairness (B11) / I.4 Cross-Event Deep Models and LLMs / I.4.1 Deep Specialists (5-seed, per-event train-only) / I.4.2 LLMs (zero-shot and CoT-v2, multi-seed): Across calibration, masking, blanking, thresholds, monthly stability, publisher fairness, deep specialists, and LLMs, M&A occupies the low-ratio-plus-signal regime rather than the failure regimes of other events.The cross-event CPU pack defines a 2×2 taxonomy by audit-ratio level and locked-test signal; deep-model and multi-seed LLM results extend that comparison.

I.4.3 Reading the cross-event deep-model and LLM evidence … LLM setting M&A CLN LGL ERN

Cross-event deep and LLM probes preserve the taxonomy’s central result: M&A is the only event with a stable positive locked-test signal, while CLN, LGL, and ERN remain null or unreliable. Independent EDT and FNSPID analyses further restrict the M&A finding to a regime-specific, corpus-dependent effect.

  • I.4.3 Reading the cross-event deep-model and LLM evidence: +0.022 and −0.03 are representative LLM positives on CLN and LGL, but skewed prediction distributions indicate base-rate matching rather than transferable signal.Two zero-shot LLMs disagree by ≥ 0.04 MCC on three of four events, and the single-seed Qwen-zs M&A value +0.115 uses a different prompt template.
  • LLM setting M&A CLN LGL ERN: +0.061 ± 0.023 is the strongest cited multi-seed CoT-strict mean, but CoT-strict shows wide cross-event spread, parse-fail rates of 17.3%–32.5%, and no reliable lift over zero.The corresponding single-seed structured CLN reference is +0.115, underscoring protocol sensitivity.
  • J.1 EDT Cross-Quarter Decomposition: 15.7×, 3.2×, and 1× are the within-quarter EDT inflation ratios for logistic regression, random forest, and gradient boosting, separating leakage from regime shift.Gradient boosting turns negative in 2020Q3 and 2021Q1 across the COVID-recovery transition, whereas logistic regression remains slightly positive in every quarter.
  • I.4.3 Reading the cross-event deep-model and LLM evidence: +0.138 → +0.068 is the M&A MCC change after pooling validation data into training, while entity ablations show transferable signal in deal-related verbs and nouns rather than names.M&A remains positive across all monthly buckets, unlike the sign-flipping or near-zero patterns reported for other events.
  • J.2 FNSPID Cross-Corpus Replication: Within, Forward, and Reverse Protocols: −0.016 is the unchanged specialist’s MCC on 4,235 FNSPID test rows, showing that the proprietary M&A lexicon does not transfer to 2009–2020 U.S. headlines.The same fixed TF-IDF+LR recipe yields no within-FNSPID signal at 30,070 training rows, with p_two = 0.46.
  • J.2 FNSPID Cross-Corpus Replication: Within, Forward, and Reverse Protocols: −0.088 is the significant reverse cross-corpus MCC when FNSPID-trained models are tested on proprietary data, indicating anti-correlated domain shift rather than merely low power.The proprietary headline MCC = 0.138, p < 10^-3 is therefore regime-specific rather than corpus-universal.
  • J.2 FNSPID Cross-Corpus Replication: Within, Forward, and Reverse Protocols: +0.045 ± 0.018 is FinBERT’s reverse-transfer MCC, below the in-domain +0.138, while joint training reaches +0.070 on proprietary data and −0.011 on FNSPID.These results support “some semantic transfer, no lexical transfer” and reinforce the regime-specific framing.
Loading 2608.17223v1…