Source-linked AI summary
Temporal Leakage in Financial News NLP: A Multi-Architecture Audit with a Regime-Specific M&A Signal
Chenhao Xue, Raslen Guesmi, Siwei Feng, Yucheng Gong, Jacob Xavier Sundram, Jordan Pang, Lan Wang, Julian Kaljuvee
TL;DR
Financial-news NLP benchmarks may overstate predictive performance because random splits expose models to temporally shared market structure. This paper audits modern architectures chronologically and finds substantial leakage, while a small surviving signal is localized to M&A articles within the studied regime.
Problem
Financial-news NLP lacks systematic evidence on leakage across modern architectures and on whether news-to-return signals survive strict temporal validation.
Source: §1.1 · p. 1, §1.2 · p. 2Method
The paper audits feature-model combinations under random and chronological splits, then conditions locked-test evaluation on event type and examines the surviving M&A signal.
Source: §1.2 · p. 2, §3.5 · p. 4Results
Random-split MCC exceeds temporal-split MCC by ∼1×–6.5×, while M&A is the only audited event type with consistently positive chronological signal.
Source: §4.1 · pp. 4–5, §7.1 · p. 6Takeaways & Limitations
Chronological splitting leaves a small, regime-specific M&A-concentrated residual rather than a universal financial-news predictor.
Source: §9 · p. 8The study is largely confined to a single 2025 regime and near-temporal test horizon, leaving generalization across structurally different regimes untested.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Financial-news direction prediction has become a popular NLP benchmark, yet reported gains depend critically on whether the train-test split is chronological or random, i.e., on temporal leakage. We audit this dependence on a 49,799-article corpus across 16 feature-model combinations spanning TF-IDF, MiniLM, FinBERT, and fine-tuned RoBERTa-large / DeBERTa-v3-large, plus separate zero/few-shot and LoRA probes of Llama-3 and Qwen2.5 LLMs: random splits inflate MCC by $1.1\times$ to $6.5\times$, tracking model capacity and feature richness, and end-to-end FinBERT fine-tuning re-amplifies rather than closes the gap (size-matched ratio $1.75\times$). Conditioning on event type, mergers and acquisitions (M&A) is the only audited category with a positive locked-test signal under near-temporal chronological evaluation (TF-IDF MCC $= 0.138$ train-only, $0.068$ under train$\cup$val refit; 10,000-permutation $p < 10^{-3}$); the signal does not transfer to FNSPID's 2009-2020 U.S. corpus, localising the headline to our 2024-2025 European-tilted M&A semantics rather than a universal predictor. Three independent role labellers converge on acquirer-tagged articles as the signal locus, a power-limited qualitative convergence rather than a hypothesis-tested asymmetry. Chronological splitting plays for financial NLP the role characteristics-purging plays for asset pricing: it strips the predictable, stale component of news and leaves a residual that is small, event-localized, and lexically shallow. We advocate leakage audits as a required disclosure for financial-NLP benchmarks.
1 Introduction
The introduction identifies temporal leakage as a structural vulnerability in financial-news NLP and presents a multi-architecture audit showing inflated random-split performance. Under chronological validation, general prediction is near-random, while M&A retains the only audited event-conditioned signal.
Motivation
- Financial-news NLP is vulnerable because articles, firms, market regimes, and return labels are jointly temporally autocorrelated.Random article splits can expose models to same-cycle vocabulary, entity, and return correlations.Source: §1.1 · p. 1
Leakage audit
- 1.1× to 6.5× random-split MCC inflation grows with model capacity and is largest for high-capacity nonlinear models on rich features.The comparison averages 10 seeds across 16 feature–model combinations.Source: §1.1 · p. 2, §1.3 · p. 2
Surviving signal
- 0.138 TF-IDF MCC with p < 10^-3 marks M&A as the only audited event-conditioned subset retaining a near-temporal locked-test signal.Across three independent role labellers, the signal is qualitatively associated with acquirer-side localization rather than a hypothesis-tested asymmetry.Source: §1.1 · p. 2, §1.4 · p. 2
Study scope
- The study audits leakage, temporal signal survival, event-conditioned recovery, and mechanisms across modern architectures and feature stacks.The audit covers 16 feature–model combinations on 49,799 articles, from TF-IDF through fine-tuned DeBERTa-v3-large, plus LLM probes.Source: §1.1 · p. 1, §1.2 · p. 2
Temporal validation
- 0.060 test MCC is achieved by the strongest temporal model, FinBERT+LR, indicating near-random general prediction under temporal validation.The introduction characterizes domain-adapted FinBERT embeddings as providing only minor recovery.Source: §1.3 · p. 2, §1.4 · p. 2
2 Related Work
Prior work spans event-driven stock prediction, financial sentiment and domain-specific language models, and text-based asset-pricing research. This paper extends that literature by quantifying chronological-evaluation gaps for fine-tuned financial models on short-window press releases.
Event-driven stock prediction
- Event-driven stock prediction evolved from neural news embeddings and tweet-price models to news-price integration, attention, inter-firm graphs, hierarchical transformers, and self-supervised augmentation.Source: §2 · p. 2
Financial sentiment, domain LMs, and financial LLMs
- Financial-text research progressed from dictionaries and rhetorical features to FinBERT, curated FOMC corpora, domain LLMs, and headline-based return prediction.Source: §2 · pp. 2–3
- The paper quantifies the chronological-evaluation gap for GPU-fine-tuned FinBERT, RoBERTa-large, and DeBERTa-v3-large on short-window press releases.Source: §2 · p. 3
Media, information, and asset prices
- Prior information and asset-pricing studies linked media content to price reversion, disaster risk, text-based factors, and text-as-data methods, while M&A research documented target-favouring wealth effects.Source: §2 · p. 3
3 Task, Data, and Label Construction
The study frames next-day stock-return direction as a binary financial-news classification task using 49,799 non-neutral articles from a timestamped 2020–2025 corpus. It uses chronological train/validation/test splits, market-adjusted labels, event-conditioned analysis, and MCC-centered evaluation with locked-test permutation testing.
Data and task
- 49,799 articles remain after removing neutral or near-zero returns from 56,409 timestamped financial-news articles spanning 2020–2025.The corpus covers 64 stock exchanges, 203 event types, and 81% 2025 articles.Source: §3.1 · p. 3
- UP/DOWN labels encode subsequent one-trading-day stock-return direction, with MCC as the primary metric and balanced accuracy as a complement.The horizon is close-to-close, or close-to-next-open for after-hours releases.Source: §3.2 · p. 3, §3.3 · p. 3
Label construction
- 85–90% of raw and market-adjusted labels agree, while the adjusted variant compares returns with the primary index of the listing exchange.The market-adjusted label is UP for positive abnormal return and DOWN otherwise; M&A results use both schemes.Source: §3.3 · p. 3
Chronological evaluation
- 21,654 training, 10,866 validation, and 17,279 test articles define a strictly chronological, near-temporal split, with test consultation exactly once.Training precedes 2025-04-01, validation covers 2025-04–2025-05, and testing begins 2025-06-01.Source: §3.4 · pp. 3–4
Event-conditioned analysis
- 12 frequent, economically interpretable event categories receive detailed analysis, with M&A selected for locked-test evaluation because deals affect valuation and convey rich semantic content.The stated mechanisms include documented wealth effects and information about acquirer, target, deal type, premium, and financing.Source: §3.5 · p. 4
Statistical testing
- 10,000 random test-label permutations define the locked M&A null test, while per-event analyses apply Benjamini–Hochberg correction at q = 0.05 across 12 events.Confidence intervals use weekly block bootstrap resampling of calendar weeks.Source: §3.6 · p. 4
4 Multi-Architecture Temporal Leakage Audit
The audit shows that random splits systematically overstate financial-news direction-prediction performance, with inflation increasing with model capacity and feature richness. Chronological evaluation exposes small temporal MCCs, while fine-tuning can re-amplify leakage and ratio claims require denominator-aware interpretation.
Leakage patterns
- 1.1×–6.5× random/temporal MCC inflation varies by architecture, with tree ensembles reaching 6.5× and FinBERT[CLS]+LR just 1.1×.The audit compares size-matched temporal and random splits while holding features and hyperparameters fixed.Source: §4 · p. 4, §4.1 · p. 5
- Adding content and numerical features raises random-split MCC while leaving temporal MCC near zero, indicating reliance on memorizable time-local patterns.Feature richness systematically widens the random-versus-temporal performance gap.Source: §4.1 · p. 5
Fine-tuning
- 2.7× random/temporal inflation appears for end-to-end FinBERT fine-tuning, versus 1.1× for FinBERT[CLS]+LR; the size-matched fine-tuned ratio is 1.75×.Fine-tuned FinBERT records random MCC 0.174 ± 0.009 and temporal MCC 0.064 ± 0.007 across 5 seeds.Source: §4.1 · p. 5, Table 2 · p. 5, Figure 1 · p. 5
Cross-corpus audit
- 28.9× EDT-RF inflation is driven by temporal MCC = 0.001, while the audit’s directional pattern replicates across the EDT corpus.The EDT audit covers 106,619 articles from 2020–2021; linear models inflate approximately 1.7× and RF reaches 28.9×.Source: §4.1 · p. 5, Table 3 · p. 5, §4.2 · p. 6
Ratio interpretation
- 0.048 ΔMCC for FinBERT-FT is more interpretable than its ratio, with nonoverlapping 95% bootstrap CIs of [0.090, 0.134] and [0.054, 0.074].The passage recommends reporting ΔMCC = MCCrandom − MCCtemporal with paired confidence intervals because near-zero temporal denominators destabilize ratios.Source: §4.1 · pp. 5–6
5 Models and Conditions
The study benchmarks four supervised feature-model baselines and separately evaluates LLMs under multiple prompting configurations. Its chronological split spans the COVID transition, so extreme RandomForest collapse may combine leakage with regime-shift generalisation failure.
Supervised baselines
- Four supervised baselines span TF-IDF, metadata-enhanced RandomForest, FinBERT, and MiniLM representations.FinBERT [CLS]+LR achieves the strongest single-model temporal result, with general MCC = 0.060.Source: §5 · p. 6
- M&A TF-IDF+LR reaches MCC = 0.138 with validation-selected hyperparameters from a 360-cell grid.The selected configuration uses nfeat = 100, C = 5.0, ngram=(1,1), sublinear_tf=False, and min_df=2.Source: §5 · p. 6
Chronological-split condition
- The EDT chronological split spans the COVID transition, confounding extreme RandomForest temporal collapse with regime-shift generalisation failure.This limitation means the collapse cannot be attributed solely to leakage.Source: §5 · p. 6
LLM evaluation
- LLM zero-shot tests compare Claude Sonnet 4.5, Claude Opus 4.7, and GPT-5.4 across five input or prompting configurations.Outputs are parsed into UP/DOWN using a fixed regex determined before test; locked M&A analyses use 10,000-permutation tests and weekly block bootstrap.Source: §5 · p. 6
6 General News Prediction Results
Under proper temporal validation, general news prediction is near-random, with FinBERT+LR the strongest temporal model. Zero-shot LLMs perform somewhat better, but general prediction remains limited by heterogeneity, staleness, label noise, and publication lag.
- MCC = 0.060: FinBERT+LR is the strongest model under proper temporal validation, while general news prediction remains near-random.Source: §6 · p. 6
- 203-event-type heterogeneity, temporal staleness, label noise, and publication lag are identified as drivers of weak general-news performance.Source: §6 · p. 6
- MCC = 0.108: multi-LLM consensus using title plus event is the best zero-shot general-news configuration.Source: §6 · p. 6
7 Event-Conditioned Results
M&A is the only event type with consistently positive predictive signal, and locked-test analyses support a text-driven, event-specific effect that remains robust across statistical and cutoff checks. The result is suggestive in rolling evaluation but survives corrected locked-test and cross-event comparisons.
Rolling-window screening
- M&A achieved mean MCC = 0.081, was positive in 7/8 rolling-window months, and had sign-test p = 0.035, but failed BH correction at q = 0.05.The rolling-window result is therefore characterized as suggestive rather than conclusive.Source: §7.1 · p. 6
Locked-test M&A signal
- MCC = 0.138 with balanced accuracy = 0.569 on the locked M&A test, with monthly MCCs of +0.126, +0.135, and +0.171.The train-only protocol fits on M&A train data, selects hyperparameters on validation, and evaluates once on the locked test.Source: §7.2 · p. 7
Statistical validation
- Observed MCC = 0.1378 produced z-score = 3.81 and two-sided p < 10^-3 under 10,000 permutations, with weekly bootstrap 95% CI = [+0.066, +0.205].The confidence interval excludes zero, and Bonferroni-12 adjustment remains below 0.05.Source: §7.2 · p. 7
Robustness
- Test MCC ∈[+0.115, +0.156] under ±7- and ±14-day cutoff perturbations, while the six-month horizon yields MCC = +0.133 and ∆= 0.005 from the headline.These checks indicate the result is not a cutoff-specific artifact.Source: §7.2 · p. 7
Event specificity and controls
- Only M&A produced a substantial positive test MCC (+0.123) among seven next-largest event categories, while text achieved val MCC 0.228 versus 0.037 for exchange tokens and −0.020 for metadata.Across the four-event panel, M&A is the unique signal-positive cell, supporting event specificity and a text-driven effect.Source: §7.2 · p. 7, §7.3 · p. 7, §7.4 · p. 7
8 What Drives the M&A Signal? Acquirer-Side Localisation (Power-Limited)
The M&A signal localizes to acquirer-tagged articles across independent role-labelling approaches, while target-side results are near zero. However, acquirer–target asymmetry remains power-limited and qualitative because bootstrap confidence intervals span zero.
Power limitation
- 95% bootstrap confidence intervals for ∆MCC = MCC_ACQ − MCC_TGT span zero under both labellers, so the asymmetry is reported as qualitative rather than hypothesis-tested.The intervals are [−0.075, +0.331] for regex and [−0.277, +0.586] for NER+dependency parsing.Source: §8 · p. 8
M&A model performance
- 0.138 test MCC: TF-IDF LogReg dominates M&A, exceeding fine-tuned FinBERT-tone 0.050 ± 0.030 and balanced DeBERTa-v3-large 0.061 ± 0.040.It also outperforms the other reported supervised and heuristic baselines.Source: §8 · p. 8
Acquirer-side localisation
- 0.160 regex-ACQUIRER MCC and 0.221 NER-ACQUIRER MCC contrast with symmetric TARGET MCC values of 0.000, 0.017, and 0.098.A regex-ACQUIRER FinBERT specialist also reaches MCC = 0.195, supporting convergent acquirer-side localisation.Source: §8 · p. 8
9 Conclusion
Chronological splitting removes predictable news components, leaving a small, regime-specific M&A residual rather than a universal financial-news predictor. The conclusion also bounds reported trading and LLM findings by their evaluation and reproducibility conditions.
Conclusion
- Chronological splitting parallels characteristics-purging by removing predictable components and leaving a regime-specific M&A residual.The residual achieves MCC = 0.138 train→val→test and +0.068 train+val→test merge, with p < 10^-3.Source: §9 · p. 8
- The M&A residual reproduces on the proprietary corpus, partially replicates on EDT, but does not transfer to FNSPID 2009–2020 US M&A.Within-FNSPID MCC = −0.011 at n = 4,235; the signal is bounded to the 2024–2025 European-tilted regime.Source: §9 · p. 8
- +2.62 is the top-quartile cost-aware Sharpe at 10 bps/side and is an upper bound, while all-trade frictionless +0.52 is the baseline.Closed-source LLM cells use prerelease identifiers, but the audit-ratio and supervised M&A headlines do not depend on them.Source: §9 · p. 8
Limitations · A Reproducibility, Compute, and Statistical Methodology · A.1 Reproducibility Details
The paper’s main limitations are proprietary data, near-temporal evaluation in a largely 2025 regime, limited statistical power, and noncausal interpretation. Reproducibility is supported by released audit artifacts, explicit return-window rules, software and hardware specifications, and documented hyperparameter grids.
Limitations
- The proprietary primary dataset cannot be publicly released, so only the M&A claim receives partial external mitigation through EDT replication.Other event-type findings remain dataset-specific.Source: Limitations · p. 9
- 81% of data is from 2025, and the three-month near-temporal test leaves generalization across structurally different regimes untested.A six-month 2025 sensitivity test yielded pooled MCC = +0.133 (p < 10^-4, but remains within-regime evidence).Source: Limitations · p. 9
- n = 786 with 11 weekly clusters; the locked M&A test reports one-sided permutation p < 10^-4, two-sided p < 10^-3, and a block-bootstrap 95% CI of [+0.066, +0.205].The acquirer asymmetry remains power-limited, with two-sided p = 0.141 under the global-specialist regex protocol and p = 0.086 under the role-specific specialist.Source: Limitations · p. 9
- The FinBERT-FT random/temporal ratio is 1.75× under size matching, compared with 2.7× in the default unequal-size protocol.The matched protocol reports random MCC = 0.112 ± 0.011, showing inflation is real but smaller than the unsized comparison suggests.Source: Limitations · p. 9
- The M&A signal is predictive correlation rather than causation and may reflect underreaction, vendor-corpus selection, microstructure, or label-construction artifacts.The paper therefore does not assign a causal explanation to the observed signal.Source: Limitations · p. 9
A.1 Reproducibility Details
- All audits use seed 42 unless otherwise noted, with released Python implementations, chronological split definitions, LLM prompts, and figure-generation scripts.Figures can be regenerated from the released JSON result files.Source: §A.1 · p. 12
A Reproducibility, Compute, and Statistical Methodology
- CPU baselines require approximately 22 min per architecture cell over 10 seeds, while six GPU fine-tuning scripts require approximately 80 min on one RTX 3090.The software stack includes Python 3.10, scikit-learn 1.6, NumPy 2.4, SciPy 1.17, sentence-transformers 2.7, transformers 4.42, and PyTorch 2.3.Source: §A.1 · p. 12
A.1 Reproducibility Details
- Return construction uses one-trading-day open-to-open returns after hours and close-to-close otherwise, with local-time normalization and deferral of nontrading-day or near-close releases.The benchmark is the listing exchange’s primary index, including S&P 500 for U.S. venues and broad national or regional European indices.Source: §A.1 · p. 12
A.2 Statistical Methodology Details … C.1.6 B6: Audit Robustness Across Extended HP Cells
The paper’s M&A result remains positive across permutation, bootstrap, horizon, cutoff, hyperparameter, and protocol checks, but is regime-specific and lexically shallow rather than a universal predictor. Auxiliary audits localize the signal to deal semantics, show limited acquirer-only power, and confirm that temporal leakage patterns persist across extended model cells.
A.2 Statistical Methodology Details; B.1 Per-Event Negative Control; B.1.1 Rolling-Window M&A Figure
- None of 12 rolling events survives BH correction at q = 0.05, so the headline relies on the locked-test permutation result rather than M&A’s uncorrected rolling p = 0.035.M&A’s corrected q* is 0.42.Source: §A.2 · p. 13, Figure 2 · p. 13
B.1.1 Rolling-Window M&A Figure
- M&A is the only rolling series positive in every month, while the full-corpus and event-conditioned averages remain near zero.This temporal pattern motivates treating M&A as an event-localized signal rather than a corpus-wide effect.Source: Figure 2 · p. 13
A.2 Statistical Methodology Details; B.1.2 M&A Permutation Null Figure
- 10,000 permutations give M&A test MCC z = 3.81 and two-sided p < 10^-3, while the 1,000-block bootstrap gives mean +0.139 with 95% CI [+0.066, +0.205].The permutation test uses shuffled labels; the bootstrap resamples 11 ISO-week blocks from June–August 2025.Source: Figure 3 · p. 13, §A.2 · p. 13
B.2 Extended HP Grid Stability
- ∆val-MCC ≤0.03 across the top-15 cells, and leave-one-axis-out selection yields test MCCs {0.131, 0.140, 0.138, 0.143, 0.138}, mean 0.138± 0.006.The extended grid therefore forms a broad plateau, and the locked-test result is robust to removing any single hyperparameter axis.Source: §B.2 · p. 13, Table 4 · p. 14, Figure 4 · p. 14
- 0.132 is the mean test MCC across cutoff offsets of −14, −7, 0, +7, and +14 days, with std = 0.017, so the result is not a one-date artifact.The corresponding test MCCs are {0.115, 0.156, 0.138, 0.134, 0.118}.Source: §B.2 · pp. 13–14
B.4 M&A Locked-Test Protocol Map; B.3 Extended-Window Sensitivity Test on the M&A Specialist
- MCC = +0.068 under train+val refit, matching the stress-test value to three decimals, while train-only is the lowest-leakage headline protocol.The extended-window and train+val protocols serve as horizon- and deployment-style checks rather than replacements for the headline evaluation.Source: §B.3 · p. 14, §B.4 · p. 15, Table 7 · p. 15
B.3 Extended-Window Sensitivity Test on the M&A Specialist
- +0.133 at n = 1275 remains statistically indistinguishable from +0.138 at n = 786, with p < 10^-4 and 95% bootstrap CI [+0.040, +0.220].Five of six extended-window months are positive, while March 2025 is −0.067; the test remains within 2025 and does not resolve broader regime shift.Source: §B.3 · pp. 14–15, Table 5 · p. 14, Table 6 · p. 15
C.1.1 B1: Calibration of the M&A Specialist; C.1.2 B2: Per-Firm Fairness Audit
- 0.138 is not concentrated in frequent acquirers: the head bucket has negative MCC, the unseen-at-train-time bucket retains 0.043, and TF-IDF achieves ROC-AUC = 0.567 with average precision = 0.655.The modest MCC of 0.138 accompanies well-separated extreme scores but uncertainty in the middle.Source: §C.1.2 · p. 16, §C.1.1 · p. 16, Table 8 · p. 16
C.2 Why Small-Data LoRA Fine-Tuning Fails on M&A Headlines … D.2 Deep M&A Specialist Multi-Seed Variance
Across robustness and multi-seed audits, the M&A signal is threshold-stable but temporally variable, lexically shallow, and concentrated beyond dominant publishers, while small-data LoRA can collapse and deep specialists remain inferior to TF-IDF. Fine-tuning also amplifies FinBERT’s random-split leakage gap rather than resolving it.
C.2 Why Small-Data LoRA Fine-Tuning Fails on M&A Headlines
- MCC = 0.000 is Llama-3-8B LoRA’s only complete open-LLM collapse, producing an almost-all-UP predictor on 731 training headlines.The model converges after roughly 137 optimizer steps despite training loss falling to approximately 0.66.Source: §C.2 · p. 17
- 0.066 MCC is Qwen2.5-7B-LoRA’s six-epoch result, still below Qwen zero-shot at 0.115 after its predict-UP rate falls from 0.986 to 0.758.The evidence supports using the pretrained checkpoint directly for ntr ≤1k short headlines, while full-parameter SFT and longer LoRA schedules remain untested.Source: §C.2 · p. 17
C.3.2 B8: Per-Month Rolling MCC on the Locked Test
- Two of three locked-test months are positive, but August 2025 turns slightly negative, making pooled MCC the honest three-month summary.The decomposition indicates short-horizon variability that can flip month by month and discourages extrapolation across other horizons.Source: §C.3.2 · p. 18
C.3.3 B9: True spaCy NER Entity Blanking
- 0.014 absolute MCC is the erosion from masking six spaCy NER entity types, while masking ORG spans alone costs 0.003 MCC.The result indicates that the signal is overwhelmingly non-entity vocabulary rather than firm-identity leakage.Source: §C.3.3 · p. 18, Table 14 · p. 18
C.3.4 B10: Per-Headline Coefficient Attribution on Acquirer Articles
- No firm name appears among the top-three TF-IDF contributors in any of ten highest-confidence acquirer-side predictions.The predictive lexicon instead consists of shared deal verbs and structural nouns such as enters, agreement, company, software, limited, holding, and board.Source: §C.3.4 · p. 18
C.3.5 B11: Publisher-Level Fairness Audit
- MCC = +0.028 in the two dominant-publisher mid bucket contrasts with signal concentration in tail and unseen-at-train publisher buckets.The evidence suggests high-volume publishers contain more routine or repetitive listings, whereas smaller and more topical publishers carry denser informative patterns.Source: §C.3.5 · p. 19, Table 15 · p. 19
D.3 Pooled 10-Seed, Multilingual, Full-SFT, and CoT Controls (DeBERTa-v3-large, XLM-R, FinBERT-tone SFT, Qwen CoT) · E LLM Benchmarks and Template Sensitivity · E.3 LLM Zero-Shot Detailed Results
Across pooled deep-transformer, full-SFT, multilingual, and chain-of-thought controls, no model recovers the locked-test TF-IDF MCC of 0.138. LLM performance is template-sensitive: structured prompts and M&A consensus generally underperform simpler baselines, while richer content helps only modestly beyond the low-feature-budget audit setting.
D.3 Pooled 10-Seed, Multilingual, Full-SFT, and CoT Controls (DeBERTa-v3-large, XLM-R, FinBERT-tone SFT, Qwen CoT)
- 0.061 ± 0.040 pooled-10 DeBERTa-v3-large MCC falls below the original 5-seed estimate of 0.085 ± 0.044, indicating the apparent lift was seed noise.The new five-seed mean was 0.038 ± 0.021.Source: §D.3 · p. 20
- 0.045 ± 0.030 mean test MCC for XLM-RoBERTa-large shows multilingual encoding does not improve the M&A result.One seed was degenerate, predicting UP for every article and obtaining MCC = 0.000.Source: §D.3 · p. 21
- 0.034 ± 0.035 mean test MCC for full-parameter FinBERT-tone SFT remains below the sampler-balanced LoRA-style baseline of 0.050 ± 0.030.Unfreezing all parameters and using class-weighted loss therefore does not rescue FinBERT-tone performance.Source: §D.3 · p. 21
- 0.042 CoT MCC for Qwen2.5-7B-Instruct trails the plain zero-shot M&A result of 0.115 and the TF-IDF specialist’s 0.138.CoT also shifts the predicted-UP rate to 0.954; the multi-seed CoT-v2 strict mean was +0.011±0.030.Source: §D.3 · pp. 20–21
E LLM Benchmarks and Template Sensitivity
- 0.4 abs-MCC is the largest title-versus-title+content difference at max_features=50, whereas richer max_features=2000 budgets make content add genuine but small signal.The audit retains the low-feature-budget protocol because feature budget interacts with the cross-architecture comparison.Source: §E.2 · pp. 21–22
- 0.138 locked-test MCC is achieved only by the TF-IDF specialist among the large-model audits; every transformer and LLM variant performs below MCC = 0.10.All comparisons use the locked M&A test with n = 786.Source: §D.3 · p. 20, Table 18 · p. 22
E.3 LLM Zero-Shot Detailed Results
- 0.065 M&A MCC for zero-shot Claude Sonnet 4.5 is below supervised TF-IDF, while role-prompt and title+event variants fall to 0.057 and 0.035.Adding structured prompt information therefore does not monotonically improve the single-model M&A result.Source: §E.2 · p. 22
- 0.058 M&A MCC for Sonnet/Opus/GPT consensus is worse than the best single model because GPT-5.4 is anti-correlated with the text-based signal.On general news, consensus instead reaches MCC = 0.108 versus the best single model’s 0.077.Source: §E.2 · p. 22, §E.3 · p. 23
E.4 Open-LLM M&A Benchmark … I.4.2 LLMs (zero-shot and CoT-v2, multi-seed)
The paper finds that the M&A signal is strongest in acquirer-tagged articles, economically meaningful when confidence-ranked, and distinct from other events, while open-LLM results are weaker and highly prompt-sensitive. Cross-event and model-family audits support a regime-specific M&A effect rather than a generalizable benchmark advantage.
E.4 Open-LLM M&A Benchmark
- Qwen2.5-7B zero-shot is the strongest open-LLM result but trails the supervised TF-IDF specialist by 17% relative, while LoRA fine-tuning degenerates.Its predicted-UP rate is 0.355 versus the test set’s true UP rate of 0.594.Source: Table 21 · p. 23, §E.4 · p. 23
- ΔMCC ≈ 0.14 separates two reasonable zero-shot prompts: M&A-specific Qwen-zs reaches +0.115, whereas the cross-event uniform prompt yields −0.022 ± 0.010.Both evaluations use the same locked test set of n = 786, making prompt sensitivity a central limitation.Source: §E.4 · p. 24
F.1 NER + Dependency Role-Attribution Pipeline
- Acquirer-tagged articles carry the stronger M&A signal under independent regex and NER labellers, with MCC gaps of +0.154 and +0.123, respectively.The NER pipeline independently assigns roles using organization entities and dependency-linked deal verbs.Source: §F.1 · p. 24
F.2 ACQUIRER/TARGET FinBERT Specialists
- MCC = 0.195 for the ACQUIRER FinBERT specialist versus 0.017 for TARGET, triangulating the role asymmetry across three model- and label-source-independent pipelines.The comparison is power-limited because the test subsets contain 125 acquirer and 85 target articles.Source: §F.2 · p. 24
G.1 General-News Detail / G.2 Non-Text Control Detail
- M&A’s text specialist outperforms general-news and non-text controls: FinBERT [CLS]+LR reaches MCC = 0.060, while the M&A validation MCC is 0.228.General-news results use a locked test set of 17,279 articles; the M&A text specialist’s validation score exceeds structured and aggregate metadata baselines.Source: §G.1 · p. 25, §G.2 · p. 25
H.1 Economic Significance Backtest / I.3 Cross-Event Full CPU Pack: Per-Event Replication of B-tests
- Sharpe +2.62 at 10 bps/side is achieved by the top-quartile-confidence strategy, whereas indiscriminate all-trade M&A prediction does not survive realistic frictions.The confidence-filtered subset contains n = 197 trades; the frictionless all-trade Sharpe is +0.52.Source: §H.1 · p. 25, Table 25 · p. 27
I.1 Per-Event Specialist Table / I.2 Cross-Event Audit Pipeline: M&A vs. Clinical Trials vs. Legal Issues vs. Earnings
- M&A test MCC = +0.123 is 7× the next-best event and uniquely transfers from validation to held-out test, while clinical-study leakage produces a 4.20× audit ratio and near-zero locked-test MCC.The cross-event audit covers M&A, clinical studies, legal issues, and earnings under event-specific validation and locked-test protocols.Source: §I.1 · p. 26, §I.2 · p. 26, Table 26 · p. 27
I.3.1 Calibration and Discrimination (B1) / I.3.2 ORG-Token Role (B5; train-only protocol) / I.3.3 NER-Based Entity Blanking (B9) / I.3.4 Decision-Threshold Sweep (B7) / I.3.5 Per-Month Rolling MCC (B8; train+val combined) / I.3.6 Publisher Fairness (B11) / I.4 Cross-Event Deep Models and LLMs / I.4.1 Deep Specialists (5-seed, per-event train-only) / I.4.2 LLMs (zero-shot and CoT-v2, multi-seed)
- Across calibration, masking, blanking, thresholds, monthly stability, publisher fairness, deep specialists, and LLMs, M&A occupies the low-ratio-plus-signal regime rather than the failure regimes of other events.The cross-event CPU pack defines a 2×2 taxonomy by audit-ratio level and locked-test signal; deep-model and multi-seed LLM results extend that comparison.Source: §I.3.6 · pp. 27–28, §I.3.1 · p. 27, §I.3.2 · p. 27
I.4.3 Reading the cross-event deep-model and LLM evidence … LLM setting M&A CLN LGL ERN
Cross-event deep and LLM probes preserve the taxonomy’s central result: M&A is the only event with a stable positive locked-test signal, while CLN, LGL, and ERN remain null or unreliable. Independent EDT and FNSPID analyses further restrict the M&A finding to a regime-specific, corpus-dependent effect.
I.4.3 Reading the cross-event deep-model and LLM evidence
- +0.022 and −0.03 are representative LLM positives on CLN and LGL, but skewed prediction distributions indicate base-rate matching rather than transferable signal.Two zero-shot LLMs disagree by ≥ 0.04 MCC on three of four events, and the single-seed Qwen-zs M&A value +0.115 uses a different prompt template.Source: §I.4.3 · p. 29
LLM setting M&A CLN LGL ERN
- +0.061 ± 0.023 is the strongest cited multi-seed CoT-strict mean, but CoT-strict shows wide cross-event spread, parse-fail rates of 17.3%–32.5%, and no reliable lift over zero.The corresponding single-seed structured CLN reference is +0.115, underscoring protocol sensitivity.Source: §I.4.3 · p. 29, Table 38 · p. 33
J.1 EDT Cross-Quarter Decomposition
- 15.7×, 3.2×, and 1× are the within-quarter EDT inflation ratios for logistic regression, random forest, and gradient boosting, separating leakage from regime shift.Gradient boosting turns negative in 2020Q3 and 2021Q1 across the COVID-recovery transition, whereas logistic regression remains slightly positive in every quarter.Source: Figure 9 · p. 30, §J.1 · p. 30
I.4.3 Reading the cross-event deep-model and LLM evidence
- +0.138 → +0.068 is the M&A MCC change after pooling validation data into training, while entity ablations show transferable signal in deal-related verbs and nouns rather than names.M&A remains positive across all monthly buckets, unlike the sign-flipping or near-zero patterns reported for other events.Source: Table 30 · p. 29, Table 32 · p. 30, Table 33 · p. 30
J.2 FNSPID Cross-Corpus Replication: Within, Forward, and Reverse Protocols
- −0.016 is the unchanged specialist’s MCC on 4,235 FNSPID test rows, showing that the proprietary M&A lexicon does not transfer to 2009–2020 U.S. headlines.The same fixed TF-IDF+LR recipe yields no within-FNSPID signal at 30,070 training rows, with p_two = 0.46.Source: §J.2 · pp. 31–32
- −0.088 is the significant reverse cross-corpus MCC when FNSPID-trained models are tested on proprietary data, indicating anti-correlated domain shift rather than merely low power.The proprietary headline MCC = 0.138, p < 10^-3 is therefore regime-specific rather than corpus-universal.Source: §J.2 · p. 32
- +0.045 ± 0.018 is FinBERT’s reverse-transfer MCC, below the in-domain +0.138, while joint training reaches +0.070 on proprietary data and −0.011 on FNSPID.These results support “some semantic transfer, no lexical transfer” and reinforce the regime-specific framing.Source: §J.2 · pp. 32–33, Table 40 · p. 34