PaperlayerView exact revision on arXiv

Source-linked AI summary

PALM: Point-in-Time Adaptation for Financial Language Models

Seunghan Lee, Jun Seo, Jaehoon Lee, Junhyeok Kang, Sangjun Han, Sungdong Yoo, Minjae Kim, Tae Yoon Lim, Dongwan Kang, Hwanil Choi, Soonyoung Lee, Wonbin Ahn

arXiv:2609.30316v1cs.LGcs.CLq-fin.CP

TL;DR

Financial backtests can be biased when a model has already seen the evaluation period, while PIT coverage requires yearly pretraining runs whose necessity had not been tested. The paper compares replaced vintages on fixed evaluation windows and proposes PALM, which updates a frozen checkpoint with a low-rank adapter. It finds no downstream degradation from staleness and reports that a small adapter outperforms continued pretraining.

  • Problem

    PIT suites require costly yearly pretraining, but whether a newer vintage improves on the one it replaces had not been tested using the same evaluation window.

    Source: §1 · p. 2

  • Method

    PALM fits a low-rank adapter on eligible text while keeping pretrained weights frozen, and the study compares vintages on a fixed evaluation window.

    Source: §3.3 · p. 3, §7 · p. 8

  • Results

    At fixed evaluation windows, staleness does not degrade downstream performance; a small adapter also outperforms continued pretraining.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The results suggest maintaining a PIT suite by updating its existing checkpoint rather than producing a new one each year, with a small adapter able to recover the gain.

    Source: §7 · p. 8

    Evidence covers one asset class over a single decade and currently public PIT suites; conclusions do not settle other markets, horizons, or much larger backbones.

    Source: §6 · p. 8

Abstract

from arXiv · show

Language models used in financial backtests suffer from look-ahead bias, as a model trained on text published after the study period has already observed the outcomes it is asked to predict. To handle this issue, point-in-time (PIT) language models are pretrained on chronologically filtered corpora and released as one checkpoint per calendar year, each with a documented cutoff. However, each additional year costs a full pretraining run, and whether that run is necessary has never been tested. In this paper, we show that the annual pretraining run is not necessary. We instead compare each checkpoint against the newer one that replaced it, and find that the newer checkpoint scores no better on the same evaluation window. Motivated by this observation, we propose PALM (Point-in-time Adaptation for financial Language Models), a simple yet effective alternative to annual pretraining that fits a low-rank adapter on text published before the decision date without modifying any pretrained weight. We further find that a small adapter is enough to add a new period to the knowledge an old checkpoint already encodes, and that this outperforms continued pretraining. We validate PALM on a decade of financial news and on various families of PIT models, whose cutoffs span two decades and whose sizes range from 1.3 to 4.2B. Code is available at: https://github.com/seunghan96/palm.

1 Introduction

Financial backtests can be inflated when a model has already read the period it is evaluated on. The paper tests whether yearly PIT pretraining is necessary and proposes PALM, an adapter-based alternative.

1 Introduction

  • A model that has read the evaluation period can recall outcomes rather than forecast them, inflating backtests [6] [19].Source: §1 · p. 1
  • PIT vintages restrict training to pre-cutoff text, but extending a suite requires a full pretraining run each year.The necessity of those repeated runs had not been tested because each vintage was evaluated on the period after its own cutoff.Source: §1 · p. 2
  • Comparisons find that staleness does not degrade downstream performance in any suite, so the newer vintage is not necessarily better.Source: §1 · p. 2
  • PALM fits a low-rank adapter on eligible text while leaving pretrained weights frozen, and the paper reports that a small adapter outperforms continued pretraining.Source: §1 · p. 2

2 Related Works

Related work studies financial language-model evaluation, PIT controls, temporal degradation, and parameter-efficient adaptation. PALM combines PIT eligibility, a financial-text focus, and an update to the model itself.

LLMs for financial text

  • Financial-text studies score documents, rank assets cross-sectionally, and report an information coefficient.Earlier work includes lexicon counting and supervised models, followed by LLMs [28].Source: §2 · p. 2

PIT language models

  • Public PIT suites use date-filtered pretraining, but prior evaluations had not tested whether a vintage's replacement was better [7] [15] [30].This work varies documented cutoffs while holding the evaluation window fixed.Source: §2 · p. 2

Temporal degradation of LLMs

  • Prior temporal-degradation work measures perplexity or knowledge probes and proposes continued pretraining, whereas this paper measures the financial decision's information coefficient.Source: §2 · p. 2

Parameter-efficient adaptation

  • Low-rank adaptation typically reduces supervised-finetuning cost, while retrieval augmentation supplies missing text but requires a point-in-time-restricted index [2, 13, 18].PALM instead uses an adapter to keep a checkpoint current; its binding constraint is the corpus publication date.Source: §2 · p. 2

Positioning of our work

  • PALM combines point-in-time eligibility, a financial-text focus, and an update to the model itself, which no prior line of work in the comparison has all together.Source: §2 · p. 2

3 Proposed Method: PALM

The method evaluates dated financial-news signals from eligible point-in-time vintages, then uses a low-rank adapter trained on text between the vintage cutoff and decision date to keep a checkpoint current without changing pretrained weights.

3.1 Problem Setup

  • A point-in-time suite consists of checkpoints trained on text through documented cutoffs, and each additional calendar year requires a full pretraining run.The corpus used for a cutoff is restricted to documents published on or before that date.Source: §3.1 · pp. 2–3
  • A vintage is eligible only when its cutoff is no later than the decision date; its staleness is the time between those dates.The prevailing convention selects the eligible vintage with the smallest staleness.Source: §3.1 · p. 3
  • To test whether newer vintages perform better, the comparison fixes a decision date and evaluation window, scores every eligible vintage, and relates performance to staleness.Existing evaluations move the window with the cutoff, preventing the effects of cutoff and evaluation period from being separated.Source: §3.1 · p. 3

3.2 Signal Extraction

  • A dated document receives a signal from the log-odds of upward versus downward continuations, read from one forward pass.The signal compares the next-token probabilities of continuations with upward and downward wordings.Source: §3.2 · p. 3
  • Signal quality is measured by the within-date Spearman correlation between signal rankings and next-period returns; the method also reports long-short return and Sharpe ratio.The long-short portfolio subtracts the date-specific mean signal and takes its sign.Source: §3.2 · p. 3

3.3 Point-in-Time Adaptation

  • PALM trains a low-rank adapter on text between an eligible vintage’s cutoff and the decision date, while keeping its pretrained weights frozen.The adapted model remains eligible for the decision date, and the update fits inside existing linear layers without changing the architecture.Source: §3.3 · p. 3

4 Experiments

The experiments hold the evaluation window fixed to test whether older PIT checkpoints lose downstream performance, then evaluate PALM across backbones. The setup spans multiple PIT suites, and the results find no staleness penalty while PALM improves scores across suites.

4.1 Experimental Setup

  • The evaluation uses public PIT suites spanning three training pipelines and two decades of cutoffs, scoring each eligible vintage on one shared window.The panel covers 252,000 article-day observations from 2014–2023; the decision date is the end of November 2020.Source: §4.1 · p. 4

4.2 Effect of Staleness

  • No suite shows a staleness slope distinguishable from zero on the fixed evaluation window, so the newer vintage is not the better one.The slopes even lean positive, opposite to the improvement annual pretraining is designed to deliver.Source: §4.2 · p. 4, Table 4 · p. 4

4.3 Effectiveness of PALM

  • PALM raises the information coefficient across backbones, and the same adapter improves all three pooled measures; on PIT-4B, its gain exceeds six more years of continued pretraining.The Table 6 caption reports that six additional years leave the score where it was, while the adapter moves it.Source: Table 5 · p. 4, §4.3 · p. 5, Table 6 · p. 5

5 Analysis

The analysis finds that PALM can outperform continued pretraining, retains gains across checkpoint ages and evaluation years, and changes scores on later-period material. Sensitivity tests indicate that the gain depends on sufficient training and eligible text, but is robust to the tested decision dates and adaptation locations.

5.1 Effect of the Adapter Capacity

  • A small rank is sufficient to approach the peak: the study adopts rank 2, the smallest rank not worse than peak performance.The gain rises to a peak at moderate rank and falls beyond it; the authors report that large rank overwrites what the checkpoint already holds.Source: §5.1 · p. 5

5.2 Comparison with Continued Pretraining

  • Six more years of continued pretraining move the IC by +0.10, while PALM moves the same snapshots by +1.37, with adapted older snapshots overtaking the later checkpoint.The adapter changes only a fraction of the weights, whereas continued pretraining moves every weight.Source: §5.2 · p. 6

5.3 Persistence of the Gain

  • No replacement age appears within two decades of tested cutoffs: old bases adapt as well as new ones, and the oldest adapted checkpoints exceed the newest unadapted ones.The cumulative daily gain rises steadily across the evaluation window and is positive in every calendar year.Source: §5.3 · p. 6, Figure 6 · p. 6

5.4 Origin of the Gain

  • PALM shifts scores for articles about firms or events unseen by the vintage, and moves later-period words in both directions rather than simply rewarding familiarity.The tested examples include electric-vehicle makers, a commodity rally, and a clinical-trial setback.Source: §5.4 · pp. 6–7
  • Scrambling eligible text leaves +0.48 of the +1.26 IC gain, while using text beyond the decision date yields +1.05 versus +1.26 for eligible text and loses at p=0.008.These interventions indicate that text content matters and that the eligible corpus suffices for the reported gain.Source: §5.4 · p. 7, Table 9 · p. 7

5.5 Sensitivity Analysis

  • Cutting training steps to a third or the learning rate fivefold removes the gain, while doubling steps changes little; shifting the decision date by two years either way preserves it.Training must pass a floor, but more training above it adds little.Source: §5.5 · p. 7, Table 10 · p. 7, Table 11 · p. 7
  • Adapting either attention or feed-forward projections alone recovers roughly half the gain from adapting both, while one tenth of eligible text delivers four fifths of the full-corpus gain.The text experiment uses the most recent eligible slice at each corpus share.Source: §5.5 · pp. 7–8, Table 13 · p. 7

6 Limitations

The evidence is limited to one asset class over a single decade and the public PIT suites available today. The gain is also established under one evaluation-to-position protocol, and its size is not claimed to be invariant to those choices.

6 Limitations

  • Evidence covers one asset class, one decade, and currently public PIT suites; other markets, horizons, and much larger backbones remain untested.The measured signal is weak in absolute terms, so conclusions about stronger signals do not follow.Source: §6 · p. 8
  • The gain is established under one protocol, and its size is not claimed to be invariant to choices from score readout through position construction.Separating which protocol choices matter would require a study designed around that question.Source: §6 · p. 8

7 Conclusion

The paper finds that staleness does not degrade downstream performance when each vintage is compared on a fixed evaluation window. It proposes PALM, which adapts an existing checkpoint on eligible text without modifying pretrained weights, as a small-update alternative to yearly pretraining.

7 Conclusion

  • With the evaluation window fixed, comparing each vintage against its replacement shows that staleness does not degrade downstream performance.Source: §7 · p. 8
  • The authors suggest updating an existing checkpoint with a small adapter rather than producing a new checkpoint each year, preserving the suite’s chronological guarantee.They propose directing the freed budget toward more architectures rather than more years of one.Source: §7 · p. 8

A Dataset

The dataset aligns dated financial articles with the next trading session’s return and splits the panel at a decision date. The four point-in-time suites differ in how their yearly checkpoints are produced.

A Dataset

  • Articles are sampled evenly across months, assigned to a trading day only if published before its 20:00 UTC close, and scored against the following session’s adjusted-close return.The decision date splits the panel into eligible adapter-training text and a separate evaluation window; the panel counts article-days, so multiple same-ticker articles on one day remain separate rows.Source: A Dataset · p. 9

B Backbones

  • Three suites train each yearly vintage from scratch, while PIT-4B publishes monthly snapshots of a continuous run.The paper uses one PIT-4B checkpoint per calendar year; eligible checkpoint counts differ by release history and suite start date.Source: B Backbones · p. 9, Table 15 · p. 9
  • All four suites use their original release settings and a shared scoring routine, without architecture changes.The shared routine keeps cross-suite scoring consistent.Source: B Backbones · p. 9

C Implementation Details

The implementation freezes pretrained weights and uses a fixed scoring routine; evaluation reports three measures derived from the same daily series. Their construction distinguishes signal ranking from portfolio return and risk.

C Implementation Details

  • All pretrained parameters are frozen, optimization settings are not tuned, and scoring uses no trained parameter or sampling.Everything except adapter rank, steps, and learning rate is fixed; a zero-initialized adapter leaves the model unchanged until an optimizer step.Source: C Implementation Details · p. 10, Table 16 · p. 10

D Three Reported Measures

  • The three measures use the same daily series but differ in how they treat each day, and the tables report IC in 10^-3 units, annualized return in percentage points, and Wins as the share of paired checkpoints improved.The article, rather than the firm, is the cross-sectional unit; days with fewer than eight articles or four distinct scores are excluded.Source: D Three Reported Measures · p. 11

D.1 Information Coefficient

  • 6.61 in units of 10^-3 is the example checkpoint’s IC, the equal-weight average of daily within-day Spearman correlations between article scores and next-session returns.The within-day rank correlation removes common market moves and is invariant to score units; the example’s daily IC is +0.0067, and its average across 767 scored days is 0.00661.Source: §D.1 · p. 11

D.2 Annualized Return

  • 9.19% is the example checkpoint’s annualized return, computed as 252 times the daily mean of its equally weighted long-short position returns.The positions use the sign of each demeaned score, not its magnitude; no transaction costs, borrowing costs, or slippage are deducted.Source: §D.2 · p. 11

D.3 Sharpe Ratio

  • 1.65 is the example checkpoint’s Sharpe ratio; the daily return mean is 0.0004 and the daily standard deviation is 0.0035.The ratio is scaled to an annual figure by 252 under the assumption that daily returns have no serial correlation; it can move differently from return when volatility changes.Source: §D.3 · p. 11

E Staleness on All Three Measures

On a fixed evaluation window, newer vintages do not rank better, while PALM improves results. The section also examines checkpoint-level variation and adapter capacity.

E Staleness on All Three Measures

  • IC slopes are positive in every suite while portfolio slopes fall, and the newer vintage ranks no better and may trade slightly worse; PALM improves both.The portfolio disagreement is concentrated in ChronoGPT, the only suite with significant portfolio slopes; the table of regressions shows neither adapted scores nor PALM gains decline with staleness.Source: Table 17 · p. 12, E Staleness on All Three Measures · p. 12, Table 18 · p. 12

F Every Eligible Checkpoint

  • 40 of 44 eligible checkpoints improve in IC with PALM, while the four that do not span all three suites and cutoffs fifteen years apart.The checkpoint-by-checkpoint results show the average is not carried by only a handful of checkpoints.Source: Table 19 · p. 13, F Every Eligible Checkpoint · p. 13

G Adapter Capacity

  • All three reported metrics peak at adapter rank three and decline beyond it, while the best rank is four or less on narrow suites and sixteen on wide suites.Portfolio measures turn negative beyond rank twelve even as IC remains positive; the main-text rank is too small for the widest suite, not ineffective.Source: Table 20 · p. 14, Table 21 · p. 14, G Adapter Capacity · p. 14

H Robustness to the Direction Words

The reported PALM gain persists when the direction words are replaced. A worked example traces how the paper scores an article and what PALM changes in that process.

H Robustness to the Direction Words

  • The adapted checkpoint’s score stays within 0.05 across all three direction-word choices, while the unadapted score ranges from 4.81 to 6.65.The gain is larger under both alternative wordings than under the reported wording; the comparison covers 19 checkpoints from two suites.Source: H Robustness to the Direction Words · p. 15, Table 22 · p. 15

I A Worked Example of the Pipeline

  • The prompt substitutes the ticker and article text, truncates at 1,500 characters, and stops before the direction word; publication date is not included.The date is instead used outside the model to assign a trading day, define its cross-section, and select the return for scoring.Source: I A Worked Example of the Pipeline · p. 16
  • The model compares probability mass on two disjoint sets of direction-word continuations, with candidate sets compared at their first differing token position.For the worked airline article, the resulting score is -1.272, placing it at the bottom of that date’s cross-section.Source: I A Worked Example of the Pipeline · p. 16
  • For the worked date, the 91 article scores have IC_t = +0.0067 and a long-short return of -0.33%.The example checkpoint’s average IC across 767 scored days is 6.61 in units of 10^-3.Source: I A Worked Example of the Pipeline · p. 16, Table 23 · p. 16
  • PALM leaves the prompt, ranking, and return evaluation unchanged, changing only the score-producing weights by fitting an adapter on text published between the vintage cutoff and decision date.The same 91 articles are scored again after adaptation.Source: I A Worked Example of the Pipeline · p. 16

J Sensitivity of the Update

The appendix sweeps vary adapter settings against the same checkpoint without an adapter; changing the decision date also changes eligible vintages and the evaluation window.

Adaptation schedule

Decision date

  • Across seven dates, gains stay positive on all measures, but recent one- and two-year windows are too short to separate portfolio outcomes; IC improves most checkpoints at every date.The eligible pool changes with the decision date.Source: J Sensitivity of the Update · p. 17, Table 25 · p. 17

Share of the eligible text

Training budget

  • At 300 steps or a learning rate of 5 × 10−5, the IC gain is under one-third of the reported setting; beyond 1,200 steps, portfolio measures turn negative while IC stays positive.One setting is applied across suites, checkpoints, and dates, with no per-backbone tuning.Source: J Sensitivity of the Update · p. 18, Table 27 · p. 18

Tokens read per article

Random seed

Related papers

Loading 2609.30316v1…