PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Seismic Foundation Models More Than Glorified Feature Extractors?

Laura Laurenti, David Liu, Brian Kulis, Men-Andrin Meier, Christopher Johnson

arXiv:2610.38003v1cs.LGcs.AI

TL;DR

Seismic waveform foundation models use targets computed from single windows, while earthquake sequences also have sparse event-occurrence dynamics. The paper compares nine shared-backbone designs across three downstream tasks and finds that objective changes redistribute skill rather than yield a universal winner. It concludes that these models act as feature extractors by the imported language-and-vision standard, and argues that resolving the constraint requires targets spanning events.

  • Problem

    Waveform foundation models use single-window targets that do not represent the sparse, self-exciting occurrence process governing earthquake sequences.

    Source: §1 · pp. 1–2

  • Method

    The authors pretrain nine shared-backbone models varying objective, bottleneck, and mixer designs, then evaluate frozen encoders on three downstream tasks.

    Source: §2 · p. 2

  • Results

    Objective changes redistribute downstream skill rather than raising it, and no checkpoint wins everywhere; masking adds 6.9 points of few-shot P-precision but costs 9.6 points of few-shot classification accuracy.

    Source: §3 · p. 3, §5 · p. 4

  • Takeaways & Limitations

    The authors classify the models as feature extractors rather than foundation models under the language-and-vision standard and argue that a target defined across events is needed to settle the question.

    Source: §5 · p. 4, §4 · p. 4

    The study varies and evaluates only window-level targets, so it cannot establish whether single-window learning targets are the binding constraint.

    Source: §4 · p. 4

Abstract

from arXiv · show

Earthquake sequences are stochastic on two separated timescales: the seismic waves of any individual earthquake propagate over seconds to minutes, while the earthquakes themselves arrive as a sparse, self-exciting occurrence process, with inter-event intervals from seconds to months. Respecting both scales in one training objective is the open challenge. Foundation models (FMs) for seismic waveforms instead import their learning targets from audio and vision, where every target is computed from a single fixed-length window. Such a window holds one realization of the wave propagation (an earthquake trace), never the occurrence process that decides when earthquakes happen. We develop nine model designs sharing one Mamba encoder-decoder, trained on a fixed corpus of 3M windows, varying the self-supervised objective, quantization bottleneck and mixer; the two key comparisons are architecture-matched, varying the objective alone. Each checkpoint is evaluated frozen on three downstream tasks. Varying the target redistributes downstream skill rather than raising it, and no checkpoint wins everywhere. Adding masking pressure to our strongest classifier re-specializes it rather than making it a generalist: it gains 6.9 points of few-shot P-wave picking precision and loses 9.6 points of few-shot classification accuracy. By the standard imported from language and vision, these models are feature extractors rather than FMs. Controls on capacity, architecture, and domain proximity do not explain why skill moves rather than rises. Against task-specific baselines the advantage is regime-dependent: our checkpoints beat them few-shot on classification and ground-motion prediction, and trail them on full data. We argue the binding constraint is the target, and that settling the question requires a learning target defined across events, the timescale that makes seismicity stochastic in the first place.

1 Introduction

Seismicity unfolds both within earthquake traces and across event catalogs, but waveform foundation models train on fixed windows that omit event occurrence. The paper tests whether changing these window-level targets redistributes downstream skill under controlled conditions.

1 Introduction

  • Earthquakes are continuous, non-stationary waveforms within traces but sparse, self-exciting events across catalogs, with intervals ranging from seconds to months.The catalog-level quantities used to characterize sequences include when the next event occurs and its magnitude.Source: §1 · p. 1
  • Existing seismic waveform foundation-model targets operate on single fixed-length windows and do not reference earthquake occurrence dynamics.This window-level framing is sufficient for local tasks such as phase picking, but not for patterns of occurrence.Source: §1 · p. 2
  • The study varies targets while fixing the corpus, heads, and splits, with architecture also fixed in its two key comparisons, and finds skill redistribution rather than improvement.Controls on capacity, architecture, and domain proximity do not explain the ceiling, though the authors distinguish this finding from their proposed explanation.Source: §1 · p. 2

2 Setup

Nine models share a Mamba encoder-decoder backbone and a paired earthquake/noise pretraining corpus. They vary objectives, bottlenecks, and mixers, then evaluate frozen encoders across three downstream tasks under distribution shift.

2 Setup

  • Nine models use a shared SEANet-style convolutional encoder-decoder paired stage-by-stage with Mamba layers, processing five-minute, three-component, 100 Hz traces.The backbone is held fixed to isolate target effects; pretraining pairs about 1.6 million CREW earthquake waveforms with about 1.5 million ambient-noise windows.Source: §2 · p. 2
  • The models vary objective composition, continuous versus residual-vector-quantized bottlenecks, and SSM-only versus SSM-plus-gated-MLP mixers.The four Rec-family models share an architecture and vary only loss composition; +MaskRaw and +MaskCode match RVQ-EMA+MLP in state-dict shape and add training-time objectives discarded at inference.Source: §2 · p. 2
  • Frozen checkpoints are tested on foreshock/aftershock classification, ground-motion regression, and P/S arrival detection, with full-data and few-shot evaluations against task-specific models.None of the three downstream corpora appears in pretraining, making each evaluation a cross-corpus generalization test.Source: §2 · p. 2

3 Results

Objective changes redistribute skill across tasks rather than producing a universally strongest checkpoint. Masking can trade classification performance for picking precision, while controls do not account for the observed ceiling.

3 Results

  • No checkpoint wins across tasks: RVQ-EMA+MLP leads classification, while Rec+Disc+Mag+Mask performs better on ground motion and S-picking.Table 1 reports frozen-encoder results for full-data and few-shot settings; its caption notes that no ordering survives across tasks.Source: §3 · p. 2, Table 1 · p. 3
  • Against task-specific baselines, the best checkpoint wins few-shot classification but trails on full-data classification; ground-motion and picking comparisons also vary by task and label regime.The CNN classification comparison is 85.5 vs. 82.4 few-shot and 96.7 vs. 99.2 full-data; on ground motion, every checkpoint beats the GCN baseline few-shot, whereas only Rec+Disc+Mag+Mask does so on all five targets full-data.Source: §3 · p. 3
  • Adding masking to RVQ-EMA+MLP raises few-shot P-precision by 6.9 points but lowers few-shot classification accuracy by 9.6 points.+MaskRaw changes P-precision from 74.1 to 81.0 and classification from 85.5 to 75.9; +MaskCode trades less in both directions.Source: §3 · p. 3
  • Classification controls find that roughly 11× width scaling recovers only part of a corpus-induced deficit, bidirectional Mamba changes nothing measurable, and adding domain-matched traces leaves metrics tied.The full-data split is insensitive to most tested variations, while few-shot performance is the discriminating and fragile metric.Source: §3 · p. 3

4 Analysis and open problems

The analysis demonstrates that a point target can collapse to a near-zero conditional mean, while the claim that single-window targets miss event-occurrence dynamics remains motivated but untested. It identifies target design, rather than architecture, as the focus of four open problems.

4 Analysis and open problems

  • A point target can yield a flat forecast 5–10× below the true spectrum even at the squared-error optimum, while its learned features can outperform those from forecasts that look right.The authors attribute the collapse to a near-zero conditional mean when the future waveform phase is nearly independent of context; the collapsed arm’s features are ∼4.5 points better than the envelope arm’s.Source: §4 · pp. 3–4
  • The claim that single-window targets discard occurrence dynamics is motivated, not tested; all varied objectives and scored tasks remain window-level.Separating this explanation from ordinary task conflict at fixed capacity requires an occurrence-level target.Source: §4 · p. 4
  • A two-timescale target could couple waveform representations with a marked temporal point process so waveform content and event history condition each other.The proposed formulation could support prospective foreshock questions, which the paper says it can currently ask only after the fact.Source: §4 · p. 4
  • Tail-aware generative targets should weight rare events by their importance rather than their frequency, given the demonstrated conditional-mean collapse.The paper frames this as an open question about objective weighting, not a tested solution.Source: §4 · pp. 3–4
  • Without a stated cost structure, the study cannot determine how to weigh classification losses against picking gains or which checkpoint to deploy.The authors propose proper scoring rules, calibration targets, and explicit cross-task costs for evaluating learned stochastic simulators.Source: §4 · p. 4
  • Targets based on a forward-model residual or neural-operator basis could reserve capacity for the stochastic component left undetermined by known wave physics.The paper notes that these approaches exist for solving physics, but not as pretraining targets.Source: §4 · p. 4

5 Conclusion

Across three frozen-encoder tasks, objective changes redistribute skill rather than raise it, and no checkpoint generalizes without a trade-off. The authors conclude that these models are competent pretrained feature extractors, not foundation models by the language-and-vision standard.

5 Conclusion

  • Across three tasks, varying the objective redistributes downstream skill rather than raising it, and no checkpoint generalizes without giving something up.The paper demonstrates point-target collapse to a near-zero conditional mean but only motivates, rather than tests, the claim that single-window targets omit occurrence dynamics.Source: §5 · p. 4

A.1 Backbone

The shared backbone pairs a SEANet-style convolutional encoder–decoder with Mamba layers at each stage and a symmetric decoder.

A.1 Backbone

  • The SEANet-style encoder–decoder uses three strided downsampling stages, each paired with Mamba selective state-space layers, followed by a symmetric decoder.The stages use strides 3, 4, and 5; Mamba is chosen over attention for linear-time scaling on 3 × 30,000-sample windows, reduced to ∼500 latent frames.Source: §A.1 · p. 8

A.2 Objectives

The objectives combine waveform and spectral reconstruction with optional adversarial, magnitude, or masking components; masking can operate on the input or an auxiliary encoder-output branch.

A.2 Objectives

  • Rec, Disc, Mag, and Mask add distinct loss components, while +MaskCode masks a separate encoder-output copy for an auxiliary code-prediction head discarded at inference.Rec combines raw-waveform and one-sided-spectrum L1/L2 losses; +MaskCode predicts the first three RVQ levels.Source: §A.2 · p. 8

A.3 Corpora

The CREW corpus supplies regional waveform traces, while the ambient-noise set provides training, validation, and test windows from hundreds of networks.

  • CREW contains approximately 1.6 million five-minute, three-component regional traces spanning 1–20°.Source: §A.3 · p. 8
  • The ambient-noise corpus includes 1,499,918 training windows, about 125,000 hours, from 462 FDSN networks.Source: §A.3 · p. 8
  • The ambient-noise set also has 169,832 validation and 162,846 test windows, with waveforms peak-normalized jointly across components to preserve amplitude ratios.Source: §A.3 · p. 8

A.4 Downstream protocol

The downstream protocol evaluates frozen encoders on classification, ground-motion prediction, and phase detection, using task-specific data splits and few-shot regimes.

  • Foreshock/aftershock classification uses 25-second NRCA windows from two months before and after the mainshock, with 70/10/20 full-data and 2% few-shot splits.Source: §A.4 · p. 8
  • Ground-motion prediction uses 915 events across 39 INGV stations and predicts PGA, PGV, and PSA at 0.3, 1.0, and 3.0 seconds.The protocol uses both a station-distance GCN head and an independent-station linear head, with 65% full-data and 8% few-shot.Source: §A.4 · p. 8
  • Phase detection labels each sample as noise, P, or S on 143,523 STEAD training traces, counting arrivals within 50 samples as correct.The few-shot regime uses 0.1% of training data; all reported results use frozen encoders.Source: §A.4 · p. 8

A.5 Scale, architecture and corpus controls

The controls vary model scale, mixer, and corpus, while the forecasting experiment compares objectives on a domain-first corpus. Forecast-loss choices yield different few-shot classification results, with a data-recipe caveat on comparisons to Table 1.

A.5 Scale, architecture and corpus controls

  • The controls include width scaling, bidirectional Mamba at approximately twice the mixer cost, and proportional versus domain-first corpus mixtures.The width-scaling runs use a pooled corpus and shorter windows, while bidirectional Mamba uses the CREW-plus-noise corpus.Source: §A.5 · p. 8

A.6 The forecasting experiment

  • The separate forecasting experiment uses approximately 3.35 million traces from STEAD, INSTANCE, SCEDC, and ETHZ, with CREW capped at about one tenth.The data are cropped to a common 6000-sample window.Source: §A.6 · p. 8
  • 86.1 ± 1.0 few-shot classification for Arm 1 versus 81.6 ± 0.9 for Arm 2, while Arm 1 produces the collapsed forecast described in the paper.The matched control reaches 78.9 few-shot versus Table 1’s 85.5; the passage attributes that entire gap to the data recipe, not the objective.Source: §A.6 · p. 9
  • The figure contrasts fixed-clock corpora with event-driven seismicity, stationary energy with rare transients, and a known audio band with task-dependent seismic spectra.It frames the window-level target as unable to reference earthquake occurrence and compression as selecting which downstream tasks retain useful signal.Source: Figure A1 · p. 10

Related papers

Loading 2610.38003v1…