Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting

1 INTRODUCTION

• Pre-specifying a monitoring measure forces a sensitivity–breadth trade-off, especially for high-dimensional or unstructured data with many possible shift directions.A measure aligned with the future shift can detect quickly, while poor alignment may yield little or no detection power.Source: §1 · p. 1
• Testing-by-betting permits adaptation when returns are centered under the null; the bettor’s accumulated wealth acts as evidence, and crossing a threshold triggers rejection.At each step, the bettor uses past data to choose a witness function and bet size, then compares the new observation with its expected value under the pre-change distribution.Source: §1 · p. 2
• For a single unstructured stream, prior adaptive changepoint methods are limited, and a suitable pre-change reference stream may be unavailable.For example, a newly deployed hospital AI scribe has limited pre-change data.Source: §1 · p. 2
• Across three real-world text and image streams, adaptive monitors outperform fixed counterparts; LLM-concept monitors are fastest when shifts are sparse and interpretable.The paper also reports that these monitors can recover shifted concepts and guide root-cause diagnosis for complex shifts.Source: §1 · p. 2
• Asymptotic e-detectors approximate the pre-change distribution with a finite calibration set and asymptotically control ARL0 as its size increases.The authors report effective ARL0 control with calibration sets as small as m = 30.Source: §1 · p. 2

2 RELATED WORK

• GenAI monitoring often uses pre-specified output scores or shift directions, which can miss or poorly align with actual drift.The accompanying overview depicts an adaptive bettor using past observations, calibration comparisons, and an accumulated-wealth alarm threshold.Source: §2 · pp. 2–3, Figure 1 · p. 3
• Classic changepoint procedures often require known or parametric pre-change distributions, whereas e-detectors offer ARL0 guarantees but existing methods rely on stream structures suited to martingale-like requirements.The paper identifies a gap in e-detectors for monitoring a single unstructured stream.Source: §2 · p. 3
• Prior shift interpretation is often post hoc, while white-box detection typically requires pre-specified directions; this work instead learns interpretable shift directions.The cited prior work includes learning natural-language explanations and searching concepts for classification performance.Source: §2 · p. 3

3.1 E-detectors for the idealized game

• An e-detector is a nonnegative adapted process whose expected value at any stopping time is bounded by the expected stopping time under the null.Crossing 1/α then guarantees ARL0 of at least 1/α.Source: §3.1 · p. 4
• In the ideal game, predictable bounded bets use centered witness-function returns; SR aggregates subgame wealth by summing, while CUSUM takes its maximum.The true pre-change expectation needed to center returns is unavailable in practice.Source: §3.1 · pp. 4–5

3.2 Asymptotic e-detectors for the approximate game

• A calibration set estimates the pre-change witness expectation by its empirical mean, and replacing the ideal return with this estimate yields the approximate detector.The witness function must be independent of the calibration dataset for the empirical average to estimate the null expectation uniformly consistently.Source: §3.2 · p. 5
• The asymptotic guarantee assumes bounded, predictable witnesses from a fixed P-Glivenko-Cantelli class, an independent IID calibration set, and uniformly integrable null stopping times.The witness functions cannot depend on the calibration data or downstream variables derived from it.Source: §3.2 · p. 5
• Under Assumption 2, the approximate detector asymptotically controls ARL0: lim inf_m→∞ EH0[τ_m] ≥ 1/α for any α ∈ (0, 1).The theorem concerns asymptotic control as calibration size m increases, not a convergence-rate guarantee.Source: §3.2 · p. 5

4 BETTING STRATEGIES

• Classifier-based Betting adaptively learns classifiers to distinguish pre- and post-change data rather than relying on a pre-specified distance measure.The framing is useful for unstructured data, where pre-specifying distances is difficult but AI/ML models can learn discriminators.Source: §4 · pp. 5–6
• Each betting step selects a representation, estimates a changepoint, fits a classifier to the resulting pre/post groups, and selects a bet scaling factor.This work focuses on representation, changepoint estimation, and classifier fitting; it fixes the scaling factor for simplicity.Source: §4 · p. 6

4.1 Bettor for black-box shifts

• Embedding bettors target broad shift detection when prior knowledge is limited or the goal is to detect any shift.One implementation estimates a changepoint in embedding space and fits logistic regression to the learned split.Source: §4.1 · p. 6

4.2 Bettor for interpretable shifts

• Interpretable bets restrict monitoring to practically relevant shifts and can use prior knowledge to accelerate detection relative to black-box bets.The bettor represents adaptively selected concepts as features; the proposed LLM pipeline brainstorms keyphrases, clusters them, identifies changing clusters, and proposes yes/no concepts.Source: §4.2 · p. 6
• Pairing interpretable concepts with sparse changepoint detection can identify when and which subset of features shifted.The method searches subsets up to a maximum size and can use an elastic-net penalty for logistic regression.Source: §4.2 · pp. 6–7

5 EXPERIMENTS

• The evaluation covers injected topical shifts in CEBaB reviews, injected bird-attribute shifts, and a real-world clinical AI-scribe shift.Adaptive bettors use embeddings, sparse autoencoder features, or GPT-4.1-mini concepts, with an Ensemble averaging Embedding and LLM-concept detectors; experiments use calibration sets of m = 30.Source: §5 · p. 7

5.1 Monitoring restaurant reviews in CEBaB

• On CEBaB reviews, LLM-concept quickly learned the ambiance concept and achieved the lowest EDD, while Ensemble ranked second fastest.All procedures with null streams were well calibrated, with empirical ARL0 bands containing or exceeding the desired level.Source: §5.1 · p. 7

5 EXPERIMENTS

• Across restaurant reviews, bird images, and AI-scribe data, adaptive monitors outperform fixed monitors, while the fastest adaptive methods vary by setting.LLM-concept has the lowest EDD on restaurant reviews; Ensemble and LLM-concept rank first and second on bird shifts; Ensemble and Embedding lead on AI-scribe data.Source: §5.2 · p. 7, §5.1 · p. 7, §5.3 · p. 8

5.2 Monitoring bird images from CUB

• On bird images, Ensemble and LLM-concept were the two fastest detectors across both shifts, and LLM-concept learned matching or overlapping concepts across most streams.For white crowns, the learned concept could be the correct concept or its opposite, such as a dark head or crown.Source: §5.2 · p. 7

5.3 Monitoring AI scribe data

• On AI-scribe data, the model update made drafts more concise, and Ensemble and Embedding outperformed SAE and LLM-concept alone.The AI operations team found the interpretable bettor actionable for considering future improvements and unanticipated effects of model updates; rheumatologist feedback also reported improved structure, less unnecessary information, and more precise medication information.Source: §5.3 · p. 8

6 DISCUSSION

• Adaptive bettors detected shifts faster than fixed monitors; LLM-concept bettors were fastest for sparse interpretable shifts, black-box bettors for broad shifts, and their ensemble performed near the top across settings.LLM-concept bettors also identified practically relevant shifts that often coincided with the true shift in experiments.Source: §6 · p. 8
• LLM-identified shifts may be correlated with, rather than the actual root cause of, a shift, so expert interpretation remains needed for comprehensive monitoring and diagnosis.Source: §6 · pp. 8–9

AI use statement

• The authors used generative AI for tasks including dataset cleaning, proof assistance, methodology feedback, implementation, figure creation, code editing, and literature summarization, with AI-assisted work reviewed by the authors.They report that LLM-generated code was tested and LLM-assisted proofs were checked line by line.Source: AI use statement · p. 9

• The document’s views, findings, and interpretations are those of its authors and do not represent or constitute an endorsement by the FDA/HHS or U.S. Government.Source: Disclaimers · p. 9
• The document does not constitute FDA guidance, an FDA position, or legally enforceable requirements.Source: Disclaimers · p. 9

CHECKLIST

• The checklist reports that theoretical assumptions and proofs are included, and points to the relevant sections and appendices for explanations and analysis.Source: CHECKLIST · p. 13
• The checklist reports that materials for reproducing empirical results, training details, error statistics, and computing infrastructure are provided.Source: CHECKLIST · p. 13

A THE ASYMPTOTIC E-DETECTOR FRAMEWORK AND PROOFS

• The monitoring setup uses a calibration sample from the pre-change distribution and evaluates average run length under the null and expected detection delay under the alternative.Source: A THE ASYMPTOTIC E-DETECTOR FRAMEWORK… · p. 14

A.1 Testing and monitoring by betting

• Betting updates wealth using returns, while an e-detector aggregates subgames started at candidate change times to handle an unknown changepoint.The passages describe summing subgame wealth and note that the Shiryaev–Roberts recursion permits a single multiplication per step when returns do not depend on the subgame index.Source: §A.1 · p. 14

A.2 Proof of Theorem 1

• The theorem’s proof establishes that the asymptotic e-detector has asymptotic ARL0 control under the null.As calibration size increases, the limit inferior of expected run length under the null is at least 1/α.Source: §A.2 · pp. 15–16
• The proof assumes bounded, predictable witness functions in a fixed P-Glivenko-Cantelli class and a calibration sample independent of the monitoring stream.Source: §A.2 · p. 15
• The argument controls the accumulated drift by bounding the conditional return bias from the empirical calibration distribution and showing its contribution vanishes as calibration size grows.It also invokes uniform integrability of the stopping times under the null.Source: §A.2 · pp. 15–16

B.1 Adaptive betting hyperparameters

• The bet scaling factor trades faster, higher-variance betting at high λ against slower, more stable betting at low λ; experiments fix λ = 0.5 and leave tuning for future work.Adaptive tuning is possible within the e-detector framework.Source: §B.1 · p. 16
• The changepoint learner uses a mean-shift estimator for dense embedding shifts and a sparse estimator for shifts affecting a subset of features, with maximum subset size B = 4 for interpretable concepts.The experiments use the mean-shift estimator for the black-box bettor and the sparse estimator for the interpretable bettor.Source: §B.1 · p. 16
• The classifier must produce calibrated probabilities; the implementation uses logistic regression and regularization to limit overfitting with high-dimensional representations.Regularization is applied to the high-dimensional setting.Source: §B.1 · p. 16
• Larger calibration sets improve the empirical approximation to the pre-change distribution and tighten ARL0 control, but require more pre-change data and, for interpretable bettors, annotation costs linear in m.The experiments found robust ARL0 control across calibration sizes, including m = 30.Source: §B.1 · p. 17

B.2 The LLM-concept bettor

• The LLM-concept pipeline clusters extracted keyphrases into topics, maps observations to topic-presence vectors, and uses sparse changepoint estimation to identify candidate changepoints and shifted topics.The implementation uses BERTopic with incremental PCA and mini-batch k-means.Source: §B.2 · pp. 17–18
• The LLM proposes yes/no concepts from top-shifted topics, keywords, representative documents, and topic prevalence changes; it then annotates observations and sparse changepoint estimation selects concepts for betting.The selected concepts and LLM define the representation function used for betting.Source: §B.2 · p. 18
• The LLM concept extractor must not exhibit temporal shifts, because such shifts would violate ARL0 guarantees; the implementation processes observations and concepts independently and retries failed queries.Future work may address temporal consistency over longer periods if providers silently change model versions or hardware.Source: §B.2 · pp. 18–19

C.1 Experimental settings

• Most monitors use 100 simulation replicates, while LLM-concept and ensemble monitors use 30 because of cost constraints; the LLM-concept bettor inherits the non-LLM ARL0 control conditions described for its feature selection and calibration.The non-LLM bettors’ simultaneous 95% bands on CEBaB and CUB null streams contain or lie above nominal ARL0.Source: §C.1 · p. 19
• CEBaB and CUB shifts are injected by upsampling observations with a target attribute in the post-change pool, while null streams draw only from the pre-change pool.The CUB population is restricted to seabird genera, and white-crown labels use species-level consensus.Source: §C.1 · p. 19
• The AI-scribe dataset contains rheumatology HPI sections from 60 days before and after a vendor model update, with 300 pre-update and 190 post-update notes.Each observation is a word-level deletion, insertion, or replacement between an AI-generated draft and the clinician-edited final note.Source: §C.1 · p. 20
• Adaptive bettors use m = 30 pre-change calibration observations and λ = 0.5; the ensemble averages the LLM-concept and embedding detectors on the same stream.The SAE and LLM-concept bettors use sparse feature selection, while BERTopic clusters keyphrases into 25 topics at each update.Source: §C.1 · p. 20
• Fixed monitors use conformal p-values with centroid or text-defined cosine directions, including informed, unrelated, and oracle directions.The oracle direction is the normalized difference between the post- and pre-change pool mean embeddings.Source: §C.1 · p. 20

C.2 Additional results

• Empirical ARL0 on CEBaB and CUB null streams is shown as controlled, with simultaneous confidence bands containing or lying above the nominal ARL0.The CEBaB ambiance and CUB yellow-bill and white-crown settings are shown separately.Source: §C.2 · p. 20, Figure C.1 · p. 21, Figure C.2 · p. 21
• AI-scribe concepts active at the ends of 18 streams are grouped into themes and compared by their mean prevalence before and after the changepoint.An LLM judge assigns themes from concept text using a codebook informed by the vendor’s stated update purpose and clinicians’ reports.Source: §C.2 · p. 21, Table C.3 · p. 22
• Median LLM-concept stream costs were $2.26 on CEBaB, $3.42 on CUB yellow bill, $1.64 on CUB white crown, and $13.59 on AI scribe.A stream took a median of 27 minutes, ranging from 11 minutes to 1.8 hours.Source: §C.2 · p. 21

D.1 Evaluating ARL0 control for varying calibration dataset sizes

• The simulations vary calibration size across m ∈ {10, 30, 100, 1000} for a fixed Beta(2, 2) score and a learned classifier-refit score.The learned-score setting uses five-dimensional standard normal data with continual changepoint estimation and logistic-classifier retraining.Source: §D.1 · p. 22
• With m = 30, the null alarm-time distribution is similar to the oracle’s for both fixed and learned scores at 1/α = 300.The comparison averages over calibration datasets; each curve uses 2,000 null streams.Source: §D.1 · p. 22, Figure D.2 · p. 23
• Empirical ARL0 exceeds nominal ARL0 in nearly every setting except m = 10 at target ARL0 = 10000; fixed scores are especially conservative, while learned scores are closer to nominal.Figure D.1 reports values as lower bounds when streams have no alarm by 100,000 steps.Source: §D.1 · p. 22, Figure D.1 · p. 24