Resilient Contrastive Pre-training under Non-Stationary Drift

Xiaoyu Yang; Jie Lu; En Yu; Wei Duan

Source-linked AI summary

Resilient Contrastive Pre-training under Non-Stationary Drift

Xiaoyu Yang, Jie Lu, En Yu, Wei Duan

arXiv:2502.07620v3Feb 11, 2025cs.LGcs.CV

TL;DR

Dynamic data streams expose a gap in contrastive pre-training, whose momentum-based methods can accumulate drift-induced bias and destabilize representations. The paper develops RCP, using a structural causal model and causal intervention to address this bias, and reports stronger downstream performance with more robust representations across drift settings.

Problem
Contrastive pre-training must reliably learn from non-stationary streams despite concept drift, which can degrade representation quality and destabilize pre-trained backbones.
Source: §1 · p. 1, §5.3 · p. 14
Method
RCP uses a structural causal model and causal intervention to mitigate bias accumulated by momentum updates under concept drift.
Source: §2.2 · p. 4, §2.3 · p. 5, §4 · pp. 8–9
Results
RCP outperforms other contrastive pre-training methods in fine-tuning results and demonstrates effectiveness across long-tailed classification, OOD detection, and domain adaptation.
Source: §1 · p. 2, §3.1 · p. 5
Takeaways & Limitations
RCP provides a straightforward and scalable approach for learning more robust and generalizable representations from evolving data streams.
Source: §1 · p. 2, §4 · pp. 8–9
The standard continuous contrastive objective does not explicitly handle accumulated bias caused by drift, and RCP relies on a wider drift adaptation window.
Source: §2.1 · p. 3, §3.5 · p. 8

Abstract

from arXiv · show

The remarkable success of large-scale contrastive pre-training has been largely driven by by vast yet static datasets. However, as the scaling paradigm evolves, this paradigm encounters a fundamental challenge when applied to dynamic data streams characterized by concept drift - unpredictable changes in the underlying data distribution. This paper aims to advance robust pre-training under such non-stationary environments. We begin by revealing that conventional contrastive pre-training methods are highly susceptible to concept drift, resulting in significant substantial bias and instability within the learned feature representations. To systematically analyze these effects, we develop a structural causal model that elucidates how drift acts as a confounder, distorting the learned representations. Based on these causal insights, we propose Resilient Contrastive Pre-training (RCP), a novel method that incorporates causal intervention. RCP formulates a causally-informed objective to mitigate drift-induced biases through targeted interventions. The method is designed for simple and scalable implementation and exhibits notable adaptability, promoting robust and autonomous pre-training on non-stationary data. Comprehensive experiments across various downstream tasks consistently demonstrate that RCP effectively alleviates the detrimental impact of concept drift, yielding more resilient and generalizable representations.

1. Introduction

Contrastive pre-training is moving from static datasets to drifting streams, where temporal shifts can bias teachers and destabilize representations. The paper diagnoses drift causally and introduces RCP to mitigate these effects.

Sequential data from heterogeneous sources exhibit non-IID characteristics and concept drift, including changing class frequencies, domains, seasons, and sensor features.Source: §1 · p. 1
Standard contrastive methods are sensitive to temporal distribution shifts, producing biased momentum teachers, collapsed similarity structures, and unstable representations.Source: §1 · p. 1
EMA teachers retain outdated knowledge under drift, amplifying misalignment between learned features and current data characteristics.Source: §1 · pp. 1–2
Under long-tailed drift, tail categories collapse while head categories dominate the feature manifold, unlike the evenly distributed space from balanced pre-training.The analysis attributes this distortion to bias accumulation in the momentum-updated teacher, which receives repeated head-class exposure.Source: §1 · p. 2, Figure 1 · p. 2
The structural causal model identifies drift D as a confounder creating the backdoor path X ←D →B →Y and confounding the direct effect of X on predictions.Source: §1 · p. 2
RCP uses causal intervention to mitigate drift-propagated bias and supports robust, generalizable representations across long-tailed classification, OOD detection, and domain adaptation.Source: §1 · p. 2

2. Methodology

The methodology models concept drift as a causal confounder in contrastive pre-training, then uses intervention to reduce drift-induced bias. RCP implements this objective within a momentum encoder framework using drift-window sampling and an intervention module.

2.1. Causal Lens on Contrastive Drift Pre-training

High-momentum teachers can become misaligned with the current distribution, accumulating stale biases that hinder adaptation to evolving concepts.The standard objective synchronizes student and momentum models over an adaptation window but does not explicitly handle accumulated bias B.Source: §2.1 · p. 3
Concept drift D affects features X and teacher bias B, creating backdoor and mediation paths that distort predictions Y.The backdoor path X ←D →B →Y introduces spurious correlations, while X →B →Y weakens the direct feature effect.Source: §2.1 · p. 3

2.2. Causal Interventional Contrastive Objective

Causal intervention replaces P_t(Y|X) with P_t(Y|do(X)) by stratifying drift contexts, removing D’s confounding pathway and targeting the genuine X →Y effect.The intervention integrates confounding factors and is approximated with a single-pass feed-forward estimator to avoid costly sampling.Source: §2.2 · p. 4

2.3. Resilient Contrastive Pre-training

RCP adapts a MoCo v3 student–teacher architecture by projecting student features into queries, producing momentum keys, and computing an interventional score from q, k, and v.The resulting network is trained with InfoNCE over one positive and all negative samples in the drift adaptation window.Source: §2.3 · p. 5
RCP’s query-key-value mechanism samples drift within the adaptation window to mitigate bias in the value, while remaining simple, efficient, and scalable.Although structurally similar to self-attention, its stated purpose is drift-bias mitigation rather than sequence-position modeling.Source: §2.3 · p. 5

3. Experiments

Across long-tailed classification, domain-shift and OOD evaluation, feature-space analysis, scaling, and adaptation-window ablations, RCP consistently improves resilience to drift. The experiments also examine whether these gains persist across model sizes and adaptation settings.

3.1. Navigating Tailed Drift Pre-training

RCP outperforms other contrastive pre-training methods on long-tailed classification, including a 7.9% accuracy on ImageNet-LT Few split under linear probing.Fine-tuning results also show additional gains on Medium and Few splits, especially on iNaturalist2018.Source: §3.1 · pp. 5–6, Table 2 · p. 6

3.2. Taming ID and OOD Drift

RCP achieves superior domain-shift results across ImageNet-V2, ImageNet-Sketch, and ImageNet-R, including stronger tail-category feature extraction under shift.The reported explanation is that RCP captures essential image information while reducing interference from drift.Source: §3.2 · p. 7, Table 3 · p. 7
RCP surpasses contrastive baselines overall for OOD detection, although DINO performs slightly better on ImageNet-O.The method is reported to produce compact intra-class features and separable decision boundaries.Source: §3.2 · p. 7, Table 4 · p. 7

3.3. Boosting Pre-training Feature Embedding

RCP attains optimal ID-versus-OOD separability, while pre-training mainly improves intra-class compactness rather than inter-class separability.The analysis links stronger ID-versus-OOD separability with improved ID intra-class compactness and identifies negative samples as important for inter-class separation.Source: §3.3 · pp. 7–8, Table 5 · p. 8

3.4. Scaling with Drift Pre-training

RCP scales with model size, delivering nearly a 4% enhancement with ViT-B/16 compared with ViT-S/16 on downstream classification.The experiments report consistent scaling behavior despite substantial distribution drift and support larger minimally curated pre-training datasets.Source: §3.4 · p. 8, Table 6 · p. 8

3.5. Ablation on Adaptation Window Length

A wider drift adaptation window yields superior RCP results, and RCP consistently outperforms MoCo v3 on Few split.This ablation indicates that the method benefits from adapting over a larger temporal window.Source: §3.5 · p. 8, Table 7 · p. 8

4. Conclusion and Outlooks

The paper introduces RCP, which uses causal inference to examine momentum-update bias and applies intervention to mitigate drift-induced bias during pre-training.

RCP is presented as a straightforward and effective pre-training paradigm tailored to concept-drift data streams.Source: §4 · p. 8
RCP uses causal inference to examine the source of bias in contrastive pre-training’s momentum update.Source: §4 · p. 9
Its causal interventional objective mitigates bias within drifting data streams and counters unpredictable distribution changes.Source: §4 · p. 9

5. Related Works

Related work spans concept-drift adaptation, causal debiasing, and contrastive pre-training, while emphasizing that conventional contrastive methods assume stationarity and lack drift adaptation.

5.1. Concept Drift

Distribution-based drift methods identify distribution changes and characterize their timing, affected features, and severity.Source: §5.1 · p. 12
Existing drift approaches include adaptive boosting, long-tailed calibration, group-drift testing, distribution forecasting, and teacher-student detection.Source: §5.1 · p. 12

5.2. Causal Inference

Causal-inference methods use backdoor, front-door, counterfactual, and structural-causal interventions to mitigate dataset, multimodal, reasoning, and representation biases.Source: §5.2 · pp. 12–13

5.3. Contrastive Pre-training

SimCLR [4], MoCo [16], MoCo v3 [6], BYOL [15], DINO [1], and DINO v2 establish contrastive or self-distillation mechanisms for invariant and stable representations.Source: §5.3 · p. 13
DenseCL [62] adds localized, multi-scale feature alignment for dense prediction, highlighting feature stabilization beyond image-level contrast.Source: §5.3 · p. 13
Contemporary contrastive methods rely on closed-world stationarity assumptions and therefore lack mechanisms for adapting to evolving data streams.Source: §5.3 · p. 14

6. Simple and Efficient Implementation with Scalability Advantages

RCP is designed for simple integration into contrastive pipelines and to scale across expanding data and larger models under drift.

RCP requires minimal modifications to architectures such as MoCo v3 [6] by adding an intervention module over student and momentum-encoder embeddings.Source: §6 · p. 14
A masked contrastive strategy supports larger effective batch sizes for capturing broader drift-adaptation windows.Source: §6 · p. 14
RCP is designed to accommodate expanding datasets while mitigating drift-induced degradation and supporting scaling with data volume and variety.Source: §6 · p. 14
RCP is intended to improve stability when pre-training larger models in dynamic environments, including models with more parameters and deeper architectures.Source: §6 · p. 14

7. Experiments

The experiments use long-tailed, fine-grained, OOD, domain-shift, and zero-shot benchmarks with multiple ViT backbones and implementation settings. Additional experiments report RCP-module improvements with DINO and applicability across momentum-teacher frameworks.

7.1. Datasets

The evaluation covers ImageNet-LT [33], iNaturalist2018 [55], OOD datasets, domain-shift datasets, and TIC-DataComp(L).Source: §7.1 · p. 14, §7.3 · p. 15
ImageNet-LT [33] contains 115k training images across 1,000 categories with class frequencies ranging from 1,280 head examples to 5 tail examples.Source: §7.1 · p. 14

7.2. Implementation Details

The experiments use ViT-Small, ViT-Base, and ViT-Large visual backbones, spanning approximately 22 million, 86 million, and 307 million parameters.Source: §7.2 · p. 15
Pre-training uses AdamW with cosine annealing, 40 warm-up epochs, and 800 epochs on 2 × 2 NVIDIA A100 GPUs unless otherwise specified.Source: §7.2 · p. 15
Fine-tuning uses model-specific learning rates and epoch settings for ViT-S/16, ViT-B/16, and ViT-L/16.Source: §7.2 · p. 15

7.3. Additional Experiments

Zero-shot experiments on TIC-DataComp(L) report superior results for the proposed methods compared with other methods on those datasets.Source: §7.3 · p. 15, Table 10 · p. 15
Adding the proposed module improves DINO under non-stationary drift, and the intervention is reported as agnostic to the backbone or contrastive framework.Source: §7.3 · pp. 15–16, Table 11 · p. 16