The CLEAR Benchmark: Continual LEArning on Real-World Imagery

2.1 Existing CL Datasets and Benchmarks

• Prior benchmarks often use synthetic temporal changes, including pixel permutations, class splits, or manually altered image conditions, whereas real-world shifts tend to be smoother.The authors argue abrupt synthetic changes may hinder transfer to future tasks and report that GDumb falls short on CLEAR.Source: §1 · p. 2

1 Introduction

• CLEAR is a continual image-recognition benchmark curated from YFCC100M, using image timestamps to form a 2004–2014 stream; its labeled portion covers 10 temporally dynamic classes plus an 11th background class across buckets 1–10.Source: §1 · p. 2
• A CLIP-based, text-prompted curation pipeline followed by crowdsourced quality assurance makes CLEAR efficient to build and scalable to larger image collections.The authors report one day of engineering effort for CLEAR’s curation.Source: §1 · pp. 2–3
• Streaming evaluation tests a model trained on one period against the next, and the evaluated period can then become the next training set.The authors report that this protocol both simplifies curation and yields more generalizable models with more accurate performance estimates.Source: §1 · p. 3
• Unsupervised pretraining on CLEAR’s first unlabeled bucket boosts state-of-the-art supervised CL methods, and a linear classifier on its features surpasses popular methods by a large margin.The authors use this result to motivate continual semi-supervised learning.Source: §1 · p. 3

2.2 Continual Learning Settings

• CLEAR adopts task-based sequential learning with the same 11-way label space across buckets, incremental domain learning, and offline training; boundary-agnostic CL remains a possible adaptation.The paper also reports both locally-iid and streaming evaluations because the former assumes each task’s data is iid.Source: §2.2 · pp. 3–4

2.3 Building Blocks for CLEAR

• YFCC100M supplies a large, metadata-rich source of real-world imagery, and CLIP retrieves subsets relevant to specified visual concepts.The authors note that YFCC100M is cumbersome to download and has noisy user-provided descriptions and hashtags.Source: §2.3 · p. 4

Stream recreation

• The pipeline turns a random 7.8M-image YFCC100M subset into 11 time-indexed buckets and reports 3.3K labeled images across 10 buckets at 99% lower annotation cost.Bucket 0 is reserved for unsupervised pretraining.Source: Figure 2 · p. 5

Concept selection

• The 10 selected concepts span trends and fashion, consumer products, and social events, with background as an additional class.Examples include computers, cameras, buses, cosplay, hockey, and soccer.Source: §3 · p. 5

Stream recreation

• Images are sorted by upload date into 11 buckets; the first 7.8M metadata-listed images form the stream, and bucket 0 is used to pretrain MoCo V2.Buckets 1–10 each have a class-balanced labeled set of 3.3K images across the 10 concepts and background class.Source: §3 · p. 5

Visio-linguistic curation

• For each bucket, CLIP ranks images by query similarity; the top 0.1% are retrieved, refined text queries can improve concept matching, and low-scoring images form the background set.The method processes each of the 10 buckets independently.Source: §3 · p. 5

Crowd-sourced validation (Mturk)

• Crowdsourced validation addresses CLIP’s roughly 20% misclassification rate, with each image checked by at least three workers and inappropriate content removed.The validation process also surfaced inappropriate images in YFCC100M.Source: §3 · pp. 5–6

4 Evaluation Protocols for Continual Learning

• Traditional iid protocols sample each timestamp's training and test sets from the same distribution, unlike real-world deployment across a domain shift.CLEAR's streaming evaluation focuses on the immediate next time period.Source: §4.1 · p. 6, §4 · p. 6, §4.2 · p. 7

4.1 Review of IID Protocols

• The iid protocol summarizes an accuracy matrix with in-domain accuracy, backward transfer, and forward transfer; CLEAR focuses on incremental domain learning with a fixed label space.Incremental class learning, where the output space changes, is left for future work.Source: §4.1 · pp. 6–7

4.2 Our Streaming Protocol

• After training on bucket Si, the streaming protocol evaluates hi on next-period Si+1, then reuses Si+1 to fit hi+1.This mirrors deployment where a model trained on today's data is tested on tomorrow's.Source: §4.2 · p. 7
• Streaming uses each period's data fully for testing when first encountered and for training later, but cannot evaluate historical tasks for learning-without-forgetting.The authors argue that evaluating an outdated task may be less relevant than estimating performance on the task at hand.Source: §4.2 · p. 7
• CLEAR supports both iid and streaming evaluations, with the paper emphasizing streaming because future-domain shifts are a bottleneck in real-world deployment.The streaming perspective is presented as historically underexplored.Source: §4.2 · p. 7

5 Approaches

• YFCC-B0 representations boost all baseline algorithms, and linear models outperform scratch-trained ResNet18 even with naive finetuning and reservoir sampling.Table 1 compares results under both iid and streaming protocols.Source: Table 1 · p. 8
• The supervised baselines include replay, architecture, distillation, and regularization methods, plus naive finetuning; models use scratch-initialized ResNet18 and offline access to each current bucket.Replay methods use a buffer sized to one bucket of training images.Source: §5 · pp. 7–8
• MoCo V2 self-supervised pretraining uses 700K unlabeled images from bucket 0, then linear evaluation classifies the extracted YFCC-B0 features.The reported performance boosts span methods including naive finetuning with simple reservoir sampling.Source: §5 · p. 8
• With a one-bucket replay buffer, α=5.0 or α=0.75*i/k improves all metrics by 1% over naive reservoir sampling.Higher α values favor adding more recent samples to the buffer.Source: Table 2 · p. 9, §5 · p. 9
• Naive finetuning uses reservoir sampling to populate the replay buffer, giving each previously visited sample a uniform inclusion probability.The implementation performs sampling once per incoming bucket, treating its samples as sharing a timestamp.Source: §5 · p. 9

In-domain Acc inflates performance

• In-domain accuracy exceeds next-domain accuracy, and accuracy continues to fall for farther-future test buckets, exposing inflation from same-bucket iid evaluation.The first-timestamp model drops from 90% on Te1 to 82% on Te10.Source: §6 · pp. 9–10, Figure 6 · p. 10
• Streaming evaluation partially addresses the future-bucket drop by training on all prior data rather than the 70% used by classic iid protocols.The accuracy matrix shows deterioration as the test bucket moves farther into the future.Source: §6 · p. 10, Figure 6 · p. 10

Unsupervised pre-training significantly boosts performance

• 91.9% In-domain acc under iid protocol and 91.4% Next-domain acc under streaming protocol are reached by finetuning a linear layer on YFCC-B0 features, versus 77.8% and 78.8% best results without unsupervised pretraining.The supervised-only values use LwF on raw RGB images; YFCC-B0 is unsupervised pretraining on the first bucket.Source: §6 · p. 10

GDumb falls short compared to other baselines

• GDumb is less competitive on CLEAR, plausibly because it trains from scratch at each bucket instead of retaining previously trained models.The authors report that finetuning is always better than training from scratch per timestamp in supplemental analysis.Source: §6 · p. 10

Biased reservoir towards more recent samples is beneficial

• Biased reservoir sampling that favors recent samples with naive finetuning improves on unbiased reservoir sampling and is competitive under both iid and streaming protocols.The analysis also varies the bias parameter in Table 2.Source: §6 · p. 10

7 Conclusion

• CLEAR is presented as the first benchmark for naturally evolving continual image classification, with conclusions emphasizing future-facing evaluation and unlabeled-data pretraining.The paper also plans a public platform and leaderboard to support reproducibility.Source: §7 · p. 10
• The authors hope real-world imagery will help researchers identify biases arising from continual distributions, while noting that the labeled YFCC100M subset was checked for inappropriate images.They frame this as an under-explored issue relevant to real-world ML deployments.Source: §7 · p. 10

Reconstructing the YFCC100M temporal stream

• The pipeline triages YFCC100M metadata and images, then sorts 7.85 million accessible, timestamp-valid images into 11 equal-sized temporal buckets.The metadata is treated as an iid sample of YFCC100M because its original order was shuffled; each bucket contains 713,626 images, and the average bucket spans about 0.8 year.Source: §1 · p. 14, Table 1 · p. 15

Text-Prompt Engineering

• CLIP retrieves candidate images for visual concepts in each bucket, using engineered subcategory prompts to improve retrieval precision and reduce crowd-review rejections.The collection procedure selects the 600 highest-scoring images per category and bucket, plus 60 low-scoring background examples per category.Source: §1 · p. 14, Table 2 · p. 15

Crowdsourced Quality Assurance (QA)

• Crowdsourced quality assurance uses three MTurk workers per image to reject images that mismatch the assigned concept or contain multiple concepts.Workers can select multiple applicable elements or choose “None of the above”; the pipeline keeps images with majority agreement on a single ground-truth label.Source: §1 · pp. 14–16
• The pipeline asks workers to flag sensitive content and removes images when more than two workers mark them sensitive.The process surfaced three sensitive images in YFCC100M; remaining verified images are sampled to compose CLEAR.Source: §1 · p. 16

2 Desiderata in CL Experiment Design

• CLEAR’s sequence of similar temporal buckets, shared output head, absent test-time task labels, limited rehearsal storage, and ten tasks address five continual-learning experiment desiderata.The experiment design is partly inspired by Farquhar et al..Source: §2 · p. 18
• The added streaming desideratum is to train now and test on a future bucket, reflecting the distribution shift that can arise before deployment.The paper notes that annotation, training, tuning, and deployment can take time, so deployed models may encounter a different distribution.Source: §2 · p. 18

3 Evaluation Metrics

• Accuracy averages diagonal and lower-triangular entries of the accuracy matrix, while Backward Transfer averages lower-triangular entries and Forward Transfer averages upper-triangular entries.The metrics summarize different regions of the accuracy matrix R.Source: §3 · p. 19
• Unlike, this paper calculates Backward Transfer without subtracting diagonal terms, making it symmetric with Forward Transfer.The authors distinguish their definition from, which also separates Remembering and Positive Backward Transfer.Source: §3 · p. 19
• Streaming evaluation reports Next-Domain Accuracy and Forward Transfer, whereas iid evaluation reports all defined metrics.Streaming reuses previous test sets as training sets.Source: §3 · p. 19

4 Reservoir Sampling and Its Biased Version

• Bucket-level sampling treats samples within an incoming bucket as sharing one timestamp and waits for the bucket to arrive before sampling.The algorithm takes incoming buckets across N timestamps and produces replay buffers with a fixed size k.Source: §4 · p. 20
• Fixed alpha is a constant non-negative value, while dynamic alpha changes with the number of samples seen so far.The paper describes dynamic alpha using i, the cumulative count of seen samples.Source: §4 · p. 20
• The sampling procedure begins with an empty buffer and carries the prior timestamp’s buffer forward before processing the new bucket.The pseudocode uses a temporary buffer for samples from the current bucket.Source: §4 · p. 20

5 Additional Experiment Results

• 81.1% is the final test accuracy for a ResNet18 trained from scratch on the full labeled CLEAR dataset using a 70-30 train-test split.The authors describe CLEAR as more challenging than MNIST or CIFAR10, likely because it uses real-world imagery.Source: §5 · p. 21
• The shallow-model experiments compare Napping, From Scratch, and Finetuning, with Finetuning initializing each timestamp from the previous timestamp’s learned weights.The section also includes linear and nonlinear two-layer MLP experiments on pre-trained features.Source: §5 · p. 21
• 88.4% to 89.0% In-domain Accuracy and 87.7% to 87.9% Next-domain Accuracy accompany a Backward Transfer drop from 88.9% to 88.3% when From Scratch sampling shifts from uniform to 75% recent samples.With Finetuning, buffering more unseen samples from the current bucket improves all metrics; under limited buffers, biased sampling favoring recent samples outperforms classic iid sampling.Source: §5 · p. 21
• The pre-trained feature comparisons include CLIP, Pretrained-ImageNet, MoCo-ImageNet, BYOL-ImageNet, and MoCo-YFCC-B0.The paper reports linear and nonlinear classification results for these models under both iid and streaming protocols.Source: §5 · pp. 21, 23, Table 6 · p. 23, Table 7 · p. 23
• The iid protocol uses a held-out test set, whereas the streaming protocol does not require one and reuses previous test sets as training data.The streaming table presents linear classification with MoCo-YFCC-B0 features; the iid table presents the corresponding linear classification setting.Source: Table 5 · p. 22, Table 4 · p. 22
• Pretrained-ImageNet performs best across metrics, while biased reservoir sampling with alpha = 0.75 * i/k or 1.0 * i/k consistently outperforms naive reservoir sampling with alpha = 1.0.The authors suggest ImageNet models may benefit from training on images overlapping CLEAR’s period, and that CLIP features may transfer poorly beyond zero-shot classification.Source: §5 · p. 23

6 Experiment Details

• Pre-trained features use resizing to 224 pixels, a 224-pixel center crop, conversion to [0,1], and ImageNet-statistics normalization, with the stated CLIP augmentation exception.Fully supervised ResNet18 training replaces CenterCrop with RandomCrop followed by RandomHorizontalFlip.Source: §6 · p. 25
• Linear classifiers match their input size to the feature dimension, while the two-layer MLP maps inputs to 2048 dimensions before ReLU and its output layer.The input feature size is 1024 for CLIP and 2048 for other pre-trained ResNet50 features.Source: §6 · p. 25
• Shallow models use cross-entropy and SGD with 0.9 momentum, no weight decay, batches of 256, and 100 epochs with a learning-rate decay after epoch 60.The starting learning rate is 1.0 for linear classification except ImageNet-pretrained features, which use 0.1; MLP experiments start at 0.1.Source: §6 · p. 25
• Fully supervised ResNet18 baselines train from scratch for 70 epochs with batches of 64, SGD momentum 0.9, weight decay 1e-5, and learning rate 0.01.The learning rate decays by a factor of 0.1 every 30 epochs.Source: §6 · p. 25
• Continual-learning baselines are run with Avalanche using method-specific settings for LwF, AGEM, SI, EWC, CWR, GDumb, and ER.GDumb and ER use memory equal to one bucket of training data: 3300 samples for streaming and 2310 for iid.Source: §6 · p. 25
• All experiments use a single GeForce RTX 2080 GPU.Source: §6 · p. 25

• The accuracy matrices vary sample storage (unlimited or one bucket), training strategy (from scratch or finetuning), and alpha value under iid evaluation.Source: Appendix: Accuracy Matrices · p. 29
• Linear-classification matrices cover CLIP, Pretrained-ImageNet, MoCo-ImageNet, BYOL-ImageNet, and MoCo-YFCC-B0 features.Source: Appendix: Accuracy Matrices · p. 29, Figure 5 · p. 30, Figure 6 · p. 31
• MLP-classification matrices cover CLIP, Pretrained-ImageNet, MoCo-ImageNet, BYOL-ImageNet, and MoCo-YFCC-B0 features.Source: Appendix: Accuracy Matrices · p. 29, Figure 10 · p. 35, Figure 11 · p. 36