Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks

Andrey Malinin; Neil Band; Ganshin; Alexander; German Chesnokov; Yarin Gal; Mark J. F. Gales; Alexey Noskov; Andrey Ploskonosov; Liudmila Prokhorenkova; Ivan Provilkov; Vatsal Raina; Vyas Raina; Roginskiy; Denis; Mariya Shmatova; Panos Tigas; Boris Yangel

Source-linked AI summary

Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks

Andrey Malinin, Neil Band, Ganshin, Alexander, German Chesnokov, Yarin Gal, Mark J. F. Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Roginskiy, Denis, Mariya Shmatova, Panos Tigas, Boris Yangel

arXiv:2107.07455v3Jul 15, 2021cs.LGcs.AIstat.ML

TL;DR

The paper addresses the limited availability of large-scale, multimodal benchmarks for evaluating robustness and uncertainty under real distributional shift. It introduces the Shifts Dataset from industrial sources across weather prediction, machine translation, and vehicle motion prediction, and reports baseline results showing task-dependent uncertainty behavior and ensemble advantages.

Problem
Existing robustness and uncertainty research has limited evaluation beyond small-scale regression and image-classification tasks, despite practical applications involving varied modalities and structured prediction.
Method
The paper constructs the Shifts Dataset from industrial services, covering tabular weather prediction, machine translation, and self-driving vehicle motion prediction under realistic distributional shifts.
Results
Across the tasks, baseline results show that uncertainty performance depends on the evaluation objective and task, while ensemble methods outperform single-model baselines in reported robustness evaluations.
Takeaways & Limitations
The dataset provides a standardized basis for evaluating uncertainty estimation and robustness on large-scale industrial tasks across multiple modalities.
Takeaways & Limitations
The evaluation assumes that alternative domains and the nature of distributional shift are unknown at training and test time, making it more challenging than settings with shift information.

Abstract

from arXiv · show

There has been significant research done on developing methods for improving robustness to distributional shift and uncertainty estimation. In contrast, only limited work has examined developing standard datasets and benchmarks for assessing these approaches. Additionally, most work on uncertainty estimation and robustness has developed new techniques based on small-scale regression or image classification tasks. However, many tasks of practical interest have different modalities, such as tabular data, audio, text, or sensor data, which offer significant challenges involving regression and discrete or continuous structured prediction. Thus, given the current state of the field, a standardized large-scale dataset of tasks across a range of modalities affected by distributional shifts is necessary. This will enable researchers to meaningfully evaluate the plethora of recently developed uncertainty quantification methods, as well as assessment criteria and state-of-the-art baselines. In this work, we propose the Shifts Dataset for evaluation of uncertainty estimates and robustness to distributional shift. The dataset, which has been collected from industrial sources and services, is composed of three tasks, with each corresponding to a particular data modality: tabular weather prediction, machine translation, and self-driving car (SDC) vehicle motion prediction. All of these data modalities and tasks are affected by real, "in-the-wild" distributional shifts and pose interesting challenges with respect to uncertainty estimation. In this work we provide a description of the dataset and baseline results for all tasks.

1 Introduction

Distributional shift is common in deployed machine-learning systems, yet benchmarks for uncertainty and robustness remain limited, especially beyond small-scale image and regression tasks. The Shifts Dataset addresses this gap with three large-scale industrial tasks spanning tabular weather prediction, machine translation, and vehicle motion prediction.

Motivation: Deployment data often differs from training data, degrading model performance and creating risks in applications such as finance, medicine, and autonomous vehicles.The paper motivates evaluating both robustness to distribution shift and predictive uncertainty.
Motivation: Existing uncertainty and robustness research has focused mainly on small-scale regression and image-classification benchmarks, leaving practical modalities and structured prediction underrepresented.The paper highlights tabular, audio, text, and sensor data as important alternatives.
Contribution: The Shifts Dataset contains industrial-source data for tabular weather prediction, machine translation, and self-driving-car vehicle motion prediction.Each task represents a different data modality and is affected by real-world distributional shift.
Contribution: The paper provides dataset analyses and ensemble-based baseline results for evaluating uncertainty estimates and robustness to distributional shift.The dataset is intended to reflect settings close to in-the-wild deployment conditions.

2 Evaluation Paradigm, Metrics, and Baselines

The paper jointly evaluates robustness and uncertainty without assuming that alternative domains or the nature of distribution shift are known in advance. It uses retention-based metrics and ensemble baselines to assess whether uncertainty identifies errors and shifted examples.

Evaluation Paradigm: Robustness and uncertainty are evaluated jointly because models may perform well on some shifted examples and poorly on underrepresented in-domain examples.This avoids treating all in-domain data as easy and all shifted data as difficult.
Evaluation Paradigm: The evaluation assumes that alternative domains and the nature of distribution shift are unknown at training and test time.This setting is intended to emulate deployments with broad, incompletely observed variation.
Dataset Construction: The dataset is canonically partitioned into realistic but significant shifts that are difficult to handle fully, while annotations support analysis rather than model construction.Weather and motion data can also be repartitioned using provided tools.
Metrics: Error-retention curves measure error as increasingly uncertain predictions are replaced by ground truth, while F1-retention curves assess classification of sufficiently good predictions.Together, these curves assess predictive performance and uncertainty quality.
Baselines: Ensemble methods are used as baselines because they provide uncertainty estimates, improve robustness, and are straightforward to apply.The paper also discusses ensemble diversity as a way to reduce effects of differing spurious correlations.

3 Tabular Weather Prediction

The Weather Prediction benchmark provides large-scale tabular regression and classification tasks with temporal and climate-related shifts. CatBoost ensembles improve predictive performance over single models, while uncertainty measures show task-dependent strengths for error detection and OOD detection.

Dataset: The Weather Prediction dataset contains 10 million 129-column entries for temperature regression and precipitation-class classification.Inputs derive from weather-station measurements and forecast-model features.
Dataset: The canonical partition separates in-domain data from shifted data using time periods and climate types, including Snow conditions for shifted development data.Alternative partitionings are possible, but the paper uses the canonical partition throughout.
Baselines: An ensemble of ten CatBoost models is trained with different random seeds, predicting distributions for regression and class probabilities for classification.The ensemble-based uncertainty approach is applied to both weather tasks.
Results: 0.16°C lower RMSE and 0.12°C lower MAE are achieved by ensembling relative to a single model, while classification accuracy rises about 2% and macro-F1 about 1%.All models perform worse on shifted data than on in-domain data.
Uncertainty Results: Ensembles significantly outperform single models across the considered weather retention measures.Retention curves are reported for evaluation and development data.
Uncertainty Results: Knowledge-uncertainty measures perform best for OOD detection, whereas total uncertainty performs best for error detection; EPKL and negative confidence are the strongest measures in their respective settings.The paper concludes that uncertainty-measure choice depends heavily on the task.

4 Machine Translation

The machine translation task evaluates robustness and uncertainty under atypical language shifts using English–Russian data, multi-hypothesis metrics, and retention-based assessments. Shifted Reddit data substantially reduces translation quality relative to in-domain news, while ensembles outperform individual models.

The task targets translation queries containing slang, profanity, poor grammar, orthography, punctuation, and emojis that challenge systems trained on standard language.
Training uses WMT’20 English–Russian data, while in-domain evaluation uses news and shifted evaluation uses Reddit with atypical language.
eGLEU weights translation hypotheses by confidence, while maxGLEU selects the highest-GLEU hypothesis from the beam; both support retention-curve evaluation.
The baseline is an ensemble of three Transformer-Big models using beam-search decoding with beam width 5 and normalized hypothesis confidence weights.
Nearly 10 BLEU and GLEU points separate in-domain news from shifted Reddit data, while the ensemble outperforms individual models.
Uncertainty and robustness are assessed with error-retention, F1-retention, F1@95%, and ROC-AUC measures based on uncertainty estimates.

5 Vehicle Motion Prediction

The vehicle motion prediction task uses a large multi-condition driving dataset to study robustness and uncertainty under shifts in location and precipitation. Shifted data degrades predictive performance, while model choice creates different trade-offs across trajectory and multimodality metrics.

The dataset contains 600,000 ten-second scenes from six locations, three seasons, three times of day, and four weather conditions.
The task predicts vehicle trajectories for T ∈ (0, 5] from five seconds of preceding context, with in-domain and shifted partitions differing along location or precipitation axes.
Performance is measured with minADE, minFDE, confidence-weighted ADE and FDE, and corrected negative log-likelihood for multimodal predictions.
The baselines are RIP-BC and RIP-DIM, with ensembles generating multiple predictions and aggregating them through model averaging.
Shifted data performs worse than in-domain data across model configurations; RIP-BC leads weighted metrics, whereas RIP-DIM leads minimum-error metrics and cNLL.
RIP-BC produces more informative uncertainty than RIP-DIM on retention metrics despite greater overall cNLL error, while uncertainty scores detect distribution shift poorly.

6 Conclusion

The paper presents the Shifts Dataset as a large, standardized benchmark for evaluating uncertainty estimates and robustness to realistic distributional shifts across industrial tasks and modalities.

The Shifts Dataset covers tabular weather prediction, machine translation, and self-driving car vehicle motion prediction.
The dataset targets large-scale industrial tasks with realistic, curated distributional shifts rather than primarily small-scale classification settings.
The paper provides baseline results using ensemble methods and frames the dataset as a step toward meaningful uncertainty-quantification evaluation.

A Assessment Metrics

The assessment framework evaluates robustness and uncertainty jointly through retention curves and operating-point metrics, while adding F1-retention to address limitations of error-based curves.

Error-retention curves: Retention curves replace predictions with oracle ground truths in decreasing uncertainty order to measure error as the retained prediction fraction changes.
Error-retention curves: R-AUC jointly assesses robustness and uncertainty quality by rewarding lower prediction error and uncertainty rankings that reject incorrect predictions first.
Error-retention curves: Error-retention curves include random and optimal rankings as lower and upper performance bounds for uncertainty quality.
F1-retention curves: F1-retention curves classify predictions as acceptable or unacceptable across uncertainty thresholds and measure F1 over the retained fraction.
F1-retention curves: F1-AUC emphasizes uncertainty-error ranking quality more than error-retention curves when errors have similar magnitudes.
Operating points: At a 95% retention operating point, models can be compared using their error or F1 rather than only an area-under-curve summary.

B Shifts Dataset General Datasheet

The Shifts Dataset was created to evaluate uncertainty quantification and robustness on large-scale industrial tasks spanning multiple modalities, with accompanying challenge, licensing, maintenance, and ethical-use plans.

Motivation: The dataset was created by Yandex Research with Yandex Weather, Yandex.Translate, and Yandex Self-Driving Group teams.
Uses: The Shifts Challenge organized three tracks around the constituent datasets, with data, baseline models, and reproduction code released through a GitHub repository.
Distribution: The Yandex-produced data uses a CC BY NC SA 4.0 license, while the code uses an Apache 2.0 license and research access is intended to be free.
Maintenance: The dataset is actively maintained by Yandex Research, with support from weather, translation, and self-driving teams.
Societal consequences and ethical use: The datasheet encourages using the dataset to improve reliability and safety while developing compute- and memory-efficient methods.

C.1 Dataset Description

The Weather Prediction dataset combines meteorological observations and forecast features with temperature regression and precipitation or cloudiness classification targets, then partitions data by time and climate to create matched and shifted splits.

Composition: Weather samples pair meteorological features with air temperature, precipitation class, and cloudiness class targets at specific locations and times.
Climate annotation: The dataset uses Koppen climate types to annotate locations and analyze robustness to climate shifts without using climate type as a training feature.
Partitioning: The full Weather Prediction dataset contains 10M records distributed across one year and five climate types.
Partitioning: The canonical partition provides in-domain and out-of-domain development and evaluation data using time and climate differences from training.
Observed shifts: Out-of-domain sets differ from in-domain sets in temperature distributions and sample locations, reflecting seasonal and climate shifts.

C.5 Additional experiments

Additional Weather experiments compare ensemble baselines and uncertainty measures across predictive, robustness, and uncertainty-quality evaluations. Combined CatBoost and FT-Transformer ensembles perform best jointly, while the strongest uncertainty measure depends on the task.

Additional baselines: Combining CatBoost and FT-Transformer ensembles yields the best joint robustness and uncertainty performance among the additional baselines.Monte-Carlo dropout ensembles are competitive with CatBoost ensembles for joint assessment, despite CatBoost’s better raw predictive quality.
Uncertainty measures: The CatBoost–FT-Transformer combination achieves the best joint uncertainty and robustness performance, with total variance the best uncertainty measure.These results are reported for the uncertainty measures evaluated in Table 11.
Task dependence: The best model and uncertainty measure depend on the task: anomaly detection favors a CatBoost ensemble with knowledge-uncertainty measures, while regression and classification use total uncertainty measures.The passage identifies variance and confidence as total-uncertainty measures for regression and classification, respectively.
Additional experiments: The experiments extend Weather evaluation beyond the canonical partition to examine whether time or climate shifts cause larger performance drops.These additional experiments use CatBoost for both regression and classification tasks.
Retention analysis: Retention curves are reported for CatBoost and FT-Transformer on both development and evaluation data in the canonical Weather dataset.The corresponding figures present retention behavior for each model across the two splits.

D.1 Dataset Description

The Shifts Machine Translation dataset pairs English and Russian text across in-domain and Reddit-based shifted partitions. It evaluates translation quality and uncertainty-aware retention using corpus- and sentence-level metrics, including GLEU-derived measures.

Composition and partitions: The dataset contains training, development, and evaluation English–Russian sentence pairs, using WMT’20 for training and separate in-domain and shifted sources for evaluation.In-domain data comes from Newstest’19 and GlobalVoices, while shifted data comes from Reddit.
Shifted data: Reddit shifted partitions contain atypical language such as slang, acronyms, poor punctuation, spelling errors, profanity, and poor grammar.These inputs are described as representative of challenges faced by machine translation services.
Annotation process: Yandex translators produced Russian targets for Reddit data after correcting source-side orthographic, grammatical, and punctuation mistakes.The correction step was used only to obtain target-side Russian translations.
Anomaly annotation: Yandex.Translate annotators labeled Reddit development and evaluation data with seven non-exclusive anomaly flags, whose occurrence and co-occurrence are analyzed.Figure 14 presents the anomaly occurrence and co-occurrence analysis.
Metrics: Evaluation uses corpus-level BLEU and sentence-level GLEU, with GLEU preferred for per-sample assessment because it is stable at the sentence level.GLEU is also reported to correlate strongly with BLEU at the corpus level.
Uncertainty-aware metrics: Because translation is multimodal, the evaluation considers expected GLEU across confidence-weighted hypotheses and maximum GLEU as an upper-bound measure.The eGLEU-based error metric is used to compute retention-curve area and valid-prediction F1.

E.1 Dataset Description

The Vehicle Motion Prediction dataset is a large, multi-condition collection of driving scenes with trajectory and uncertainty-prediction tasks. Its partitions vary location and precipitation to evaluate robustness and uncertainty under distributional shift.

Dataset composition: The dataset contains 600,000 ten-second scenes spanning six locations, three seasons, three times of day, and four weather conditions.Each scene contains five seconds of context followed by five seconds of prediction targets.
Prediction task: The task predicts vehicle trajectories during T ∈ (0, 5] from scene information observed during T ∈ [−5, 0].Scenes include dynamic-object states and an HD map, with state data sampled at 5Hz.
Prediction requests: Prediction requests identify visible vehicles requiring forecasts, and evaluation is restricted to requests with fully observed ground-truth trajectories.Requests also carry non-mutually exclusive maneuver tags covering ten maneuver types.
Distributional shift: In-domain partitions match training-set location and precipitation, whereas shifted partitions differ along one or more of those axes.Development and evaluation partitions support robustness and uncertainty-estimation assessment.
Trajectory confidence: Models provide per-trajectory confidence scores that order the plausibility of multiple predicted trajectories for each input.The confidence scores are non-negative and sum to one, forming a probability distribution.
Request uncertainty: The task also evaluates scalar uncertainty for each prediction request, intended to identify unfamiliar or high-risk scene contexts in deployment.The dataset describes both per-trajectory and per-request uncertainty levels.

E.3 Performance Metrics

The section defines motion-prediction metrics for trajectory accuracy, confidence quality, uncertainty, and robustness. It also describes RIP’s trajectory-generation and scoring pipeline and explains why likelihood-based metrics better preserve multimodal predictions than additive displacement losses.

Standard Performance Metrics: ADE and FDE measure trajectory quality across all timesteps and at the final timestep, respectively.Stochastic models evaluate sampled trajectories using aggregation operators such as minimum or mean.
Confidence-Aware Metrics: RIP evaluates per-trajectory confidence by selecting the highest-scoring trajectory and by weighting trajectories according to their confidence scores.The framework also aggregates the top trajectories’ confidence scores into a per-request uncertainty score.
Multimodal Prediction Metrics: Additive displacement metrics can cause mode collapse, producing a weighted geometric median instead of distinct plausible trajectories.At a T-junction, this can yield a straight trajectory even when the true alternatives go left or right.
Multimodal Prediction Metrics: Likelihood-based metrics avoid this failure by allowing separate weighted distributions for multiple trajectory modes.For a bimodal Gaussian example, weightedADE favors the collapsed median, whereas cNLL favors predicting both distinct modes.
Robust Imitative Planning: RIP generates trajectories from ensemble members, scores them with model likelihoods, retains the top D trajectories, and reports normalized confidence scores plus an overall uncertainty score.The implementation combines per-model trajectory scores through trajectory-level and prediction-request-level aggregation operators.
Experimental Setup: The evaluation compares behavioral-cloning and deep-imitative backbones with Deep Ensembles and Dropout Ensembles across in-distribution, shifted, and full datasets.Teacher-forced conditioning was replaced with predicted values in one implementation choice, improving performance across datasets and metrics.