PaperlayerView exact revision on arXiv

Source-linked AI summary

Rank-N-Contrast: Learning Continuous Representations for Regression

Kaiwen Zha, Peng Cao, Jeany Son, Yuzhe Yang, Dina Katabi

arXiv:2210.01189v2cs.LGcs.AIcs.CV

TL;DR

Deep regression methods can learn fragmented representations that miss continuous relationships, and existing representation-learning methods overlook continuity in regression. Rank-N-Contrast ranks samples by their labels and contrasts them by relative rankings; experiments across five real-world datasets verify its superior performance, robustness, and efficiency.

  • Problem

    Deep regression methods often produce fragmented representations that miss continuous relationships, while representation-learning methods overlook continuity in regression.

    Source: §1 · p. 2

  • Method

    Rank-N-Contrast learns continuous regression representations by ranking samples according to their labels and contrasting them based on relative rankings.

    Source: §7 · p. 10

  • Results

    Experiments on five real-world datasets verify RNC's superior performance, robustness, and efficiency for learning continuous targets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments highlight better data efficiency, robustness to corruptions and spurious targets, and generalization to unseen targets.

    Source: §7 · p. 10

    RNC cannot discern spurious or incidental correlations between dataset inputs and targets.

    Source: §6 · p. 10

Abstract

from arXiv · show

Deep regression models typically learn in an end-to-end fashion without explicitly emphasizing a regression-aware representation. Consequently, the learned representations exhibit fragmentation and fail to capture the continuous nature of sample orders, inducing suboptimal results across a wide range of regression tasks. To fill the gap, we propose Rank-N-Contrast (RNC), a framework that learns continuous representations for regression by contrasting samples against each other based on their rankings in the target space. We demonstrate, theoretically and empirically, that RNC guarantees the desired order of learned representations in accordance with the target orders, enjoying not only better performance but also significantly improved robustness, efficiency, and generalization. Extensive experiments using five real-world regression datasets that span computer vision, human-computer interaction, and healthcare verify that RNC achieves state-of-the-art performance, highlighting its intriguing properties including better data efficiency, robustness to spurious targets and data corruptions, and generalization to distribution shifts. Code is available at: https://github.com/kaiwenzha/Rank-N-Contrast.

1 Introduction

RNC addresses the failure of prediction-focused regression methods and existing representation-learning schemes to capture continuous target order in learned features. It ranks samples by target distances, contrasts them accordingly, and is evaluated across five real-world datasets.

1 Introduction

  • Regression methods commonly predict targets directly with L1 or L2 losses, while some instead cast regression as classification.The paper situates these approaches against the need for representations that capture continuity.Source: §1 · pp. 1–2
  • Prediction-focused methods often learn fragmented representations that capture spurious factors such as webcam identity instead of continuous target relationships.SupCon representations also fail to capture the continuous order between samples on SkyFinder.Source: §1 · p. 2
  • RNC learns regression-aware representations by ranking samples according to their target values and contrasting them by relative rankings.The framework then uses the representation to predict continuous targets and can pair it with any regression method.Source: §1 · p. 2
  • Optimizing LRNC theoretically orders learned features according to continuous labels, supporting downstream regression.The introduction also reports that RNC learns representations capturing intrinsic ordered relationships between samples.Source: §1 · p. 2
  • Experiments on five real-world datasets across computer vision, human-computer interaction, and healthcare report superior performance, robustness, and efficiency for RNC.Further analyses report data efficiency, robustness to spurious targets and corruptions, and generalization to unseen targets.Source: §1 · p. 2

2 Related Work

Prior regression work includes direct prediction, classification and ordinal formulations, and embedding regularization. RNC instead emphasizes continuity in feature representations and is described as compatible with existing regression schemes and complementary to regression augmentation.

2 Related Work

  • Regression methods include direct target prediction, bin-based classification, ordinal classification, and embedding regularization for uncertainty, entropy, or imbalance.RNC differs by emphasizing continuity in feature space while remaining compatible with prior regression schemes.Source: §2 · p. 2
  • Figure 2 illustrates context-dependent positive and negative pairs: for the 20-year-old anchor, a 30-year-old positive makes both the baby and 70-year-old samples negatives.When the baby is paired positively with the anchor, only the 70-year-old sample is a negative.Source: Figure 2 · p. 3
  • C-Mixup adjusts mixed-pair sampling probabilities using target similarities, and the paper describes RNC as orthogonal and complementary to it.C-Mixup adapts the original mixup method.Source: §2 · p. 3
  • Contrastive learning and SupCon have been widely studied for classification, while a few methods adapt SupCon to ordered labels in specific applications.RNC uses label continuity to design a representation-learning scheme for generic regression tasks.Source: §2 · p. 3

3 Our Approach: Rank-N-Contrast (RNC)

RNC trains representations by ranking samples according to their label distances from an anchor and contrasting them through a ranking-based likelihood. The resulting feature similarities more closely track label similarities than those learned with vanilla L1 loss.

3 Our Approach: Rank-N-Contrast (RNC)

  • RNC forms a two-view augmented batch, learns embeddings with an encoder, freezes that encoder, and trains a predictor on top using a regression loss.The setup uses an encoder and predictor to map inputs to continuous targets.Source: §3 · p. 3
  • LRNC ranks samples by target distance from an anchor and contrasts them according to their relative rankings.For each anchor, nearer samples are favored over samples at greater label distances, with the process repeated across ranks and anchors.Source: §3 · pp. 3–4
  • The likelihood for a sample uses a denominator over samples of higher rank, and maximizing it raises the sample's probability of emerging at the top of that set.Similarity between embeddings and a temperature parameter determine the likelihood.Source: §3 · pp. 3–4
  • The per-sample RNC loss averages negative log-likelihoods across other batch samples, aligning feature-embedding orders with label-space orders for each anchor.The loss is enumerated over all 2N samples as anchors.Source: §3 · p. 4
  • On temperature regression, RNC feature similarities show higher Spearman and Kendall rank correlations with label similarities than vanilla L1 features.Figure 3 compares feature-similarity matrices for models trained with vanilla L1 and RNC, with data points sorted by ground-truth labels.Source: §3 · p. 4, Figure 3 · p. 4, Table 1
  • LRNC treats any other batch sample as a possible positive, with negatives determined by whether their label distance from the anchor exceeds the positive's.This differs from classification contrastive learning, where positives share a class or input and other samples are negatives.Source: §3 · p. 4

4 Theoretical Analysis

The analysis shows that sufficiently low Rank-N-Contrast loss yields embeddings ordered by label distances, and extends this guarantee from batches to the full feature space. It also connects ordered embeddings to improved generalization bounds.

4 Theoretical Analysis

  • δ-ordered embeddings keep similarities for equal label distances within δ, while ordered label distances are separated by at least 1/δ.The similarity gap between samples with different label distances is larger than that for equal label distances.Source: §4 · p. 5
  • L⋆ is a lower bound for LRNC, and feature embeddings can approach it arbitrarily closely.The theorems establish the lower bound and its tightness; sufficiently near the bound, embeddings are δ-ordered.Source: §4 · p. 5
  • For every 0 < δ < 1, some ε > 0 makes LRNC < L⋆ + ε sufficient for δ-ordered embeddings.This is the main theorem linking optimization of LRNC to the ordering property.Source: §4 · p. 5
  • Optimizing expected loss over random batches makes the entire feature space δ-ordered with high probability when that expectation is sufficiently low.The argument uses the fact that any triplet appears in some batch and applies Markov’s inequality to batch loss.Source: §4 · p. 5
  • A δ-ordered embedding yields a better generalization bound, while experiments also report robustness to corruptions, reduced training data, and unseen targets.The analysis connects ordered embeddings to reduced regressor complexity and better generalization from training to testing.Source: §4 · p. 5

5 Experiments

Experiments evaluate RNC on five regression datasets spanning computer vision, human-computer interaction, and healthcare, with benchmark comparisons and analyses of robustness, data efficiency, and generalization.

5 Experiments

  • The five datasets cover age, brain-age, gaze direction, and temperature prediction; evaluation reports MAE for age, brain-age, and temperature, angular error for gaze, and R2.ResNet-18 is the main backbone for four image datasets, while TUAB uses a 24-layer 1D ResNet.Source: §5 · p. 6

5.1 Main Results

  • RNC surpasses state-of-the-art methods across the four benchmark datasets; adding its representation reduces prediction error by 5.8%, 9.3%, 11.7%, and 7.0% on average on AgeDB, TUAB, MPIIFaceGaze, and SkyFinder, respectively.RNC is best on both reported metrics across these datasets.Source: Table 3 · p. 7, §5.1 · p. 7
  • RNC outperforms representation-learning and regression-learning competitors across datasets, while some representation methods underperform vanilla L1; the reported gains stem from LRNC rather than pretraining.The comparison includes SIMCLR, DINO, and SUPCON, as well as feature-regularization and dataset-specific methods.Source: §5.1 · p. 7

5.2 Analysis

  • On AgeDB corruptions and subsampled IMDB-WIKI, RNC(L1) shows less performance degradation than L1 as corruption severity rises or training samples decrease.Corruption tests average results over 19 types and varying severity levels.Source: §5.2 · p. 8, Figure 4 · p. 8, Figure 5 · p. 8
  • RNC(L1) transfers better than L1 in both transfer directions, captures temperature rather than webcam clusters under spurious targets, and improves more on unseen than seen targets without sacrificing seen-target performance.The transfer comparisons use linear probing and fine-tuning; unseen-target tests use two IMDB-WIKI training distributions with missing target ranges.Source: §5.2 · pp. 8–9, Table 4 · p. 8, Figure 6 · p. 8

5.3 Ablation Studies

  • Larger positive sets improve AgeDB results, all-positive batches fully guarantee feature order, negative L1/L2 similarity beats cosine, and freezing the encoder outperforms fine-tuning.The training-scheme comparison attributes fine-tuning's weaker result to L1 updates disrupting the LRNC-learned embedding order.Source: §5.3 · p. 9

5.4 Further Discussions

  • RNC works with or without augmentation because it contrasts label distances; its training time is comparable to standard contrastive methods, and gains arise from LRNC rather than the two-stage scheme.Training a predictor on representations from competing regression baselines does not improve performance and can be detrimental.Source: §5.4 · p. 10

6 Broader Impacts and Limitations

The section flags ethical risks from using regression on sensitive personal attributes, especially when learned features are opaque or minority groups are underrepresented. It also identifies limits in handling spurious correlations, subgroup evaluation, and unlabeled learning.

6 Broader Impacts and Limitations

  • RNC’s learned features are opaque, and unbalanced representation of minority groups provides no assurance of reliable performance for those groups.The section also warns that sensitive-attribute use could reinforce bias or enable discrimination; these concerns apply to deep regression models generally.Source: §6 · p. 10
  • RNC cannot distinguish spurious input-target correlations, which could contribute to discriminatory or unjust treatment when predicting personal attributes.The authors call for further work on ethical regression learning.Source: §6 · p. 10
  • The evaluation focuses on overall regression accuracy, leaving performance disparities across demographic subgroups, including minority groups, unassessed.The authors identify subgroup analysis as future work.Source: §6 · p. 10
  • RNC’s ranking-based representation learning requires labels; adapting it to unlabeled data would require modifications reserved for future work.Source: §6 · p. 10

7 Conclusion

The conclusion presents RNC as a ranking-based framework for continuous regression representations and reports benefits in performance, data efficiency, robustness, and generalization. The accompanying proof passages define a higher-rank sample set and construct embeddings approaching the loss optimum.

7 Conclusion

  • RNC ranks samples by labels and contrasts them using relative rankings; experiments report superior performance, data efficiency, robustness to corruptions and spurious targets, and generalization to unseen targets.Source: §7 · p. 10

A.1 Proof of Theorem 1

  • A.1 defines S_i,j as samples whose label distance from the anchor is at least that of v_j, then uses this set in the proof’s algebra.Source: §A.1 · p. 14

A.2 Proof of Theorem 2

  • A.2 constructs embeddings with L_RNC < L* + ε and relates their similarities to whether samples have equal or greater label distances from an anchor.Source: §A.2 · pp. 14–15

A.3 Proof of Theorem 3

The proof links a sufficiently low RNC loss to δ-controlled similarity relationships: equal label distances yield near-equal similarities, while unequal distances preserve an ordered margin.

A.3 Proof of Theorem 3

  • When L_RNC < L* + ε, samples at equal label distances have feature similarities differing by less than δ.Source: §A.3 · pp. 15–16
  • The proof bounds η_i,m by δ using a softplus inequality before deriving the equal-distance and ordered-distance similarity conditions.Source: §A.3 · p. 16
  • For unequal label distances under the loss condition, a closer-label pair has similarity greater than a farther-label pair by more than δ.Source: §A.3 · pp. 16–17

B Additional Theoretical Analysis

The analysis argues that δ-ordered embeddings restrict the candidate regression hypotheses without excluding the optimal hypothesis. This restriction reduces the stated Rademacher-complexity generalization bound.

B Additional Theoretical Analysis

C Dataset Details

The evaluation covers five real-world regression datasets spanning face-based age and gaze estimation, EEG brain-age estimation, and webcam temperature prediction. Data augmentation is tailored to each dataset’s input modality.

C Dataset Details

  • AgeDB predicts age from 16,488 celebrity face images, with ages 0–101 and 12,208 training, 2,140 validation, and 2,140 test images [44].Source: C Dataset Details · p. 18
  • TUAB estimates brain age from 1,385 non-pathological, 21-channel EEG signals sampled at 200 Hz, split into 1,246 training and 139 test subjects.Source: C Dataset Details · p. 18
  • MPIIFaceGaze estimates two-dimensional pitch and yaw from 45,000 images, split by participant into 33,000 training, 6,000 validation, and 6,000 test images [51] [52].Source: C Dataset Details · p. 18
  • SkyFinder predicts outdoor temperature from 35,417 images captured by 44 cameras, with temperatures from −20 °C to 49 °C [7].Source: C Dataset Details · p. 18
  • IMDB-WIKI supplies celebrity face images for testing reduced-data training, transfer learning, and generalization to unseen age targets; its validation and test sets each contain 11,022 images [44].Source: C Dataset Details · p. 18

D Details of Data Augmentation

E.1 End-to-End Regression Methods

The experiments compare RNC with end-to-end regression, representation-learning, and specialized regression baselines using dataset-specific training settings. Additional analyses report compatible results with ResNet-50 and standard deviations aligned with the main-paper results.

E.1 End-to-End Regression Methods

  • End-to-end baselines include direct target prediction with L1, MSE, or Huber losses, binned-distribution methods DEX and DLDL-V2, and ordered-threshold methods OR and CORN.Source: §E.1 · p. 19

E.2 State-of-the-art Representation Learning Methods

  • Representation-learning baselines are SIMCLR and DINO, alongside SUPCON adapted to regression by treating target-range bins as classes.Source: §E.2 · pp. 19–20

E.3 State-of-the-art Regression Learning Methods

  • Specialized regression baselines include LDS+FDS, L2CS-NET, LDE, RANKSIM, and ORDINAL ENTROPY, with dataset-specific state-of-the-art methods noted for AgeDB, MPIIFaceGaze, and SkyFinder.Source: §E.3 · p. 20

F Details of Experiment Settings

  • Training uses SGD, cosine learning-rate annealing, batch size 256, grid-searched learning rates and weight decay, and 400 encoder or one-stage epochs followed by 100 predictor epochs for two-stage methods.Source: F Details of Experiment Settings · p. 20

G.1 Impact of Model Architectures

  • ResNet-50 results on the visual datasets are consistent with ResNet-18 results, indicating compatibility with different encoder architectures.Source: §G.1 · p. 20, Table 8 · p. 20

G.2 Standard Deviations of Results

  • Across five random seeds, the best-result prediction errors and standard deviations align with the results reported in the main paper.Source: §G.2 · p. 21, Table 9 · p. 21

G.3 Impact of Data Augmentation

Removing augmentation reduces performance for both L1 and RNC(L1), but RNC(L1) outperforms L1 with or without it. Further analyses find that two-stage training does not benefit end-to-end baselines, while RNC’s training efficiency is comparable to SUPCON.

G.3 Impact of Data Augmentation

  • RNC(L1) outperforms L1 both with and without augmentation, although removing augmentation decreases performance for both methods.Source: §G.3 · p. 21, Table 10 · p. 21
  • Unlike RNC’s label-based contrast, unsupervised contrastive learning uses augmented views to define its pretext task, and removing augmentation causes model collapse.Source: §G.3 · p. 21
  • RNC does not require augmentation to create positive views because LRNC contrasts samples by label distance rather than identity; augmentation instead supports generalization, with removal causing a similar performance drop for RNC(L1) and L1.Source: §G.3 · p. 21

G.4 Two-Stage Training Scheme for End-to-End Regression Methods

  • Adding a second-stage L1 predictor does not improve end-to-end regression methods and can harm performance, supporting the conclusion that RNC’s benefit comes from LRNC rather than the training scheme.Source: §G.4 · p. 22, Table 11 · p. 22

G.5 Training Efficiency

  • RNC’s average per-epoch wall-clock training efficiency is comparable to SUPCON across the four evaluated datasets.Source: §G.5 · p. 22, Table 12 · p. 22

Related papers

Loading 2210.01189v2…