PaperlayerView exact revision on arXiv

Source-linked AI summary

A Stationary (and Therefore Compatible) Representation is All You Need

Niccolò Biondi, Federico Pernici, Simone Ricci, Alberto Del Bimbo

arXiv:2606.12488v1cs.LG

TL;DR

Compatible representation learning seeks model updates that preserve interchangeability without reprocessing gallery features. This paper proves that stationary d-Simplex representations imply compatibility and proposes a cross-entropy–contrastive training approach that captures higher-order dependencies, achieving uninterrupted retrieval and state-of-the-art performance across sequential fine-tuning experiments.

  • Problem

    Compatible representation learning lacks sufficient theoretical analysis and must preserve gallery-feature usability when models are updated.

  • Method

    The paper proves compatibility for stationary d-Simplex representations and combines cross-entropy with contrastive loss to capture higher-order dependencies during sequential fine-tuning.

  • Results

    Stationary d-Simplex representations enable uninterrupted retrieval while leveraging improved-model replacements and achieve state-of-the-art performance, particularly after multiple updates.

  • Takeaways & Limitations

    Stationarity provides a theoretically grounded and practically effective route to compatible representations for sequentially updated visual-search models.

  • Takeaways & Limitations

    The theoretical analysis relies on the Unconstrained Feature Model and Linear Probe Model assumptions, including an expressive backbone and symmetric d-Simplex geometry.

Abstract

from arXiv · show

Learning compatible representations aims to learn feature representations that can be used interchangeably over time whenever a model undergoes updates. In this paper, we demonstrate that stationary representations learned by d-Simplex fixed classifiers imply compatibility as in its formal definition. This result establishes a foundation for future works and can be directly exploited in practical learning scenarios. We address the challenge of learning compatibility using $d$-Simplex fixed classifiers when the model is sequentially fine-tuned. Learning according to a d-Simplex fixed classifier with the cross-entropy loss aligns feature distributions at the first-order statistics. Consequently, it may not fully capture higher-order dependencies in the representation between model updates. To address this issue, we demonstrate that training the model using a $d$-Simplex fixed classifier through a convex combination of the cross-entropy loss and a contrastive loss not only captures higher-order dependencies, but is also equivalent to learning with the cross-entropy under the compatibility constraints. We confirm our findings with extensive experiments also considering a new scenario where a pre-trained model is sequentially fine-tuned and occasionally replaced with an improved model. We show that stationary representations enable uninterrupted retrieval services (without reprocessing gallery images) while improving performance during model updates and replacements, achieving state-of-the-art. Code at https://github.com/miccunifi/iamcl2r.

1 INTRODUCTION

The paper establishes theoretically that stationary representations learned with d-Simplex fixed classifiers imply compatibility, and develops a sequential fine-tuning approach combining cross-entropy and contrastive losses. Experiments validate the method in open-set recognition and a model-replacement scenario, achieving state-of-the-art compatible representation learning.

  • Theoretical foundation: Stationary representations learned with d-Simplex fixed classifiers satisfy the compatibility inequalities under the Unconstrained Feature Model.The result bridges empirical compatibility outcomes with theoretical analysis.
  • Sequential fine-tuning: Sequential fine-tuning can align old and new models through first-order statistics, motivating a method that addresses dependencies beyond those statistics.The paper targets compatible representation learning when new representations are learned from new data.
  • Sequential fine-tuning: A convex combination of cross-entropy and contrastive loss with a d-Simplex fixed classifier is equivalent to learning under compatibility constraints.This formulation is presented as the practical response to limitations of first-order alignment.
  • Experimental validation: The method achieves state-of-the-art performance in open-set image recognition and is evaluated when a pre-trained model is sequentially fine-tuned and occasionally replaced.The latter is introduced as a novel evaluation scenario.
  • Theoretical foundation: The paper extends prior stationary-representation results by proving that stationarity implies compatibility without approximation and verifying both compatibility inequalities using cosine distance.The extension explicitly leverages the hyperspherical representation space.

2 RELATED WORK

Prior compatible representation learning studied retraining or sequential fine-tuning through representation comparison, feature mapping, and distillation. This work builds on theoretical d-Simplex analyses and Simplex-ETF fixed geometry to establish compatibility through stationarity.

  • Compatible Representation Learning: Early methods retrained models with old and new data, using old representations or classifiers to regularize the learned representation.BCT directly compares new and old representations and uses the old classifier as a regularizer; later works extended this approach.
  • Compatible Representation Learning: Sequential fine-tuning methods include FAN’s feature-adaptation mapping and CVS’s output-distillation composition for aligning updated and old models.FAN maps old features into the updated model’s representation space, while CVS applies composed distillation functions to network outputs.
  • Compatible Representation Learning: Earlier d-Simplex theory showed optimal approximation of compatibility inequalities, satisfying only the first inequality while maximizing prototype pairwise distances.The present work instead claims stationarity from d-Simplex fixed classifiers satisfies both compatibility inequalities without approximation.
  • Neural Collapse: Neural Collapse describes within-class feature collapse to prototypes whose configuration forms a Simplex-ETF with equal pairwise angles.This phenomenon occurs in the terminal phase of training of deep neural networks.
  • Neural Collapse: Simplex-ETF fixed geometry was also configured before training to preserve future class regions, supporting this work’s UFM/LPM-based proof assumptions.The assumptions require an expressive backbone for independent feature analysis and the symmetry of the d-Simplex ETF fixed classifier.

3 STATIONARITY IMPLIES COMPATIBILITY

Theorem 1 establishes that representations learned with a d-Simplex fixed classifier are compatible on average when class hyperspherical caps shrink after updates. This provides a theoretical basis for avoiding gallery re-indexing during model updates.

  • Compatibility: Compatibility supports retrieval with updated query features without costly re-indexing of the gallery feature set.Search retrieves nearest gallery features to a query using a distance function d(·, ·).
  • d-Simplex fixed classifier: d-Simplex classifiers use K = d + 1 equidistant, maximally separated class prototypes, producing stationary learned features.Pre-allocating classes reserves feature-space regions for future classes.
  • Theorem 1: Theorem 1 links feature stationarity from a d-Simplex fixed classifier to compatibility between models learned at different tasks.The models use K pre-allocated classes and prototypes w1, w2, …, wK, with Kk < Kt < K.
  • Theorem 1: Under shrinking hyperspherical caps after updates, the earlier and later models satisfy the compatibility inequalities on average.The result applies to representation models ϕk and ϕt learned at the k-th and t-th tasks.

4 STATIONARITY AND COMPATIBILITY IN SEQUENTIAL FINE-TUNING

In sequential fine-tuning, d-Simplex cross-entropy aligns only first-order feature statistics across updates, while the HOC loss adds contrastive mutual-information modeling to capture higher-order dependencies and preserve compatibility. Proposition 1 establishes that HOC training is equivalent to cross-entropy under compatibility constraints.

  • Limitation of cross-entropy: Sequential d-Simplex cross-entropy aligns each class’s features to first-order distribution statistics preserved across model updates.The representation at task t is evaluated against the previously learned model at task t−1.
  • Higher-Order Compatibility loss: HOC loss combines d-Simplex cross-entropy with an InfoNCE contrastive loss weighted by λ and 1 − λ.LHOC(ϕt) = λ LSCE(ϕt) + (1 −λ) LiNCE(ϕt, ϕt−1), with λ ∈ [0, 1].
  • Higher-Order Compatibility loss: The contrastive term approximates mutual information between successive representations and captures higher-order dependencies across model updates.It is described as an approximation of the KL divergence between the joint distribution and product of marginals.
  • Compatibility guarantee: Proposition 1 shows that HOC training preserves compatibility while accounting for higher-order dependencies between sequentially fine-tuned representations.Training ϕt with HOC is equivalent to cross-entropy training under compatibility constraints.

5 EXPERIMENTAL RESULTS

Experiments across sequential fine-tuning scenarios show that stationary d-Simplex representations provide strong compatibility and accuracy, with d-Simplex-HOC generally delivering the best trade-off. They also enable model replacements to improve accuracy without sacrificing compatibility.

  • Experimental scenarios: The experiments evaluate compatibility in CL2R and model-replacement scenarios using sequentially fine-tuned visual-search models.The CL2R setting begins with a model trained from scratch, while the IAM-CL2R setting occasionally replaces it with an improved model.
  • CL2R results: Stationary d-Simplex training is crucial for compatibility, while most methods decline as tasks increase, with a significant drop at 31 tasks.The exception is d-Simplex-FD and d-Simplex-HOC, which avoid the consistent decline observed for other approaches.
  • CL2R results: d-Simplex-HOC achieves the strongest compatibility through AC and ACA while maintaining high AA, especially in the 31-task experiments.Its high ACA indicates the best trade-off between compatibility and accuracy, attributed to modeling higher-order dependencies.
  • Model-replacement results: Both d-Simplex-HOC and d-Simplex-FD benefit from improved model replacements, whereas other methods lose average accuracy after replacement.d-Simplex-HOC alone maintains high compatibility while leveraging additional knowledge, and it achieves the highest performance in the larger, more complex scenario.
  • Model-replacement results: Only d-Simplex-FD and d-Simplex-HOC maintain consistent compatibility with good accuracy across 31-task updates and architecture replacements.The d-Simplex representation acts as a common interface when replacement architectures produce feature vectors of different sizes.

6 ABLATION STUDIES

The ablations show that d-Simplex-HOC is robust to the number of pre-allocated classes, while its performance depends on hyper-parameters, learning rate, replay, and fine-tuning data. The results identify λ = 0.1, ρ = 5, and 300 images per class as effective settings, while model replacement remains beneficial without rehearsal.

  • Number of Pre-allocated Classes: d-Simplex-HOC compatibility and accuracy are not significantly influenced by the number of pre-allocated classes.This was evaluated over 16 CIFAR100/10 tasks in the CL2R scenario using 100 to 10,000 pre-allocated classes.
  • Hyper-parameters Values: λ = 0.1 achieves the highest ACA and the best trade-off between compatibility and accuracy, while ρ = 5 provides the best overall performance.The hyper-parameters are used in Eq. 2 and Eq. 5, respectively.
  • Learning Rate: High learning rates lower AA, AC, and ACA by strongly modifying representations, whereas low learning rates improve compatibility at the cost of accuracy.The experiments of Sec. 5.3 use a learning rate equal to 0.001.
  • Number of Images in the Experience Replay Buffer: d-Simplex-HOC exploits improvements from model replacement even in the rehearsal-free case.This result concerns IAM-CL2R with two model replacements and 31 CIFAR100R/10 tasks.
  • Number of Images per Class for Fine-tuning: AC and ACA decrease with fewer fine-tuning images per class, while AA remains relatively high; experiments use 300 images per CIFAR100R class.The evaluation varies from 500 to 5 images per class in IAM-CL2R with two model replacements.

7 CONCLUSIONS

The paper shows that stationary representations learned by d-Simplex fixed classifiers satisfy compatibility inequalities in expectation. A convex combination of cross-entropy and contrastive losses captures higher-order dependencies during sequential fine-tuning, while experiments demonstrate uninterrupted retrieval and state-of-the-art performance.

  • Theoretical findings: Stationary representations learned by d-Simplex fixed classifiers satisfy compatibility inequalities in expectation.This establishes a theoretical connection between stationarity and compatible representation learning.
  • Training method: A convex combination of cross-entropy and contrastive losses captures higher-order dependencies between representations during sequential fine-tuning.The method is trained with a d-Simplex fixed classifier.
  • Experimental validation: In sequential-fine-tuning visual-search experiments, stationary representations enabled uninterrupted retrieval while leveraging improvements from occasional model replacements.The replacement scenario was newly considered in the experiments, and the passage contrasts this capability with all other methods.
  • Experimental validation: The proposed method achieved state-of-the-art performance in all reported experiments.The conclusion specifically highlights the method’s performance across the experimental evaluation.

APPENDIX A STATIONARITY-COMPATIBILITY THEOREM

The appendix proves that stationarity in d-Simplex fixed classifiers implies compatibility across model updates, while showing why trainable classifiers cannot preserve compatibility when new classes are introduced. The theorem is supported by hyperspherical-cap geometry and empirical verification across representation dimensions.

  • Limitations of Trainable Classifiers: Trainable classifiers therefore fail to satisfy compatibility conditions when new classes are introduced.Retraining causes the nearest-neighbor angle density to decrease exponentially as class count and dimensionality increase, making prior prototype positions unlikely to persist.
  • Limitations of Trainable Classifiers: Adding m new classes reduces expected intra- and inter-class feature distances, disrupting the concentric hyperspherical-cap arrangement required for compatibility.This reduction follows from the decrease in expected nearest-neighbor angles as the number of classes increases.
  • Limitations of Trainable Classifiers: Class preallocation does not resolve the problem because unsupervised prototypes collapse during training and the intended angular margin tends to zero.Consequently, adding new classes still reduces expected intra- and inter-class distances.
  • Stationarity implies Compatibility: Stationary representations learned with a d-Simplex fixed classifier satisfy compatibility inequalities in expectation across model updates.Theorem 1 assumes shrinking class hyperspherical caps after updates; Figs. 12a and 12b verify the result across varying representation dimensions.
  • Proof setup: The proof models representations as independent hyperspherical-cap variables and evaluates compatibility using cosine distance under equal inter-class prototype separations.Closed-form integral solutions are unavailable except in the 2D case, so Monte Carlo integration is used.

APPENDIX B HIGHER-ORDER COMPATIBILITY LOSS

The HOC loss is equivalent to cross-entropy training under compatibility constraints for sequentially fine-tuned d-Simplex representations. Its contrastive component approximates dependence between successive representations, enabling higher-order compatibility.

  • Higher-order compatibility loss: Proposition 1 establishes that HOC-loss training is equivalent to cross-entropy training under compatibility constraints.This applies when ϕ_t is obtained by fine-tuning ϕ_{t−1} under a d-Simplex fixed classifier.
  • Higher-order compatibility loss: The constrained optimization becomes tractable by combining cross-entropy with a divergence term, while d-Simplex cross-entropy directly satisfies the second constraint.The method focuses the divergence term on the first constraint because the second is already satisfied by L_SCE(ϕ_t).
  • Higher-order compatibility loss: The contrastive loss L_iNCE(ϕ_t, ϕ_{t−1}) approximates the Kullback-Leibler divergence between the joint distribution and the product of marginals.This approximation makes HOC training equivalent to the constrained optimization problem.
  • Higher-order compatibility loss: The equivalence and mutual-information approximation enable HOC training to capture higher-order dependencies while learning compatible representations.The dependency is between the representations ϕ_t and ϕ_{t−1}.

APPENDIX C TINYIMAGENET20 AND CUB20 CLASSES

The appendix defines TinyImageNet-200/20 and CUB-180/20 evaluation protocols using disjoint 20-class test sets. TinyImageNet-20 is drawn from ImageNet-1k classes, while CUB-20 contains CUB-200 classes 181–200.

  • TinyImageNet20: TinyImageNet-200/20 trains on TinyImageNet-200 and evaluates on a TinyImageNet-20 test set drawn from 20 ImageNet-1k classes.The test classes are n04116512, n03447721, n11939491, n02951585, n02437616, n03538406, n02095889, n02169497, n03127925, n01532829, n03394916, n02727426, n09835506, n02105641, n03598930, n04228054, n03743016, n01582220, n04485082, and n03483316.
  • CUB20: CUB-180/20 trains on the first 180 CUB-200 classes and evaluates on the remaining 20 classes in CUB-20.CUB-20 comprises classes 181–200, from Worm-eating Warbler through Common Yellowthroat.

APPENDIX D d-SIMPLEX FIXED CLASSIFIER PYTORCH CODE

Appendix D provides GPU-based PyTorch code for generating a fixed d-Simplex classifier matrix W for K pre-allocated classes, offering faster computation than CPU implementations.

  • Implementation: The implementation generates a d-Simplex classifier matrix W on the GPU for K pre-allocated classes.The function is named dsimplex_fixed_classifier(K).
  • Implementation: The code constructs, centers, normalizes, transfers, and freezes W before returning it.It initializes W, applies CUDA transfer, mean subtraction, norm division, and sets requires_grad to False.
Loading 2606.12488v1…