PaperlayerView exact revision on arXiv

Source-linked AI summary

Rank-1 Weight-Corrected Trainable Weight Averaging for Out-of-Distribution Detection

Akito Shinohara, Hiroaki Aizawa

arXiv:2610.33909v1cs.CVcs.AIcs.LG

TL;DR

OOD detection must identify unknown inputs that can trigger overconfident predictions, while standard TWA couples dominant weight components to other checkpoint parameters. We add learned rank-one correction directions to TWA; on ResNet50, results improve over fine-tuning and TWA for MSP and RankWeight, while Mahalanobis results are mixed.

  • Problem

    Standard TWA assigns one coefficient per checkpoint, coupling dominant low-rank weights with other parameters and limiting their independent adjustment for OOD detection.

    Source: §1 · p. 2

  • Method

    The method adds leading rank-one weight components extracted by SVD as TWA correction directions, learning their contributions with the ID classification objective.

    Source: §1 · p. 2

  • Results

    On CIFAR-100 with SVHN, Textures, and Places as OOD datasets, rank-1 weight-corrected TWA improves over FT and TWA with MSP and RankWeight on ResNet50, with and without OE; Mahalanobis results are mixed.

    Source: §5 · p. 14

  • Takeaways & Limitations

    Effectiveness depends on the OOD scoring protocol, and target-layer analyses with MSP and Energy favor correction at the final classification layer.

    Source: §1 · p. 2, §5 · p. 14

    Evaluation is limited to CIFAR-100 as ID with SVHN, Textures, and Places as OOD datasets; large-scale and near-OOD evaluation remain outside the study’s scope.

    Source: §5 · p. 14

Abstract

from arXiv · show

Out-of-distribution (OOD) detection identifies inputs outside the training distribution. Rank-based analyses such as RankFeat and RankWeight suggest that OOD overconfidence is closely related to dominant low-rank components in feature and weight spaces. Since weight averaging operates directly in parameter space and improves model generalization, we consider it for OOD detection from a dominant-component perspective. Trainable Weight Averaging (TWA) assigns a single coefficient to each checkpoint, limiting independent control of dominant weight components. We extend TWA with leading rank-one weight components extracted by Singular Value Decomposition as correction directions, whose contributions are learned through the in-distribution classification objective. On ResNet50, our method improves over fine-tuning and standard TWA across the evaluated datasets with MSP and RankWeight, both with and without Outlier Exposure. Mahalanobis-based evaluation yields mixed results, indicating that the benefit depends on the OOD scoring protocol. Target-layer analyses with MSP and Energy favor correction near the classifier.

1 Introduction

The introduction links OOD overconfidence to dominant weight components that standard TWA cannot control independently, then proposes learned rank-one corrections and reports scoring-dependent results.

1 Introduction

  • Rank-based analyses connect OOD behavior to dominant low-rank components, while standard TWA’s single checkpoint coefficient couples those components to the remaining parameters.This coupling limits independent adjustment of components potentially related to OOD overconfidence without also changing parameters useful for ID classification.Source: §1 · p. 2
  • The method adds SVD-derived leading rank-one weight components as TWA correction directions and learns their contributions using the ID classification objective.The proposed design avoids hard removal of dominant components.Source: §1 · p. 2
  • The expanded subspace retains the standard architecture and inference cost, and the method can be combined with Outlier Exposure.The introduction also identifies checkpoint range, target layer, OE compatibility, and backbone architecture as analysis dimensions.Source: §1 · p. 2
  • On ResNet50, the method improves over FT and TWA with MSP and RankWeight, with and without OE; Mahalanobis results are mixed.The reported evaluation covers the evaluated OOD datasets; the benefit depends on the scoring protocol.Source: §1 · p. 2
  • Target-layer analyses with MSP and Energy favor applying correction at the final classification layer.The introduction describes further analyses of checkpoint ranges, OE compatibility, and backbone architectures.Source: §1 · p. 2

2 Related Work

Related work spans output scoring, feature-space methods, data-supervised training, and parameter-space weight averaging. The paper positions rank-one correction directions as a way to address standard TWA’s coupled control of checkpoint components.

2 Related Work

  • OOD methods use post-hoc output scores, feature statistics or activations, and training-time feature regularization or auxiliary outlier data.The paper distinguishes these perspectives from its focus on parameter structure induced by weight averaging.Source: §2 · p. 3
  • Weight averaging methods include SWA, Model Soups, SWAD, and TWA; rank-based analyses associate dominant spectral components with OOD behavior and model performance.RankFeat reports dominant rank-one components in OOD features, while RankWeight reports improved performance from controlling dominant low-rank weight components.Source: §2 · p. 3
  • Standard TWA learns one coefficient per checkpoint and does not separately adjust dominant weight components, motivating rank-one correction directions in its subspace.TWA learns averaging coefficients in the subspace spanned by candidate weights.Source: §2 · p. 3

3 Proposed Method

The method extends TWA by adding leading rank-one weight components as learned correction directions. It optimizes the expanded subspace using the in-distribution classification objective.

3.1 Motivation

  • The extension adds learned correction directions from dominant weight components to TWA, avoiding deterministic removal that could discard information useful for in-distribution classification.The correction contributions are learned with the ID classification objective, without changing the architecture or adding inference-time cost.Source: §3.1 · pp. 3–4

3.2 Overview

  • The search space combines checkpoint parameters with negative dominant-component vectors, extending the standard TWA checkpoint subspace.The final model is optimized in this expanded space using the ID classification objective.Source: Figure 1 · p. 4, §3.2 · p. 5

3.3 Dominant Weight Component Extraction

  • For each checkpoint, SVD extracts the largest-singular-value rank-one component from selected weight matrices to form a dominant-component vector.Convolutional weights are reshaped for extraction and the components are restored to their original tensor shape before placement in the vector.Source: §3.3 · p. 5

3.4 Subspace Optimization

  • The method optimizes parameters in an orthonormal basis of the expanded subspace, projecting ID-training gradients onto that subspace.The coefficient objective includes an L2 regularization term.Source: §3.4 · p. 6

3.5 Adaptive Control of Dominant Weight Components

  • The expanded representation independently controls checkpoint residuals and dominant components through coefficients α_m and α_m − β_m, respectively.The learned coefficients can reduce dominant-component contributions when beneficial while retaining components useful for ID classification; −r_m is a correction direction, not deterministic removal.Source: §3.5 · pp. 6–7
  • The overall procedure follows TWA’s checkpoint-subspace training pipeline, augmented with correction directions extracted from leading rank-one components.It samples checkpoints, constructs the expanded subspace, and trains using projected gradients on ID data.Source: §3.5 · p. 7

4 Experiments

Experiments test rank-1 weight-corrected TWA on CIFAR-100-based OOD detection, comparing it with fine-tuning and standard TWA across scoring protocols, training settings, and model variants. Results favor the method on ResNet50 with MSP and RankWeight, but Mahalanobis outcomes are mixed.

4.1 Experimental Setup

  • Experiments use CIFAR-100 as ID data, three OOD test sets, and ResNet50 as the primary backbone, with additional backbone and scoring-protocol evaluations.The setup also varies checkpoint ranges and correction layers; TinyImages300K supplies auxiliary outlier data in the OE setting.Source: §4.1 · p. 8

4.2 Main Results

  • Without OE, rank-1 correction consistently improves TWA across OOD datasets on AUROC and FPR95 with MSP and RankWeight.For Energy Score, AUROC improves on all datasets, but FPR95 is essentially unchanged on Places; FT remains competitive for Energy-based detection on Textures.Source: §4.2 · p. 9, Table 1 · p. 9
  • 75.04% CIFAR-100 accuracy is achieved by the proposed method without OE, versus 72.55% for TWA and 73.94% for FT.With OE, accuracy rises from 69.94% for TWA to 72.69% for the proposed method, although FT remains highest.Source: §4.2 · p. 9, Table 2 · p. 9, §4.4 · p. 11

4.3 Analysis of the Proposed Method

  • Across checkpoint ranges, correction improves MSP results and Energy AUROC; the final classification layer performs best for MSP and Energy.The (0,101) range gives the strongest overall performance, while late-only ranges are not consistently superior. Layer4 is also effective, whereas all-layer or convolution-only correction can limit or degrade MSP detection. Across backbones, gains are clearest on ResNet50 and vary with the scoring protocol.Source: §4.3 · pp. 10–11, Table 3 · p. 10, Table 4 · p. 11

4.4 Compatibility with Outlier Exposure

  • With OE, the method consistently improves TWA across OOD datasets for MSP, Energy Score, and RankWeight.For Energy on SVHN, AUROC rises from 85.4% to 90.1% and FPR95 falls from 75.00% to 59.89%; FT remains competitive for some FPR95 values.Source: §4.4 · pp. 11–12, Table 7 · p. 13

4.5 Feature-based OOD Detection

  • Mahalanobis results are mixed: the proposed method underperforms TWA on SVHN and Places, while improving both metrics on Textures without OE.On Textures with OE, AUROC improves over TWA but FPR95 worsens, and FT performs best on both metrics.Source: §4.5 · pp. 12–13, Table 8 · p. 13

4.6 Loss Landscape Visualization

  • Without OE, the proposed method retains a flat loss landscape comparable to TWA, while both occupy flatter basins than FT.The visualization concerns the loss landscapes of FT, TWA, and the proposed method.Source: §4.6 · pp. 13–14, Figure 2 · p. 14

5 Conclusion

Rank-1 weight-corrected TWA augments the checkpoint subspace with learned leading rank-one directions and improves over fine-tuning and standard TWA with MSP and RankWeight on the evaluated ResNet50 settings. Its Mahalanobis results are mixed, and evaluation remains limited to CIFAR-100 ID with SVHN, Textures, and Places as OOD datasets.

5 Conclusion

  • With MSP and RankWeight, rank-1 weight-corrected TWA improves over fine-tuning and standard TWA on ResNet50, with and without Outlier Exposure.The evaluation uses CIFAR-100 as ID and SVHN, Textures, and Places as OOD datasets.Source: §5 · p. 14
  • Analyses favor correction near the classifier and show gains across checkpoint ranges, while effects vary by backbone and the method retains weight averaging’s flat-solution behavior.The conclusion also notes that correction effectiveness depends on target layer, backbone, and OOD scoring protocol.Source: §5 · p. 14
  • Mahalanobis results are mixed; the evaluation does not establish why degradation occurs and excludes large-scale and near-OOD settings.The study mainly uses the leading rank-one component and manually selects target layers; adaptive rank and layer selection are left for future work.Source: §5 · p. 14

Related papers

Loading 2610.33909v1…