PaperlayerView exact revision on arXiv

Source-linked AI summary

Barycentric Anchored Weight Search for Robust Model Merging

Antoni Buades, Jose-Luis Lisani

arXiv:2610.14017v1cs.LGcs.AIcs.CV

TL;DR

Fine-tuning can improve target-task performance while leaving useful checkpoints unused and reducing accuracy on unseen data. BAS searches barycentric candidates with pretrained-anchored moves, guided by small OOD validation subsets. The reported results show improved OOD performance across datasets, with a trade-off in ImageNet accuracy and single-model deployment without ensemble-time overhead.

  • Problem

    Fine-tuning may leave useful checkpoints unused and create a robustness gap in which target-task gains accompany sharp declines on unseen data.

    Source: §1 · p. 1

  • Method

    BAS uses two fine-tuned checkpoints to form barycenters and search pretrained-anchored weight candidates using small OOD validation subsets.

    Source: §1 · p. 2

  • Results

    BAS selected with small OOD validation subsets improves accuracy across the considered OOD datasets, including datasets beyond the one used for selection, while ImageNet accuracy varies with the validation set.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    The selected weights remain a single model, avoiding ensemble-time memory and compute overhead while often transferring across OOD datasets.

    Source: §4.3 · p. 6

    The study is limited to two fine-tuned checkpoints, a small number of anchored step sizes, CLIP ViT-B/32, and ImageNet-derived distribution shifts.

    Source: §5 · p. 7

Abstract

from arXiv · show

We propose Barycentric Anchored Search, a post-hoc weight-space procedure for improving fine-tuned vision models under distribution shift without additional gradient training or increased inference cost. Given a pre-trained initialization and two checkpoints fine-tuned on a source dataset, the method constructs barycentric candidate weights and performs a small derivative-free search along directions connecting each candidate to the pre-trained model. Candidate weights are selected using a small validation subset of one or more out-of-distribution (OOD) datasets; the selected model is then evaluated on the held-out portions of those datasets, on additional OOD benchmarks, and on the ImageNet validation set to quantify the robustness-accuracy trade-off. The method is inspired by simplex-based derivative-free optimization, including Nelder-Mead and Torczon multidirectional search, but is specialized to the geometry of neural network weight merging. Unlike Model Soup, which typically benefits from many fine-tuned checkpoints, and unlike Model Stock, which applies a fixed geometric merge, our approach uses only two fine-tuned checkpoints and iteratively adapts the merge using validation feedback from OOD data. Experiments with CLIP ViT-B/32 show that the proposed search substantially improves performance on held-out OOD data, including ImageNet-A, ImageNet-R, and ImageNet-Sketch, while keeping ImageNet accuracy competitive. The resulting weights also improve transfer to OOD datasets not used during candidate selection, outperforming the original fine-tuned checkpoints, naive model soups, and Model Stock in our evaluations. These results suggest that lightweight optimization in anchored weight space provides a practical alternative to fixed weight-averaging rules, geometric merging, or single-model selection.

1 INTRODUCTION

Fine-tuning can leave useful checkpoints unused and create a robustness gap between in-distribution and unseen data. BAS searches barycenters and pretrained-anchored candidates using small OOD validation subsets, aiming to improve OOD accuracy while largely preserving ImageNet performance.

1 INTRODUCTION

  • Fine-tuning can leave potentially useful checkpoints unused and produce a robustness gap, with gains on a target task accompanied by declines on unseen data.Weight-space combination methods seek to improve accuracy and robustness without increasing inference costs.Source: §1 · p. 1
  • BAS searches around barycenters using perturbations anchored to the pretrained model, retaining candidates that improve accuracy on small OOD validation subsets.The final predictor remains a single model.Source: §1 · p. 2
  • BAS uses a pretrained checkpoint and two fine-tuned checkpoints to search averaged and anchored candidates without additional training.It adapts simplex-based derivative-free search ideas to weight merging and uses small OOD validation subsets to guide candidate selection.Source: §1 · p. 2
  • BAS is compared empirically with matched baselines, including naive averaging and a reproduced Model Stock baseline, for OOD accuracy and ImageNet performance.The contribution passage reports that BAS can improve OOD accuracy while largely preserving ImageNet performance.Source: §1 · p. 2

2 RELATED WORK

Related work addresses checkpoint combination, robustness under distribution shift, and weight-space geometry. BAS differs by using a short validation-guided search around barycenters rather than a fixed averaging or interpolation rule.

Weight averaging and flat minima

  • Weight averaging and interpolation are motivated by loss-landscape geometry and the possibility that combining suitable solutions reaches flatter regions associated with better generalization.BAS instead performs a short validation-guided search around barycenters.Source: §2 · p. 2

Model Soup

  • Model Soup averages fine-tuned models and can improve accuracy and robustness without extra inference cost, while greedy selection adds candidates only when validation accuracy improves.Its reported benefits are especially associated with checkpoints in a shared basin.Source: §2 · p. 2

Robust fine-tuning and anchored interpolation

  • Robust fine-tuning and anchored interpolation combine pretrained and fine-tuned solutions, while Model Stock uses layer-wise interpolation and WiSE-FT interpolates pretrained and fine-tuned weights.The cited work presents these approaches as ways to improve robustness under distribution shift or reach competitive performance with fewer checkpoints.Source: §2 · pp. 2–3

3 METHOD

BAS begins with pretrained and task-adapted checkpoints, forms barycenters, and tests candidate moves anchored to the pretrained weights using validation accuracy. Its practical two-refinement procedure is a restricted, coarse-to-fine simplex-like search, not a method with claimed convergence guarantees.

3 METHOD

  • The method seeks a single checkpoint with improved validation performance using weight-space operations and model evaluation, starting from pretrained and same-task fine-tuned models.The general setup assumes the models share an initialization and are adapted to the same task.Source: §3 · p. 3

3.1 BARYCENTRIC ANCHORED SEARCH: GENERAL ALGORITHM

  • BAS forms barycenters of its current vertices and evaluates anchored perturbations, retaining candidates according to validation performance.The perturbation parameter controls contraction toward the pretrained anchor when positive and expansion away from it when negative; the process repeats over iterations.Source: §3.1 · p. 3

3.2 BARYCENTRIC ANCHORED SEARCH: TWO REFINEMENTS. PRACTICAL ALGORITHM

  • The practical algorithm starts with the pretrained model and two fine-tuned candidates, then evaluates eight positive and negative step sizes in each of two refinements.The second refinement forms another barycenter and uses smaller step sizes than the first.Source: §3.2 · pp. 3–4

3.3 INTERPRETATION. RELATION TO DIRECT-SEARCH METHODS

  • BAS is simplex-like but updates multiple candidates around the pretrained anchor, rather than maintaining a full parameter-space simplex.It uses validation evaluations to accept improved candidates and reduces tested step sizes across iterations, but makes no convergence claim to a minimum.Source: §3.3 · p. 4

4 EXPERIMENTS

Experiments test whether a small validation-guided search over weights from a pre-trained model and two fine-tuned checkpoints improves robustness under distribution shift. BAS-guided models outperform the baselines on average OOD accuracy, with ImageNet accuracy varying according to the validation dataset used.

4.1 EXPERIMENTAL PROTOCOL

  • Experiments use CLIP ViT-B/32 and evaluate ImageNet plus ImageNet-A, ImageNet-R, and ImageNet-Sketch, with OOD validation and test images split 20% and 80%.ImageNet uses approximately 2% of its training set for validation and its official validation set for testing.Source: §4.1 · pp. 4–5

4.2 RELATION TO MODEL SOUPS AND MODEL STOCK

  • BAS uses two fine-tuned checkpoints and a pre-trained model, then evaluates barycentric and locally anchored candidates using validation accuracy, rather than merely accepting or rejecting full checkpoints as soup ingredients.The compared methods use matched starting weights; BAS performs no gradient-based optimization, and validation and test subsets are disjoint.Source: §4.2 · p. 5, §4.1 · p. 5

4.3 RESULTS AND DISCUSSION

  • Both post-hoc merging baselines improve on the individual fine-tuned checkpoints; the naive soup raises ImageNet accuracy from 0.7840 to 0.7897, while Model Stock reaches 0.7967.Model Stock also improves over the fine-tuned checkpoints on ImageNet-A, ImageNet-R, and ImageNet-Sketch.Source: §4.3 · pp. 5–6
  • ImageNet-A-guided BAS reached 0.3571 on ImageNet-A, versus 0.3239 for Model Stock and 0.2595 for the naive soup, while also reaching 0.7150 on ImageNet-R and 0.4828 on ImageNet-Sketch.ImageNet-R-guided BAS achieved the best ImageNet-A accuracy, 0.3580, and tied the best ImageNet-R accuracy, 0.7150; ImageNet-Sketch-guided BAS achieved the best ImageNet-Sketch accuracy, 0.4870.Source: §4.3 · p. 6
  • BAS guided by ImageNet-A, ImageNet-R, and ImageNet-Sketch achieved average OOD accuracies of 0.5183, 0.5136, and 0.5191, respectively, outperforming all baselines.The average is computed over the three held-out OOD test sets.Source: §4.3 · p. 6, Table 1 · p. 6
  • With ImageNet-A or ImageNet-Sketch validation, BAS obtained ImageNet accuracies of 0.7635 and 0.7721, respectively; ImageNet-R-guided BAS obtained 0.7131.The paper reports the first two values as below Model Stock but close to the original fine-tuned checkpoints, and identifies 0.7131 as the largest ImageNet drop.Source: §4.3 · p. 6

5 CONCLUSIONS

Barycentric Anchored Search uses a short validation-guided search over candidates built from pre-trained and fine-tuned weights. OOD-guided searches improve accuracy across several OOD datasets, with some ImageNet accuracy trade-off; the study is limited to CLIP ViT-B/32 and ImageNet-derived shifts.

5 CONCLUSIONS

  • BAS forms barycentric candidates from pre-trained and two fine-tuned checkpoints, then searches along anchored directions using validation feedback.The search is inspired by derivative-free simplex optimization and is adapted to neural-network weight merging.Source: §5 · pp. 6–7
  • Across ImageNet-A, ImageNet-R, and ImageNet-Sketch, OOD-guided BAS achieves higher average OOD accuracy than the pre-trained model, fine-tuned checkpoints, naive soup, and Model Stock.The final predictor remains a single model, without ensembling or increased inference-time cost.Source: §5 · p. 7
  • OOD validation shifts BAS toward better robustness at the cost of some ImageNet accuracy, while ImageNet validation preserves ImageNet accuracy.Changing the validation distribution adapts the same small set of precomputed weights to different deployment priorities.Source: §5 · p. 7
  • The study focuses on CLIP ViT-B/32 and ImageNet-derived distribution shifts, leaving other architectures, pre-training families, benchmarks, and non-vision tasks for future evaluation.The authors also identify larger checkpoint sets, deeper schedules, adaptive λ choices, and further analysis of checkpoint geometry as future directions.Source: §5 · p. 7
  • BAS complements Model Soups and Model Stock by retaining single-model deployment while adding validation-driven adaptation of merged weights.The authors characterize constrained barycentric search with a small amount of validation feedback as a mechanism for robust model merging.Source: §5 · p. 7

Related papers

Loading 2610.14017v1…