Source-linked AI summary
AnyUp: Universal Feature Upsampling
Thomas Wimmer, Prune Truong, Marie-Julie Rakotosaona, Michael Oechsle, Federico Tombari, Bernt Schiele, Jan Eric Lenssen
TL;DR
Existing learned feature upsamplers usually require encoder-specific retraining, limiting their use across vision feature types and large models. AnyUp introduces a feature-agnostic upsampling architecture with window attention and crop-based training, achieving state-of-the-art quality while generalizing across unseen feature types and preserving semantics. Its main limitation is the assumption that upsampled features are linear combinations of low-resolution input features.
Problem
Existing learned feature upsamplers are generally tied to a specific encoder and require costly repeated encoder queries during retraining, which can be infeasible for large vision models.
Method
AnyUp combines a feature-agnostic layer with window attention, crop-based training, and consistency regularization to upsample arbitrary feature types across resolutions and dimensionalities.
Results
AnyUp establishes state-of-the-art feature upsampling, generalizes to feature types unseen during training, and preserves original feature semantics.
Takeaways & Limitations
AnyUp provides a single, task-agnostic upsampler that can be trained once and applied to features from different vision encoders at inference time.
Takeaways & Limitations
AnyUp assumes that upsampled features can be computed as linear combinations of low-resolution input features, potentially leaving sub-patch spatial information unused.
Abstract
from arXiv · showhide
We introduce AnyUp, a method for feature upsampling that can be applied to any vision feature at any resolution, without encoder-specific training. Existing learning-based upsamplers for features like DINO or CLIP need to be re-trained for every feature extractor and thus do not generalize to different feature types at inference time. In this work, we propose an inference-time feature-agnostic upsampling architecture to alleviate this limitation and improve upsampling quality. In our experiments, AnyUp sets a new state of the art for upsampled features, generalizes to different feature types, and preserves feature semantics while being efficient and easy to apply to a wide range of downstream tasks.
1 INTRODUCTION
AnyUp addresses the need to retrain feature upsamplers for each encoder by providing a feature-agnostic method that generalizes across feature types, resolutions, and dimensionalities. It achieves state-of-the-art upsampling while preserving feature quality and semantics.
- Existing learning-based upsamplers are tied to their training encoder and require repeated encoder queries, making retraining costly or infeasible for large vision models.This limitation motivates AnyUp’s inference-time encoder agnosticism.
- The feature-agnostic layer processes varying feature types and dimensionalities, while window attention and crop-based training further improve upsampling quality.These components are combined with consistency regularization to retain the input feature space.
- AnyUp establishes a new state of the art for feature upsampling and robustly generalizes to feature types absent during training.The authors also report high fidelity through minimized distortion of original feature semantics.
- AnyUp is the first learnable upsampler designed to work with any input feature, resolution, dimensionality, and downstream task at inference time.The model is trained once and can be applied across feature sources without encoder-specific retraining.
- AnyUp produces sharp output feature maps while preserving input feature quality, unlike prior methods that can introduce smoothing, halos, or feature distribution shifts.The visual comparison highlights these artifacts in LoftUp, JAFAR, FeatUp, and Guided Filter.
2 RELATED WORK
Feature upsampling methods trade off quality, robustness, flexibility, and computational cost. Training-free methods are broadly applicable but can blur difficult inputs, whereas learnable methods improve quality but typically specialize to one encoder or task.
- Prior feature upsampling methods can be categorized by their capabilities and reliance on data.The paper summarizes these categories in Table 1.
- Training-free methods support arbitrary input features and resolutions with low computational cost, making bilinear upsampling a common default.Guided Filtering can work well on favorable samples but may require per-sample tuning and produce excessive or insufficient blur on harder images.
- Learnable upsamplers improve quality and robustness over learning-free approaches but generally lose feature independence by fitting a specific encoder and input dimensionality.Some methods are additionally restricted to a particular downstream task or model.
3 TASK FORMULATION
Feature upsampling maps low-resolution encoder features to a high-resolution representation guided by the corresponding RGB image. This is needed because modern vision encoders downsample images for efficiency while downstream applications often require dense, high-resolution features.
- Given RGB image I_hr and low-resolution features p = e(I_hr), the goal is to produce high-resolution features q = f(p, I_hr).The input and output share channel dimension c, while spatial sizes expand from h × w to H × W.
- Large pretrained vision encoders learn latent resolutions h × w below the image resolution H × W to make image processing efficient.This applies to both convolutional networks and vision transformers.
- High-resolution features are needed for downstream tasks involving dense pixel-wise predictions and aggregating information across multiple 3D views.Feature upsampling therefore restores spatial detail needed after encoder downsampling.
4 LEARNING ENCODER-AGNOSTIC FEATURE UPSAMPLING
AnyUp combines a feature-agnostic convolution layer, local-window attention, and crop-based consistency training to upsample variable feature types and resolutions. The design processes channels independently, averages their basis-filter contributions, and restricts references to local regions while preserving feature structure.
- 4.1 FEATURE-AGNOSTIC UPSAMPLING BY DESIGN: AnyUp replaces fixed-dimensional feature processing with a feature-agnostic layer that represents arbitrary source features using canonical-dimensional outputs.Each input channel is convolved with learned basis filters, normalized across the basis dimension, and aggregated across channels.
- 4.1 FEATURE-AGNOSTIC UPSAMPLING BY DESIGN: Local-window attention limits each high-resolution query to nearby coarse features, simplifying the upsampling problem compared with global attention.The motivation is to avoid references from vastly unrelated and distant image areas.
- 4.3.1 DATA SAMPLING: Training high-resolution feature targets is difficult because extreme resolutions are computationally infeasible and can move vision models out of distribution.Local-crop supervision avoids requiring full-resolution reference features during training.
- 4.3.1 DATA SAMPLING: AnyUp trains on local image crops by matching the predicted crop features to features independently extracted from the corresponding high-resolution crop.The predicted full output is cropped to the target region before applying the feature-matching objective.
- 4.3.2 OBJECTIVE FUNCTION: Input- and self-consistency regularization improve locality and help preserve the input feature space during upsampling.The input-consistency term compares input features with appropriately downsampled predicted outputs and supports sub-object-level tasks.
5 EXPERIMENTS
Experiments show that AnyUp achieves strong downstream performance across semantic, geometric, resolution-change, feature-preservation, and cross-model evaluations while remaining efficient and generalizable.
- 5.1 COMPARISON TO PRIOR ART: AnyUp provides state-of-the-art performance for downstream semantic segmentation.Results are reported using mean Intersection-over-Union and pixel-wise accuracy.
- 5.1 COMPARISON TO PRIOR ART: AnyUp outperforms all competitors on depth and surface-normal estimation, demonstrating strong preservation of local feature structure.The evaluation uses RMSE and angular-threshold accuracy on NYUv2.
- 5.1 COMPARISON TO PRIOR ART: Across most resolution changes, AnyUp outperforms competing methods, with only a slight disadvantage when upsampling from 16 →112 pixels.The comparison varies both input feature and output resolutions on COCO semantic segmentation and depth estimation.
- 5.1 COMPARISON TO PRIOR ART: AnyUp preserves input features best while improving higher-resolution prediction quality compared with learnable and heuristic methods.The feature-space preservation evaluation uses linear probes trained on original low-resolution DINOv2 features.
- 5.2 GENERALIZATION: Training on a smaller ViT causes no significant degradation in upsampling quality, while linear probing generally follows ViT-L ≥ViT-B ≥ViT-S.The model-size study varies both the training and test models.
- 5.4 COMPUTATIONAL PERFORMANCE: AnyUp’s window-based attention reduces runtime and memory requirements by more than 50% versus JAFAR and LoftUp, while remaining slightly less efficient than FeatUp.The analysis compares parameter count, FLOPs, runtime, and forward/backward memory usage.
- 5.2 GENERALIZATION AND 5.3 ABLATION STUDY: AnyUp matches or surpasses prior upsampling methods when trained on a fundamentally different feature extractor, with small gains possible when train and test extractors match.The experiments also examine training across multiple feature extractors and ablate the proposed components.
6 LIMITATIONS
The method’s scope is limited by unavailable comparison components and by an assumption that high-resolution features are linear combinations of low-resolution features.
- 6 LIMITATIONS: FeatSharp is excluded because its learned denoising weights were unavailable at submission time.The authors note that the denoising step could be prepended to their pipeline, but do not evaluate it.
- 6 LIMITATIONS: AnyUp assumes that upsampled features can be computed as linear combinations of low-resolution input features, potentially missing sub-patch information encoded in feature channels.A larger and more complex upsampling model could potentially extract that information.
7 CONCLUSION
AnyUp is presented as a universal feature-upsampling method that operates across resolutions and unseen feature representations while achieving state-of-the-art quality.
- 7 CONCLUSION: AnyUp upsamples features from any resolution to any resolution and generalizes to feature representations absent during training.The authors identify this as the method’s central contribution.
- 7 CONCLUSION: The method combines a feature-agnostic layer, windowed attention, and a training strategy to achieve state-of-the-art upsampling quality.The code and models are publicly available.
B.1 PERFORMANCE ANALYSIS
Window attention exploits sparsity to reduce the computational requirements of AnyUp’s main upsampling operation.
- B.1 PERFORMANCE ANALYSIS: Window attention reduces attention computation from global H × W × h × w to H × W × σh × σw for windows sized σ relative to the input.The implementation uses sparse CUDA kernels through the NATTEN library.
B.2 COMPARISON AGAINST FEATUP’S IMPLICIT UPSAMPLING
AnyUp is compared with FeatUp’s implicit upsampling using pre-trained DINOv2 linear probes, because FeatUp’s inference-time optimization is too costly for higher-resolution evaluation. AnyUp outperforms prior works in this comparison.
- FeatUp’s implicit upsampling is too costly at higher resolutions to train linear probes for semantic segmentation or depth and normal estimation.The evaluation therefore uses pre-trained DINOv2 probes and 224px images rather than the 448px setting used elsewhere.
- AnyUp outperforms all prior works when evaluated with pre-trained DINOv2 linear probes.The comparison uses feature-space preservation results for DINOv2 ViT-S features upsampled to 224px.
C IMPLEMENTATION DETAILS
AnyUp is trained for 100,000 steps on ImageNet with random local crops, using AdamW and a single NVIDIA-H100 GPU. The reported training run takes around five hours, while the semantic-segmentation results reflect a corrected probing bug relative to prior work.
- AnyUp is trained for 100,000 steps on ImageNet with batch size 4 and four random local crops per training image.The method uses AdamW with a learning rate of 2e-4.
- Training takes around five hours on a single NVIDIA-H100 GPU.
- Semantic-segmentation linear-probing results differ from Couairon et al. because a bug in the probing training was fixed.All experiments use the official published weights for concurrent works.
C.1 SELF-CONSISTENCY REGULARIZATION AND DATA AUGMENTATIONS
AnyUp uses self-consistency regularization with diverse image augmentations and local-window attention to improve robustness and avoid unrelated distant references. Qualitative results show sharp boundaries, fine details, and strong preservation of feature structure across segmentation and depth tasks.
- Randomly sampled augmentations are applied to test images as part of the self-consistency training setup.
- Self-consistency regularization uses increasingly strong and diverse image augmentations to improve upsampling robustness without requiring ground-truth features.The regularization is computed at 224x224 resolution.
- Local-window attention restricts feature references to nearby regions, avoiding artifacts from unrelated distant objects.The windows are computed relative to the feature-map size, simplifying the upsampling problem.
- AnyUp preserves sharp edges and feature locality for depth estimation while retaining complete objects such as the white board.The figure notes that scale-shift alignment partly reduces LoftUp’s feature-distribution shift for depth estimation.
- AnyUp outperforms prior upsampling methods in semantic segmentation by segmenting fine details and producing cleaner outputs.
- AnyUp produces sharp object boundaries for depth estimation and matches the ground truth best among the compared qualitative results.