PaperlayerView exact revision on arXiv

Source-linked AI summary

Upsample Anything: A Simple and Hard to Beat Baseline for Feature Upsampling

Minseok Seo, Mark Hamilton, Changick Kim

arXiv:2511.16301v2cs.CV

TL;DR

Vision Foundation Model features are spatially coarse, while pixel-level applications need high-resolution outputs. Upsample Anything fits anisotropic, edge-aware kernels for each image and transfers them to features; the paper reports strong results across tasks with a fast per-image optimization.

  • Problem

    Vision Foundation Model features are typically downsampled by 14–16×, limiting fine spatial detail and requiring costly, memory-intensive decoders for pixel-level prediction.

    Source: §1 · p. 2

  • Method

    Upsample Anything uses test-time optimization to learn per-pixel anisotropic kernels from image reconstruction, then applies them to low-resolution foundation features.

    Source: §1 · pp. 2–3

  • Results

    The method reports state-of-the-art performance without learnable modules, significant gains in feature and depth upsampling, and 0.419 seconds per 224×224 image.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The learned kernels serve as a universal, plug-and-play upsampling operator across diverse resolutions and domains without pre-training or architectural constraints.

    Source: §1 · p. 3, §6 · p. 8

    On noisy or heavily perturbed inputs, reconstruction-based optimization can fit corruptions rather than underlying structure; adding a denoising stage is outside the current scope.

    Source: §15 · p. 13

Abstract

from arXiv · show

We present \textbf{Upsample Anything}, a lightweight test-time optimization (TTO) framework that restores low-resolution features to high-resolution, pixel-wise outputs without any training. Although Vision Foundation Models demonstrate strong generalization across diverse downstream tasks, their representations are typically downsampled by 14x/16x (e.g., ViT), which limits their direct use in pixel-level applications. Existing feature upsampling approaches depend on dataset-specific retraining or heavy implicit optimization, restricting scalability and generalization. Upsample Anything addresses these issues through a simple per-image optimization that learns an anisotropic Gaussian kernel combining spatial and range cues, effectively bridging Gaussian Splatting and Joint Bilateral Upsampling. The learned kernel acts as a universal, edge-aware operator that transfers seamlessly across architectures and modalities, enabling precise high-resolution reconstruction of features, depth, or probability maps. It runs in only $\approx0.419 \text{s}$ per 224x224 image and achieves state-of-the-art performance on semantic segmentation, depth estimation, and both depth and probability map upsampling. \textbf{Project page:} \href{https://seominseok0429.github.io/Upsample-Anything/}{https://seominseok0429.github.io/Upsample-Anything/}

1. Introduction

Vision Foundation Models provide transferable features but downsample them, leaving pixel-level systems to recover detail with costly decoders. Upsample Anything instead learns image-specific, edge-aware mixing kernels at test time, addressing limits of training-based and slower feature-upsampling approaches.

1. Introduction

  • Foundation features are typically downsampled by 14–16×, so pixel-level systems rely on computationally expensive, memory-intensive decoders to recover spatial detail.Such decoders can be difficult to generalize to new architectures or resolutions.Source: §1 · p. 2

2.2. Feature Upsampling

  • Feature upsampling methods restore spatial detail in pretrained representations, but learned approaches rely on dataset-level training and may generalize poorly to unseen domains, architectures, resolutions, or out-of-distribution data.FeatUp offers a learnable JBU module and an implicit-function module, with the latter optimized per image.Source: §1 · p. 2, §2.2 · p. 3

1. Introduction

  • Dataset-trained upsamplers require retraining when the backbone or target dataset changes, while FeatUp (Implicit) takes an average of 49 seconds to converge on a 224-sized image.The dataset-trained methods also face heavy memory use and often operate only up to 112–224 pixels.Source: §1 · p. 2
  • Upsample Anything learns per-pixel anisotropic kernels through lightweight test-time optimization, then applies them to foundation features for high-resolution outputs across pixel-level tasks.The method reports state-of-the-art or near-SOTA results on multiple pixel-level benchmarks, including semantic segmentation and depth estimation.Source: §1 · pp. 2–3

2.1. Joint Bilateral Upsampling

  • Joint Bilateral Upsampling transfers structure from a high-resolution guidance image to a low-resolution signal by combining spatial proximity and color similarity, but its fixed hand-crafted kernel limits quality.It is a non-learning method for edge-preserving upsampling of signals such as depth or label maps.Source: §2.1 · p. 3

3. Preliminaries

The method views joint bilateral upsampling through the lens of Gaussian splatting and extends it with image-specific anisotropic kernels. It fits those kernels by reconstructing the guidance image, then transfers them to low-resolution foundation features for high-resolution rendering.

3. Preliminaries

  • 2D Gaussian Splatting renders image-plane kernels by normalized weighted summation, equivalent to a spatially varying anisotropic convolution.Its kernels encode scale and orientation in their covariance; rendering requires no depth sorting because they lie on one plane.Source: §3 · p. 3
  • Joint Bilateral Upsampling is a discrete, isotropic, guidance-modulated instance of 2D Gaussian Splatting; Upsample Anything extends it with optimized per-pixel anisotropic covariances and range scales.This allows locally varying kernel orientations and scales while retaining edge-preserving behavior.Source: §3 · pp. 3–4

4.1. Overview

  • Excluding feature extraction, optimization and inference take approximately 0.419 seconds for a 224×224 image, as the method transfers learned kernels to render high-resolution features.The overview describes test-time optimization followed by feature rendering using the same spatial–range weighting mechanism.Source: §4.1 · p. 4

4.2. Algorithm Design

  • Unlike standard JBU’s global, isotropic kernels, Upsample Anything assigns each low-resolution pixel a Gaussian with separate spatial scales, orientation, and range scale.The design targets greater expressivity near complex structures.Source: §4.2 · p. 4
  • Feature rendering strictly reweights existing low-resolution features without generating content, supporting transfer across backbones and tasks.The paper characterizes its per-image, pixel-wise mixing weights as resolution-free, model-agnostic, and robust to unseen domains.Source: §4.2 · p. 5

4.3. Test-Time Optimization

  • Test-time optimization downsamples the high-resolution guidance image and fits image-specific kernels by reconstructing it from the downsampled version.The resulting kernels encode how spatially and photometrically similar neighbors should be blended.Source: §4.3 · p. 5
  • After optimization, the learned parameters are transferred directly to the foundation feature space to render low-resolution features as a high-resolution feature map.The same kernels are reused rather than separately learned for feature values.Source: §4.3 · p. 5

5. Experiments

The experiments evaluate Upsample Anything across segmentation, depth estimation, raw depth-map and probability-map upsampling, multiple backbones, and varying resolutions. Results report strong segmentation and geometric-task performance, sharp reconstructions at low resolutions, and efficient scaling to large outputs.

5.1. Experimental Setting

  • Evaluation spans segmentation, NYUv2 depth estimation, depth-map upsampling on NYUv2 and Middlebury, and Cityscapes probability-map upsampling across five backbones.The implementation uses fully parallel PyTorch computations and 50 per-image optimization iterations.Source: §5.1 · p. 5, §5.2 · p. 5

5.3. quantitative results

  • Upsample Anything attains the best segmentation accuracy on COCO, PASCAL-VOC, and ADE20K, with AnyUp consistently second.With strong backbones, all methods’ eventual gains over bilinear are described as modest.Source: Table 1 · p. 5, §5.3 · p. 5
  • On NYUv2, depth and surface-normal estimation reaches RMSE 0.498, δ1 0.829, and mean 21.5°.For raw depth-map upsampling, Upsample Anything leads on Middlebury, while bilinear has lower NYUv2 RMSE (0.159 vs. 0.237).Source: Table 2 · p. 6, §5.3 · p. 6, Table 3 · p. 6
  • The probability-map variant predicts at feature resolution, then upsamples probabilities, achieving the lowest computational cost and highest accuracy in Table 1.The authors suggest upsampling task probabilities rather than intermediate features as a promising segmentation paradigm.Source: §5.3 · pp. 5–6

5.4. Qualitative results

  • Across input resolutions, Upsample Anything preserves sharp boundaries and fine structures at 7×7 and 4×4, while AnyUp tends to oversmooth.The comparison covers varying input resolutions.Source: §5.4 · p. 6, Figure 5 · p. 7

5.5. Ablation Study

  • Linear memory growth lets Upsample Anything run stably at 1024×1024, where AnyUp cannot execute, despite slightly higher computational cost.The method avoids dense pairwise affinity maps.Source: §5.5 · p. 7, Table 4 · p. 7

5.4. Qualitative results

  • Across backbones, Upsample Anything produces sharper edges, richer textures, and more distinct feature clustering than AnyUP.Figure 7 computes target-image cosine similarity against a reference feature averaged within the reference mask.Source: Figure 6 · p. 8, Figure 7 · p. 8

5.5. Ablation Study

  • GSJBU balances speed, convergence, and accuracy; 50 TTO iterations give the best segmentation accuracy and fastest inference (0.419s), while PSNR reaches 35.60 after about 500 iterations.The authors adopt 50 iterations as the default.Source: Table 5 · p. 8, §5.5 · p. 8, Table 6 · p. 8

6. Conclusion

Upsample Anything connects Joint Bilateral Upsampling and Gaussian Splatting in a lightweight test-time optimization framework. It reports efficient, state-of-the-art upsampling, while noting instability under severe occlusions or low-SNR guidance.

6. Conclusion

  • Upsample Anything connects Joint Bilateral Upsampling and Gaussian Splatting under a continuous formulation, without pre-training or architectural constraints.It is presented as a universal, plug-and-play framework combining JBU’s simplicity with Gaussian representation’s expressive power.Source: §6 · p. 8
  • 0.419 seconds per 224×224 image; the framework reports significant gains in feature and depth upsampling and state-of-the-art performance without learnable modules.The conclusion characterizes its upsampling as efficient and robust across diverse resolutions and domains.Source: §6 · p. 8
  • Severe occlusions or low-SNR guidance may make optimization unstable.The authors identify improved robustness and adaptability under challenging conditions as future work.Source: §6 · p. 8

7. Semantic Segmentation on Cityscapes

On Cityscapes, feature- and probability-map upsampling produced results close to bilinear interpolation under the stated LoftUp evaluation setup. The authors attribute the narrow differences to the dataset’s large, regular structures, coarse annotations, and a sufficiently trained segmentation head.

7. Semantic Segmentation on Cityscapes

  • On Cityscapes, all methods, including feature- and probability-level upsampling, produced almost the same mIoU as bilinear interpolation under the stronger evaluation setup.The setup used 448×448 inputs, 100 epochs, and a learning-rate scheduler; the reported differences remain within a narrow range.Source: §7 · p. 9, Table 7 · p. 9
  • The authors report that an automated code audit and manual reviews found no implementation inconsistencies or bugs.Source: §7 · p. 9
  • Cityscapes’ large, regular structures and relatively coarse annotations leave little room for gains because bilinear interpolation is already near optimal with a sufficiently trained segmentation head.The authors contrast this with COCO, PASCAL-VOC, and ADE20K, where small objects and complex boundaries are associated with clear upsampling benefits.Source: §7 · p. 9

8. Details of Probabilistic Map Upsampling in Table 1.

The section describes a segmentation pipeline that predicts from downsampled features with a lightweight 1×1 convolution, then upsamples the probability map. Figure 8 contrasts this approach with decoder-based and pretrained-upsampler pipelines.

8. Details of Probabilistic Map Upsampling in Table 1

  • The probability-map configuration predicts segmentation from downsampled features with a lightweight 1×1 convolution, then upsamples the map to high-resolution outputs.This is the configuration associated with Table 1 and Figure 8-(c).Source: §8 · p. 9
  • The Upsample Anything pipeline handles both feature and segmentation upsampling without additional training, unlike the figure’s decoder-based and pretrained-upsampler alternatives.Figure 8 contrasts task-specific decoders, pretrained feature upsamplers, and the proposed test-time optimization pipeline.Source: Figure 8 · p. 9
  • Because computation occurs on a small feature map, heavier or more complex decoders are expected to be feasible without large computational overhead.The authors describe this as potential for the pipeline, while decoder design is beyond the work’s scope.Source: §8 · p. 9

9. Details of Depth Estimation in Table 2.

The depth experiments use a DPT-based setup with features from a pretrained Vision Foundation Model encoder. Because prior implementations did not specify DPT’s internal upsampling details, the authors reimplemented the head and used it consistently.

9. Details of Depth Estimation in Table 2

  • The depth-estimation experiments follow the DPT-based setup used in prior works [23].The authors note that DPT includes an internal upsampling stage.Source: §9 · p. 9
  • The algorithm maps an input image to a predicted depth map by extracting multi-scale pretrained-encoder features and passing them through a DPT head.The stated example encoder is DINOv2; the head applies convolutions, ReLU, and an optional inversion step before outputting depth.Source: §9 · p. 10
  • The authors reimplemented the DPT head as described in their algorithm because exact internal upsampling details were unavailable, then used it consistently across depth experiments.Source: §9 · p. 10

10. From 2D Low-Resolution Feature Maps to 3D High-Resolution Feature Volumes

The framework extends test-time upsampling from 2D low-resolution features to dense 3D feature volumes, and its visualizations show adaptive kernels and sharp segment boundaries.

10. From 2D Low-Resolution Feature Maps to 3D High-Resolution Feature Volumes

  • The method expands 2D low-resolution features into dense 3D volumes by optimizing anisotropic kernels with RGB-D guidance, then reuses them to upsample semantic features.The resulting volume captures local geometric and appearance structure, with depth continuity and object boundaries, despite no explicit 3D supervision.Source: §10 · p. 10

11. Gaussian Blob Visualization

  • Learned Gaussian blobs show locally coherent orientations in some regions, but overparameterization and redundancy leave some blobs noisy or difficult to interpret.Overlapping or consistently oriented blobs around features such as eyes, noses, and corners suggest nearby kernels capture similar local structures.Source: §11 · p. 10

12. Segment-then-Upsample Pipeline Visualization Results

  • Upsampling segmentation logits by 16× after low-resolution segmentation produces visually sharper boundaries and finer structures than bilinear interpolation.The results are described as semantically consistent despite the large upsampling ratio.Source: §12 · p. 10

13. Formal Relation Between Joint Bilateral Upsampling and Gaussian Splatting

The section frames Joint Bilateral Upsampling (JBU) as a Gaussian Splatting evaluation in a lifted space, then relates anisotropic extensions and continuous-grid convergence to that view.

Formal relation

  • JBU equals a normalized Gaussian mixture evaluated in the lifted spatial-and-guidance domain, with centers on the lifted low-resolution grid and fixed block-diagonal covariance.The section presents this as a conceptual link, not a claim that JBU and Gaussian Splatting are mathematically equivalent in general.Source: §13 · pp. 10–12

Alternative formulations

  • The lifted-domain formulation is query-independent, while keeping the domain 2D with query-dependent amplitude matches common Gaussian Splatting implementations.The passage describes the query-independent form as mathematically cleaner.Source: §13 · p. 12

Anisotropic generalization

  • The anisotropic per-center model reduces exactly to JBU when spatial covariance is isotropic and the range scale is shared across centers.The generalized model enables per-center covariance learning beyond this isotropic limit.Source: §13 · pp. 12–13

Discrete-to-continuous convergence

  • As the low-resolution grid spacing tends to zero, discrete JBU and its normalized ratios converge to the continuous lifted-domain Gaussian Splatting counterpart under the stated regularity assumption.The assumption is that the low-resolution feature map admits a bandlimited or Lipschitz-continuous interpolation.Source: §13 · p. 12

14. Hyperparameter Table

The listed hyperparameter settings are practical initializations for test-time optimization, not strict requirements.

Hyperparameter settings

  • After 50 optimization steps, spatial and range parameters are refined, so final performance depends only weakly on their initial values.Initialization based on receptive-field size, guidance-image dynamic range, and desired locality mainly accelerates stable convergence.Source: §14 · p. 13
  • The table’s settings function as soft priors rather than strict hyperparameter requirements.Suboptimal initial values are eventually corrected by optimization, according to the passage.Source: §14 · p. 13

15. Limitation under Low-SNR or Corrupted Inputs

The method’s image-reconstruction objective is vulnerable to severe input corruption, although the paper reports robustness to moderate perturbations and stronger performance on clean or multimodal inputs.

Low-SNR and corrupted inputs

  • Extremely low-SNR images or severe corruption are a clear limitation, despite robustness across datasets and moderate perturbations such as ImageNet-C.The authors attribute this boundary to test-time optimization reconstructing the input image and assuming a clean, reliable signal.Source: §15 · p. 13
  • With salt-and-pepper artifacts or heavy sensor perturbations, optimization tends to fit the corruption rather than recover underlying structure.The paper illustrates increasing corruption levels of 10% and 20%.Source: §15 · p. 13
  • AnyUp remains stable under noise, whereas Upsample Anything faithfully reconstructs the noisy signal and unintentionally amplifies noise in RGB and feature outputs.The authors note that this limitation is common to TTO image-restoration methods optimized directly on corrupted inputs.Source: §15 · p. 13

Scope and comparison

  • The authors characterize Upsample Anything as excelling on visually clean inputs and multimodal signals such as RGB-D or 3D features, while AnyUp is more robust under corruption.A denoising stage is suggested as a possible mitigation but is outside the current scope.Source: §15 · pp. 13–14
Loading 2511.16301v2…