FeatUp: A Model-Agnostic Framework for Features at Any Resolution

• Deep features sacrifice spatial detail: ResNet-50 yields 7×7 features from 224×224 input, while ViTs also reduce resolution and impede dense prediction.Source: §1 · pp. 1–2
• FeatUp aggregates low-resolution outputs across jittered images under a multiview consistency loss, using either a feedforward upsampler or a single-image implicit representation.Source: §1 · p. 2, Figure 2 · p. 2
• The feedforward upsampler generalizes Joint Bilateral Upsampling and uses a CUDA kernel orders of magnitude faster and less memory-intensive than existing implementations.Source: §1 · p. 2
• Upsampled features preserve their underlying semantics and can replace ordinary features to improve dense prediction and model explainability without retraining.Source: §1 · p. 2

• Joint Bilateral Upsampling combines spatial filtering of a low-resolution signal with intensity filtering from high-resolution guidance.Source: §2 · p. 2
• FeatUp offers a fast learned blur-kernel downsampler and an attention downsampler combining predicted salience with spatially invariant kernels.Source: Figure 3 · p. 3
• Image super-resolution methods are not well-adapted to ultra-low-resolution, high-dimensional deep features.Source: §2 · p. 3
• Bilinear, nearest-neighbor, and bicubic interpolation can blur features or ignore image structure; larger-input inference is costly and can degrade semantics.Source: §2 · p. 3
• Learned adaptive operators include deconvolutions, resize-convolutions, PAC, CARAFE, SAPA, and DGF, but some are task-specific or do not faithfully upsample existing features.Source: §2 · pp. 3–4
• FeatUp contrasts an implicit network fitted to one image with a JBU stack trained across an image corpus for fast upsampling.Source: Figure 4 · p. 4

3 METHODS

• FeatUp infers a high-resolution feature map by matching learned-downsampled versions to backbone features extracted from jittered image views.Small pads, scales, and horizontal flips provide sub-feature information for training the upsampler.Source: §3 · p. 4

3.1 CHOOSING A DOWNSAMPLER

• The downsampler offers a normalized learned blur or saliency-conditioned spatially varying blur; kernel size should grow for larger-receptive-field models.The attention option uses a 1×1 convolution to predict a saliency map.Source: §3.1 · p. 5

3.2 CHOOSING AN UPSAMPLER

• Guided JBU stacks use the original image to upsample backbone features, with CUDA inference up to 10× faster and using up to two orders of magnitude less memory than naive PyTorch.The feedforward architecture is independent of the backbone architecture.Source: §3.2 · pp. 5–6
• Both FeatUp variants consistently outperform other approaches on CAM faithfulness, linear-probe semantic segmentation, and linear-probe depth estimation.Table 1 compares these tasks across feature upsamplers.Source: Table 1 · p. 7
• Fourier color features let the implicit network use high-frequency image color information and significantly speed convergence.The authors report this enables graceful use of high-resolution image information without techniques like CRFs.Source: §3.2 · p. 7
• The per-image implicit MLP maps coordinates and image intensities to features queryable at any resolution, using over two orders of magnitude fewer parameters than a 224 × 224 explicit representation.Fourier color features provide high-frequency image information and significantly speed convergence.Source: §3.2 · p. 7

3.3 ADDITIONAL METHOD DETAILS

• Top 128 PCA components explain ∼96% of single-image feature variance and speed ResNet-50 implicit training by 60×.The authors report reduced memory use without an observable effect on learned feature quality.Source: §3.3 · p. 7
• A λtv = 0.05 total-variation prior suppresses spurious noise in implicit feature magnitudes; JBU omits it because it does not overfit.The prior is faster than regularizing full features and avoids prescribing how individual components organize.Source: §3.3 · p. 7

4.1 QUALITATIVE COMPARISONS

• FeatUp qualitatively improves feature-map upsampling over several baselines and supports higher PCA components and small-object retrieval.The comparisons use a PCA fitted on each image’s low-resolution ViT features; the same PCA maps upsampled features into RGB space.Source: §4.1 · p. 8, §4 · p. 8

4.2 TRANSFER LEARNING FOR SEMANTIC SEGMENTATION AND DEPTH ESTIMATION

• Both FeatUp variants outperform all baselines across the reported segmentation and depth linear-probe experiments, without retraining the probes.The evaluations use frozen ViT-S/16 features upsampled from 14x14 to 224x224.Source: §4.2 · p. 8

4.4 END-TO-END SEMANTIC SEGMENTATION

• FeatUp (JBU) outperforms bilinear, resize-conv, IndexNet, A2U, CARAFE, SAPA, and FADE in the SegFormer ADE20k comparison.The table caption also describes FeatUp as competitive in parameter and floating-point operation count.Source: Table 2 · p. 9

4.3 CLASS ACTIVATION MAP QUALITY

• The class activation map evaluation uses Average Drop and Average Increase to assess whether salient regions affect classifier outputs as expected.It evaluates 14x14-to-224x224 upsampling with a frozen ViT-S/16 on 2,000 validation images; the supplied passage gives no comparative outcome.Source: §4.3 · p. 9

5 CONCLUSION

• Deep models produce high-quality features at spatial resolutions that can be prohibitively low.Source: §5 · p. 9
• FeatUp uses multiview consistency; its JBU-based variant uses spatial priors, while its implicit variant supports arbitrary-resolution features.Source: §5 · p. 9
• Both FeatUp variants outperform a wide range of baselines across linear-probe transfer learning, model interpretability, and end-to-end semantic segmentation.Source: §5 · p. 9

6.1 WEBSITE, VIDEO, AND CODE

• FeatUp’s accompanying website provides additional details and a short explanatory video, and the authors provide code.Source: §6.1 · p. 16

6.2 STRIDED BASELINE IMPLEMENTATION

• The strided ViT-S/16 baseline is limited to 16x upsampling, and its maximum factor is impractical because of GPU memory requirements.Source: §6.2 · p. 16

6.3 COMPARISON TO IMAGE-UPSAMPLING METHODS

• DIP, ZSSR, and LIIF do not effectively upsample high-dimensional feature maps at the large factors handled by FeatUp.The Figure 8 caption compares these image-upsampling methods on full feature maps and zoomed-in regions.Source: §6.3 · p. 16, Figure 8 · p. 16

6.4 ABLATION STUDIES

• Fourier color features improve resolution and high-frequency detail, while the attention downsampler reduces odd edge and halo effects.The magnitude regularizer provides smoothing; without uncertainty loss, the upsampler can blur ResNet features.Source: §6.4 · p. 17, Figure 9 · p. 17
• The implicit network avoids significant artifacts seen with an explicit feature buffer, and a simple downsampler makes those artifacts less dramatic.Source: §6.4 · p. 17
• The regularizers tolerate some setting changes, but larger values can interfere with reconstruction and remove important smoothing behavior.The passage notes an optimal λ range and shows the effect of doubling both regularizer terms.Source: §6.4 · p. 17, Figure 11 · p. 18
• Removing the guidance MLP, temperature-weighted softmax, or softmax degrades SegFormer segmentation performance; removing the MLP is most detrimental.Source: §6.4 · p. 18, Table 4 · p. 18

6.5 VISUALIZING ADDITIONAL PCA COMPONENTS

• Higher PCA components separate finer categories, including a skater from a skateboard, trees from background, and clouds from sky.FeatUp upsamples each subobject’s features precisely to the object it represents.Source: Figure 12 · p. 19

6.6 SALIENCY MAP DETAILS

• The downsampler starts with blur or pooling, then uses a learned kernel to better match a network’s receptive-field size.Its pooling is analogous to FeatUp’s approximation of how a network pools information into features.Source: §6.6 · p. 20
• A content-dependent 1x1 convolution predicts salience, letting the downsampler adapt to dynamic receptive fields and important objects.The resulting spatially varying blur kernel generalizes learned fixed pooling; average pooling and common resizing methods are special cases of the attention pooling strategy.Source: §6.6 · p. 20

6.7 VISUALIZING DOWNSAMPLER SALIENCE AND KERNELS

• Fine-grained objects receive higher salience, while regions around important objects can receive lower salience, allowing salient regions to dominate other embeddings.The examples include the sky between hands and a skateboard.Source: Figure 13 · p. 20, §6.6 · p. 20

6.8 VISUALIZING PREDICTED UNCERTAINTY

• In ViT features, location-shifting spurious pink tokens receive high predicted uncertainty; similar effects are not seen in DINO or convolutional networks.The tokens may have another role, such as class-token-like information aggregation.Source: Figure 14 · p. 21

6.9 IMPROVING IMAGE RETRIEVAL FOR SMALL OBJECTS

• FeatUp retrieves only the queried traffic cones in a cluttered scene, while bilinear interpolation blends their features with the ground.The comparison uses a DINO query point and shows the target match and similarity heatmap.Source: Figure 15 · p. 21

6.9.1 LINEAR PROBE DETAILS

• A linear probe trained on 14x14 COCO features is frozen for validation, and FeatUp’s gains indicate higher resolution without compromising the original feature space.These segmentation and depth maps assess feature quality, not direct task applications.Source: §6.9.1 · p. 22

6.10 AVERAGE DROP AND AVERAGE INCREASE DETAILS

• Average Drop measures the confidence change after masking an image with its CAM, retaining the top 50% of CAM values and blurring the rest.The comparison uses classifier softmax confidence on the original and CAM-masked samples.Source: §6.10 · p. 22
• Average Increase counts cases where classifier confidence rises on CAM-masked inputs, evaluated on 2,000 correctly classified ImageNet validation images.The sample is limited to images where the label and model prediction match.Source: §6.10 · p. 22

6.11 PERFORMANCE BENCHMARKING

• The CUDA JBU kernel is benchmarked against PyTorch Unfold and TorchScript implementations, with the caption reporting lower memory overhead and faster inference.Table 6 presents the implementation comparison, but the supplied text gives no numerical values.Source: Table 6 · p. 23, §6.11 · p. 23
• FeatUp (JBU) remains competitive in GFLOP usage when varying upsampling factor, feature dimension, and target spatial dimension.The other attributes are held constant in each experiment.Source: Figure 16 · p. 23
• FeatUp (JBU)’s peak memory closely follows resize-conv and SAPA, while it is as fast as baselines and outperforms CARAFE.Figure 17 compares peak memory and inference time across upsampling factors; the figure caption describes JBU as more efficient than CARAFE at smaller factors.Source: §6.11 · p. 23, Figure 17 · p. 24
• Strided and large-image baselines become computationally infeasible beyond 8× upsampling, even with batch size 1.Figure 17 reports their metrics only up to 8× upsampling.Source: §6.11 · p. 24, Figure 17 · p. 24

6.12 ADDITIONAL QUALITATIVE RESULTS

• Both FeatUp variants produce CAMs that align more closely with image structure, refining the original maps without the discontinuous patches seen with other upsampling schemes.The CAMs combine the semantic advantages of low-resolution features with the spatial advantages of large images.Source: §6.12 · p. 25, Figure 18 · p. 25
• On COCO-Stuff, FeatUp methods produce more cohesive object and background labels and smooth artifacts in low-resolution features.The examples use linear-probe transfer learning from ViT features upsampled from 14x14 to 224x224.Source: §6.12 · p. 25, Figure 19 · p. 25
• For depth estimation, both FeatUp variants yield sharper object boundaries and smoother interiors that more closely align with true depth.The figure examples use 14x14 ViT features upsampled to 224x224.Source: §6.12 · p. 25, Figure 20 · p. 26
• Figure 21 presents end-to-end semantic-segmentation performance for upsampling methods using jointly trained architectures rather than linear probes.The supplied figure passage does not report comparative outcomes.Source: Figure 21 · p. 26

6.13 LIMITATIONS

• The implicit network captures fine detail but can produce halo effects.The figure highlights a soccer player as an example.Source: Figure 22 · p. 27
• The implicit network can rarely transfer image patterns unrelated to object semantics, while JBU captures broader contours with less sensitivity to fine detail.The figure gives a rug pattern as an example of transferred image detail.Source: Figure 22 · p. 27

6.14 IMPLEMENTATION DETAILS

• FeatUp training uses frozen, community-pretrained backbones, and Table 7 lists the training hyperparameters.Source: §6.14 · p. 27, Table 7 · p. 27