PaperlayerView exact revision on arXiv

Source-linked AI summary

NAF: Zero-Shot Feature Upsampling via Neighborhood Attention Filtering

Loick Chambon, Paul Couairon, Eloi Zablocki, Alexandre Boulch, Nicolas Thome, Matthieu Cord

arXiv:2511.18452v1cs.CV

TL;DR

Reduced spatial resolution in Vision Foundation Model features complicates pixel-level tasks, while existing upsamplers trade fixed filter forms against VFM-specific retraining. NAF uses image-guided attention to upsample features zero-shot across VFMs, achieving state-of-the-art results across tasks and extending to image denoising. Its current limitation is the lack of a principled way to identify which VFM representations best support upsampling.

  • Problem

    VFM features are spatially downsampled, while classical upsamplers use fixed forms and learnable alternatives depend on VFM-specific guidance and retraining.

    Source: §1 · pp. 1–2

  • Method

    NAF uses Cross-Scale Neighborhood Attention to aggregate low-resolution features with guidance derived solely from the input image, independently of the target VFM feature distribution.

    Source: §1 · p. 2, §3.1 · p. 3

  • Results

    NAF achieves state-of-the-art performance across multiple tasks and datasets, including use with very large VFMs, and also demonstrates generalization to image denoising.

    Source: §1 · p. 2, Conclusion · p. 8

  • Takeaways & Limitations

    NAF’s local interpolation is described as fast and lightweight relative to previous VFM upsamplers, while supporting scaling to previously unseen ratios.

    Source: Conclusion · p. 8

    The authors lack a principled framework for identifying which VFMs provide the most informative patch representations for learning upsampling, and report no clear gains from combining or using larger or stronger VFMs.

    Source: E. Limitations and Perspectives · p. 19

Abstract

from arXiv · show

Vision Foundation Models (VFMs) extract spatially downsampled representations, posing challenges for pixel-level tasks. Existing upsampling approaches face a fundamental trade-off: classical filters are fast and broadly applicable but rely on fixed forms, while modern upsamplers achieve superior accuracy through learnable, VFM-specific forms at the cost of retraining for each VFM. We introduce Neighborhood Attention Filtering (NAF), which bridges this gap by learning adaptive spatial-and-content weights through Cross-Scale Neighborhood Attention and Rotary Position Embeddings (RoPE), guided solely by the high-resolution input image. NAF operates zero-shot: it upsamples features from any VFM without retraining, making it the first VFM-agnostic architecture to outperform VFM-specific upsamplers and achieve state-of-the-art performance across multiple downstream tasks. It maintains high efficiency, scaling to 2K feature maps and reconstructing intermediate-resolution maps at 18 FPS. Beyond feature upsampling, NAF demonstrates strong performance on image restoration, highlighting its versatility. Code and checkpoints are available at https://github.com/valeoai/NAF.

1. Introduction

Vision Foundation Models produce spatially reduced features that limit fine-grained tasks, while existing upsamplers trade adaptable accuracy against efficiency and VFM-specific retraining. NAF uses image-guided attention to upsample features zero-shot across VFMs, with reported strong performance and applications beyond upsampling.

1. Introduction

  • Reduced spatial resolution limits VFM feature effectiveness for fine-grained tasks, while increasing input size can degrade performance and impose prohibitive computation and memory costs.Only a few VFMs handle non-standard resolutions effectively [19] [41] [46]; for most, larger inputs degrade performance [6].Source: §1 · pp. 1–2
  • Classical filters use fixed forms, whereas learnable upsamplers offer higher-quality results but require retraining when the underlying VFM changes.Learnable methods also bring complex pipelines, low throughput, high memory consumption, and large parameter counts.Source: §1 · p. 2
  • NAF combines image-based guidance and relative spatial proximity through Cross-Scale Neighborhood Attention and Rotary Position Embeddings.It predicts spectral coefficients whose IDFT reconstructs an adaptive, data-dependent upsampling filter.Source: §1 · p. 2
  • NAF is a VFM-agnostic upsampler that generalizes zero-shot to features from any VFM, guided solely by high-resolution images.It uses Cross-Scale Neighborhood Attention for content-aware local interpolation, with strong similarities to classical filtering and IDFT.Source: §1 · p. 2
  • NAF achieves state-of-the-art performance across diverse vision tasks and datasets, across VFM families and model sizes, and extends to image denoising.The introduction also reports high-throughput zero-shot upsampling to resolutions up to 2K and use with very large VFMs.Source: §1 · p. 2

2. Background and Related Work

Feature upsampling uses local aggregation to reconstruct high-resolution features, with guidance helping filters preserve edges and fine structure. Prior methods range from fixed filters to task-specific or VFM-specific learned upsamplers; NAF derives guidance from the input image alone.

2. Background and Related Work

  • Classical spatial-and-content-aware filters use guidance-dependent weights to aggregate a local neighborhood, but fixed kernel shapes and handcrafted similarity functions limit their adaptability.The guidance can help preserve edges and fine structures; bilateral and joint bilateral filters are examples,.Source: §2 · p. 2
  • Learned feature upsampling optimizes parameterized aggregation kernels using guidance that typically encodes the input image and low-resolution features.The general formulation uses a trainable image encoder and learnable kernel parameters.Source: §2 · p. 2
  • Table 1 compares methods for ×16 upsampling of (384, 28, 28) input features; its maximum ratio is the largest scale fitting on one A100 40GB GPU.Methods are sorted by maximum ratio and FPS.Source: Table 1 · p. 3
  • Task-designed methods such as CARAFE [50], DySample [29], and SAPA [31] require retraining for each downstream task and are not primarily designed for general VFM features.They include content-aware kernels or sampling points, and local-similarity-based reweighting.Source: §2 · p. 3
  • VFM-specific methods such as FeatUp [11], LiFT [43], FeatSharp [36], JAFAR [6], and LoftUp rely on low-resolution VFM features for guidance.Some attention-based methods support continuous upsampling to arbitrary resolutions.Source: §2 · p. 3
  • NAF derives guidance solely from a lightweight encoder applied to the input image, avoiding dependence on VFM low-resolution features; it is reported roughly ×4 faster than AnyUp with 25% fewer parameters and superior reconstruction quality.AnyUp [51] is VFM-agnostic but still uses low-resolution VFM features for guidance.Source: §2 · p. 3

3. Method: NAF

NAF reconstructs high-resolution VFM features through image-guided, localized attention, learning adaptive spatial and content weights. Its dual-branch encoder and image-only guidance support zero-shot transfer, while training uses feature reconstruction supervision.

3.1. Architecture

  • NAF reconstructs high-resolution features by aggregating nearby low-resolution VFM features with attention, learning spectral coefficients whose inverse discrete Fourier transform yields an adaptive spatial kernel.The input image guides reconstruction at the target resolution.Source: §3.1 · p. 3
  • Queries and keys derive from an image-guidance encoder with pixel-wise 1×1 and contextual 3×3 branches, and RoPE encodes relative position.The branches each contain L blocks and output C/2 channels before concatenation; keys are pooled onto the low-resolution grid.Source: §3.1 · pp. 3–4, Figure 3 · p. 4
  • Local neighborhoods align receptive fields across scales, enabling image-only attention that transfers across VFMs without retraining and uses about 40% fewer GFLOPs than JAFAR [6].The formulation parallels joint filtering: RoPE supplies spatial information and query-key similarity supplies content-adaptive weights.Source: §3.1 · p. 4

3.2. Training

  • Training pairs high-resolution VFM features with features from bilinearly downsampled images, then minimizes an ℓ2 reconstruction loss without extra regularizers.The high-resolution image guides upsampling of the low-resolution features.Source: §3.2 · p. 5

4. Experiments

Experiments assess upsampled features through linear probing for segmentation and depth, then test open-vocabulary and video segmentation transfer. NAF reports the strongest results across the evaluated VFMs, datasets, model sizes, and downstream transfer settings.

4. Experiments

  • Evaluation uses linear probing on Pascal VOC segmentation and NYUv2 depth estimation across VFMs, with features upsampled to original image resolution.The experiments also test transfer to open-vocabulary segmentation and video object segmentation propagation.Source: §4.1 · p. 5, §4.1.1 · p. 5, §4.1.2 · p. 6

4.1.1. Semantic segmentation.

  • +5.58 mIoU average over Nearest, NAF leads across tested VFMs; JAFAR gains +5.12 mIoU but requires retraining for each VFM.NAF also applies to DINOv3-7B, where VFM-dependent upsamplers run out of memory.Source: §4.1.1 · p. 5
  • +4.23 mIoU average over nearest-neighbor, with NAF best across COCO, Pascal VOC, ADE20K, and Cityscapes using DINOv3-B features.The reported results also show a substantial improvement on Cityscapes for fine-grained segmentation.Source: §4.1.1 · pp. 5–6, Table 3 · p. 6

4.1.2. Depth estimation

  • +3.16 δ1 average over Nearest across tested VFMs for depth estimation; NAF gains +0.57 over AnyUp and +12.69 on DINOv3-7B versus nearest-neighbor.Source: §4.1.2 · p. 6

4.1.1. Semantic segmentation.

  • +5.27 mIoU average over nearest-neighbor, with NAF setting state-of-the-art scores across DINOv2-R Small, Base, and Large models.Source: Table 4 · p. 6, §4.1.1 · p. 6

4.2. Downstream transfer

  • +1.04 mIoU over the baseline in open-vocabulary segmentation, versus +0.63 for JAFAR; NAF has the highest average across evaluated VFMs.ProxyCLIP uses NAF as a drop-in replacement for bilinear upsampling without additional training.Source: §4.2 · p. 6, Table 5 · p. 7
  • +3.37 mIoU average over the baseline in DAVIS video segmentation propagation, where NAF achieves the best overall performance.The evaluation replaces bilinear upsampling with alternative upsamplers and propagates masks using feature-space similarity matching.Source: §4.2 · p. 6, Table 5 · p. 7

5. Ablations

The ablations examine the dual-branch image encoder, attention-key pooling, and spatial positional encoding. They favor a context branch, pooled keys without an added convolution, and RoPE.

5. Ablations

  • The dual-branch encoder benefits from a context branch, outperforms Inception-style blocks while remaining simpler and more efficient, and gains accuracy as C and L increase with diminishing returns.The context branch captures local structure beyond per-pixel information; larger C and L also increase computational cost.Source: §5 · p. 7, Table 6 · p. 7
  • The main experiments use a lightweight configuration with C = 256 and L = 2, while NAF++ uses C = 768 and L = 5 for improved accuracy at additional parameter and speed cost.The lightweight configuration has roughly the same number of parameters as the previous state-of-the-art VFM-specific upsampler.Source: §5 · p. 7
  • Both AvgPool and MaxPool outperform Bilinear keys, while adding a convolution after pooling significantly degrades performance.The authors attribute the degradation likely to broken channel alignment between queries and keys, and use AvgPool in all experiments.Source: §5 · p. 7, Table 7 · p. 8
  • RoPE achieves the best performance among the tested spatial encodings, and positional encoding is crucial for spatial awareness.The compared alternatives include Gaussian and Manhattan kernels and no positional encoding; RoPE captures relative geometry without additional parameters.Source: §5 · pp. 7–8, Table 7 · p. 8

6. Extension: Image Restoration

NAF’s filter-based formulation also applies to standard RGB restoration images, not only low-resolution features. In denoising tests it performs strongly across noise types, although Restormer remains the top performer with roughly 40× more parameters.

6. Extension: Image Restoration

  • NAF’s filter-based formulation accepts either low-resolution feature maps or standard RGB images because it imposes no constraints on input or guidance dimensionality or structure.For restoration, the guidance and input are the same image, so average pooling becomes the identity.Source: §6 · p. 8
  • The denoising evaluation reconstructs clean 448 × 448 RGB images corrupted with Gaussian or channel-wise salt-and-pepper noise, including fixed and dynamically sampled noise.The denoising model uses the feature-upsampling architecture with a larger neighborhood kernel of 15 and is trained with L1, L2, and SSIM losses.Source: §6 · p. 8
  • NAF achieves strong denoising results across all tested noise levels and types, while Restormer remains the top performer with roughly 40× more parameters.The comparison covers Gaussian and channel-wise salt-and-pepper noise, with fixed and dynamic corruption settings.Source: §6 · p. 8, Table 8 · p. 8
  • Conventional models perform well on Gaussian noise but degrade significantly on salt-and-pepper corruption, whereas NAF recovers structural details in both cases.Restormer is specifically designed for image restoration; NAF was not.Source: §6 · p. 8

Conclusion

The conclusion presents NAF as a VFM-agnostic feature upsampler that scales to very large models and achieves state-of-the-art results on many downstream tasks. Its local interpolation is described as fast and lightweight, with potential beyond upsampling.

Conclusion

  • NAF scales features from any VFM, including very large 7B-VFMs, and achieves state-of-the-art results on many downstream tasks.The conclusion describes its core as Cross-Scale Neighborhood Attention that does not depend on the target feature distribution.Source: Conclusion · p. 8
  • NAF’s local interpolation design is fast and lightweight compared with previous VFM upsamplers, enabling previously unseen scaling ratios.Source: Conclusion · p. 8
  • The conclusion presents NAF as a versatile module for use in tasks beyond feature upsampling.Source: Conclusion · p. 8

Supplementary Material

The supplementary material analyzes NAF’s RoPE-based attention as an adaptive filter with a Fourier interpretation, then documents its training, evaluation setups, and visualized behavior.

A. Mathematical Discussions / A.1. RoPE Expansion

  • NAF’s RoPE-neighborhood attention is described as learning the IDFT of an adaptive upsampling kernel rather than merely reweighting inputs by image-encoder distance.The derivation relates its attention-weighted interpolation to Joint Bilateral Filtering.Source: A. Mathematical Discussions · p. 11

A.1. RoPE Expansion

  • RoPE rotates paired channels using axial position; its attention interaction combines feature coherence and orthogonality, modulated by relative horizontal or vertical distance.The rotation angle uses normalized coordinates and channel-specific wavelengths.Source: §A.1 · pp. 11–12

A.2. Representation with magnitudes and angles.

  • The magnitude-and-angle decomposition treats feature magnitudes as signal strength and couples spatial phase difference with semantic phase alignment through cosine angle addition.High attention scores occur when the semantic relationship compensates for spatial offset, forming a spatially varying matched filter.Source: §A.2 · p. 12

1. Learning Fourier Coefficients.

  • Channel-pair magnitudes act as spectral power, letting the encoder control each frequency band’s contribution to the interpolation kernel.Low frequencies represent global structure and high frequencies fine detail.Source: §1 · p. 13

2. Spatially-Varying Filter Synthesis.

  • NAF acts as a spatially varying band-pass filter: it can suppress high frequencies in smooth regions and boost them at sharp boundaries.The RoPE term encodes spatial shift, while summing spectral bands reconstructs the impulse response for the target sub-pixel position.Source: §2 · p. 13, §3 · p. 13

3. Shift Demodulation.

  • Attention maps show boundary and shape discrimination, while uniform regions exhibit a decreasing attention pattern like a Gaussian filter.Source: Figure 7 · p. 13

B.1. Training Details

  • Training uses 25k initial iterations followed by 2.5k iterations with larger target images; the full procedure takes about one hour and 9 GB on an A100.The second stage yields an average +0.4 mIoU gain on VOC linear-probing segmentation; the final model uses L = 2, C = 256, and kernel size 9.Source: §B.1 · pp. 13–14

B.2. Task setups / B.3. Visualizations

  • Evaluations include linear-probing segmentation on four datasets, NYUv2 depth estimation, video propagation, and open-vocabulary inference; visualizations show smoother features and fewer fragmented segmentation artifacts.NAF’s depth maps are smoother in flat regions while retaining sharp object boundaries.Source: §B.2 · p. 14, §B.3 · p. 14

B.4. Generalization results

NAF generalizes across VFM families and scales, with the strongest reported average segmentation gains; its learned representations also support filtering without upsampling. Results cover representation choice and feature inputs at different scaling ratios.

B.4. Generalization results

  • NAF leads average gains by +7.46 mIoU, a +1.23 mIoU gain over the next previous state of the art across randomly paired VFMs and datasets.Increasing input size ×2 yields +4.65 mIoU versus +4.13 mIoU for bilinear on standard images, while ×4 yields a -6.25 mIoU drop.Source: §B.4 · p. 14, Table 9 · p. 15

B.5. Learning representation

  • Training representation affects transfer: smaller or less abstract features often score higher, while raw RGB and multi-VFM training do not improve results.Segmentation visualizations show NAF preserving structure where JAFAR and AnyUp tend to emphasize colors and produce noisy semantics.Source: §B.5 · pp. 15–16, Figure 9 · p. 16

B.6. VFM upsamplers as filters

  • As a feature filter without upsampling, NAF achieves the best average improvement over other filtering approaches, at +0.75 mIoU.The filtered features are subsequently bilinearly upsampled before linear classification.Source: §B.6 · pp. 16–17

B.7. Scaling ratio

  • NAF achieves the best or second-best score across tested scaling ratios and improves mIoU for degraded representations.Larger input images do not always improve scores, including for PE-Core-B.Source: §B.7 · p. 17

C.1. Evaluation Setup

The denoising evaluation tests NAF across Gaussian and channel-wise salt-and-pepper noise levels. It reports high-quality reconstructions, including salt-and-pepper noise beyond the training range.

C.1. Evaluation Setup

  • Denoisers were trained on ImageNet for 25k steps using 448×448 inputs; NAF retained its architecture but increased neighborhood kernel size from 9 to 15.The evaluated denoisers were DNCNN, IRCNN, RedNet, and Restormer; convolutional models used batch size 32 and Restormer batch size 1 on one A100 40GB GPU.Source: §C.1 · pp. 17–18

C.2. Visualizations

  • NAF effectively denoises channel-wise salt-and-pepper noise at intensity 0.6 despite training only up to 0.5, and reconstructs other tested noise levels well.Training ranges were σ ∈[0.1, 0.5] for Gaussian noise and p ∈[0.1, 0.5] for salt-and-pepper noise.Source: §C.2 · p. 18

D. Computation footprint

NAF combines flexible any-scale upsampling with computational efficiency, including at large output resolutions. Its efficiency compares favorably with attention-based alternatives and is comparable to convolutional FeatUp across many configurations.

D. Computation footprint

  • NAF is the only compared approach capable of ×72 upsampling to 2048 × 2048 features.The efficiency study varies embedding dimension at fixed ×16 upsampling and varies upsampling ratio with 384 channels.Source: D. Computation footprint · p. 18
  • NAF provides approximately 2–3× speedup and memory reduction over AnyUp in the embedding study, and an approximately 3× advantage at high upscaling levels.At low upsampling ratios, AnyUp is slightly more efficient; NAF also has substantially higher memory and computational efficiency than JAFAR.Source: D. Computation footprint · p. 18
  • Across many configurations, NAF’s efficiency is comparable to FeatUp while supporting any-scale outputs and consistently outperforming attention-based alternatives.FeatUp is intrinsically constrained to fixed output resolutions.Source: D. Computation footprint · pp. 18–19

E. Limitations and Perspectives

NAF achieves state-of-the-art performance across multiple datasets and tasks, while its design and training-representation selection leave opportunities for improvement. The paper points to medical imaging, remote sensing, and image restoration as application directions.

E. Limitations and Perspectives

  • NAF achieves state-of-the-art performance compared with VFM-specific and VFM-agnostic upsamplers across multiple datasets and tasks.The authors also identify its architecture as adaptable across domains, particularly for denoising.Source: E. Limitations and Perspectives · p. 19
  • NAF uses a fixed neighborhood kernel; dynamic kernel adaptation could potentially reduce per-interpolation cost and improve reconstruction accuracy through flexible sampling.The authors suggest deformable attention or deformable convolutions as possible analogues.Source: E. Limitations and Perspectives · p. 19
  • The authors lack a principled way to identify which VFM representations best support learning the upsampler, and stronger or combined VFMs show no clear gains.They call for deeper understanding of the representations that support consistent zero-shot upsampling.Source: E. Limitations and Perspectives · p. 19
  • Medical imaging and remote sensing are proposed as future directions where preserving high-resolution spatial representations may be valuable.The authors also identify image restoration as a prospective application beyond demonstrated denoising.Source: E. Limitations and Perspectives · p. 19

Related papers

Loading 2511.18452v1…