PaperlayerView exact revision on arXiv

Source-linked AI summary

JAFAR: Jack up Any Feature at Any Resolution

Paul Couairon, Loick Chambon, Louis Serrano, Jean-Emmanuel Haugeard, Matthieu Cord, Nicolas Thome

arXiv:2506.11136v3cs.CVeess.IV

TL;DR

Foundation encoders downsample spatial information by 14× to 16×, creating a bottleneck for downstream tasks requiring pixel-level accuracy. JAFAR is a lightweight attention-based upsampler that supports arbitrary resolutions by aligning high-resolution image-derived queries with semantically enriched encoder-feature keys. JAFAR consistently outperforms existing upsampling methods across diverse downstream tasks as a drop-in module.

  • Problem

    Foundation encoders downsample spatial information by 14× to 16×, creating a bottleneck for downstream tasks requiring pixel-level accuracy.

  • Method

    JAFAR is a lightweight attention-based upsampler that supports arbitrary resolutions by aligning high-resolution image-derived queries with semantically enriched encoder-feature keys.

  • Results

    JAFAR consistently outperforms existing upsampling methods across diverse downstream tasks as a drop-in module.

  • Takeaways & Limitations

    Low-resolution, low-factor training generalizes to significantly higher output scales without high-resolution supervision.

  • Takeaways & Limitations

    Global attention can become computationally and memory prohibitive with large key sets, such as 64 × 64 or higher.

Abstract

from arXiv · show

Foundation Vision Encoders have become essential for a wide range of dense vision tasks. However, their low-resolution spatial feature outputs necessitate feature upsampling to produce the high-resolution modalities required for downstream tasks. In this work, we introduce JAFAR, a lightweight and flexible feature upsampler that enhances the spatial resolution of visual features from any Foundation Vision Encoder to an arbitrary target resolution. JAFAR employs an attention-based module designed to promote semantic alignment between high-resolution queries, derived from low-level image features, and semantically enriched low-resolution keys, using Spatial Feature Transform (SFT) modulation. Notably, despite the absence of high-resolution supervision, we demonstrate that learning at low upsampling ratios and resolutions generalizes remarkably well to significantly higher output scales. Extensive experiments show that JAFAR effectively recovers fine-grained spatial details and consistently outperforms existing feature upsampling methods across a diverse set of downstream tasks. Project page at https://jafar-upsampler.github.io

1 Introduction

Foundation vision encoders provide semantically rich but spatially coarse features, creating a bottleneck for dense prediction. JAFAR addresses this with a lightweight, task-agnostic upsampler supporting arbitrary resolutions and outperforming existing methods across downstream tasks.

  • Foundation encoders downsample spatial information by 14× to 16×, producing coarse features that limit pixel-level accuracy.
  • JAFAR targets task-agnostic training, arbitrary output resolutions, compatibility with any vision encoder, and minimal inference overhead.
  • Training at low resolutions and low upsampling ratios generalizes robustly to significantly higher output scales without high-resolution supervision.The training objective requires no high-resolution supervision signal.
  • JAFAR uses high-resolution low-level queries and semantically enriched low-resolution keys to align spatial detail with semantic context.
  • JAFAR consistently outperforms existing upsampling methods across diverse downstream tasks when used as a drop-in module.

2 Related Work

Feature upsampling methods range from simple interpolation to learned modules and feature modulation, trading computational efficiency, scalability, and output quality.

  • Feature Upsampling: Traditional interpolation methods provide simple, efficient baselines but do not adapt to the underlying content of feature maps.
  • Architectural Design for Upsampling Modules: LiFT uses a lightweight CNN for fixed-factor upsampling, while FeatUp provides fast JBU and more accurate implicit-network variants for continuous resolution prediction.
  • Semantic Guidance and Feature Modulation: Feature modulation methods use conditioning-derived scale and shift parameters to guide spatially or semantically conditioned transformations.

3 JAFAR

JAFAR reconstructs high-resolution features through cross-attention between image-guided queries and semantically modulated keys. Its annotation-free low-resolution training scheme supports generalization to much larger output resolutions.

  • 3 JAFAR: JAFAR uses cross-attention to interpolate low-resolution encoder features into arbitrary target resolutions using high-resolution image guidance.Queries retain low-level color and texture, while keys combine semantic and spatial information.
  • 3.1 Architecture: The attention kernel is built from shared image representations, with SFT-enriched keys promoting semantic alignment before feature interpolation.
  • 3.1 Architecture: Queries are pooled to the target resolution, while keys are downsampled to the encoder-feature resolution and enriched with semantic modulation.
  • 3.2 Training Pipeline: JAFAR trains from multi-resolution views of the same image using an alignment loss combining cosine similarity and L2 distance.
  • 3.2 Training Pipeline: Although trained only with moderate upsampling factors up to 4×, JAFAR generalizes to much higher resolutions without ground-truth high-resolution features.
  • Comparison with LiFT: Unlike LiFT’s fixed 2× design, JAFAR maintains a resolution-agnostic design that avoids reliance on iterative upsampling or bilinear fallback.

4 Experiments

Across qualitative comparisons, dense prediction, explainability, open-vocabulary segmentation, and BEV segmentation, JAFAR produces sharper features and generally outperforms existing upsampling methods. Ablations attribute these gains to SFT-based key modulation and a moderate number of attention heads.

  • Qualitative Comparisons: JAFAR produces sharp feature maps that capture image structure and suppress positional-embedding artifacts better than competing upsamplers.The qualitative evaluation uses a shared three-dimensional PCA basis; JAFAR is sharper and more coherent than baselines while avoiding their blur, noise, or artifacts.
  • Downstream Task Performance: JAFAR consistently outperforms other baselines across segmentation benchmarks and improves downstream performance across diverse tasks.It achieves the highest performance on all four semantic segmentation benchmarks and improves vehicle-IoU in complex BEV architectures.
  • Depth Estimation: JAFAR reaches competitive depth-estimation scores without task-specific training, ranking second among baselines and surpassing FeatUp, LiFT, and all task-dependent methods except ReSFU.Depth evaluation uses RMSE and δ1 < 1.25 with pseudo-labels generated by Depth Anything V2.
  • Zero-Shot Open-Vocabulary Segmentation: JAFAR improves zero-shot open-vocabulary segmentation, achieving the highest performance on both mIoU and accuracy even on the 150-class ADE20K benchmark.The setup is training-free with CLIP-ViT-B/16, and JAFAR significantly outperforms all baselines, including FeatUp among task-agnostic methods.
  • Ablations: SFT-based key modulation outperforms concatenation, while performance improves through four attention heads before declining as per-head dimensionality becomes too small.The ablation compares linear projection, concatenation, and SFT strategies for constructing keys.

5 Conclusion

JAFAR is a lightweight, attention-based feature upsampler that works with any foundation vision encoder and arbitrary output resolutions without original-size supervision or downstream annotations. It outperforms prior state-of-the-art upsamplers across diverse downstream tasks, while remaining limited by backbone-specific training and feature-level artifacts.

  • JAFAR upscales features from any foundation vision encoder to arbitrary output resolutions without supervision at the original image size or downstream-task annotations.The method uses a simple, task-agnostic training objective.
  • JAFAR outperforms prior state-of-the-art upsamplers across a variety of downstream tasks despite not being trained specifically for those tasks.
  • JAFAR lays groundwork for a unified feature upsampler that could enable more efficient architectures for dense vision tasks.
  • The current method requires training a separate upsampler for each backbone, while future work targets backbone-independent inference and fewer feature-level artifacts for sharper outputs.

A.1 Evaluation

JAFAR is evaluated through attribution metrics, visualizations, and downstream transfer tasks, where it produces sharper, more coherent, and more accurate representations than alternative upsamplers.

  • A.1 Evaluation: ADCC combines coherency, low complexity, and low Average Drop into one score for direct comparison of attribution quality.Coherency measures stability under explanation-guided masking, while Complexity uses the ℓ0 norm to count activated pixels.
  • JAFAR produces sharper, content-aware feature maps with fewer artifacts than training-free, task-dependent, and task-agnostic baselines.The comparison uses 32 × 32 DINOv2-S/14 features upsampled to 448 × 448.
  • JAFAR yields sharper and more accurate Grad-CAM explanations with fewer artifacts than alternative upsampling methods.The evaluation upsamples ViT-B/16 features from 14 × 14 to 224 × 224 before extracting Grad-CAMs.
  • JAFAR produces more coherent semantic segmentation, with improved delineation of object boundaries and background regions.The visualization uses DINOv2-S/14 features upsampled from 32 × 32 to 448 × 448 on COCO-Stuff.
  • B.5 Attention Maps Visualization: Attention concentrates on semantically related low-resolution keys, while broader receptive fields help counter spatial misalignments and positional artifacts.The authors suggest localized variants could reduce runtime and memory, but caution against making attention overly local.

C Additionnal Comparisons With Task-Agnostic Baselines

JAFAR is compared with task-agnostic baselines FeatUp and LiFT for semantic segmentation using their respective feature extractors and backbone checkpoints. Across datasets and output resolutions, JAFAR generally improves performance, with one stated exception for FeatUp on ADE20K at 448.

  • C Additionnal Comparisons With Task-Agnostic Baselines: The reported FeatUp and LiFT results were trained in the authors’ codebase, with additional official-checkpoint comparisons provided separately.The official-checkpoint evaluations use FeatUp on DINOv2 ViT-S/14 and LiFT on DINO ViT-S/16.
  • C Additionnal Comparisons With Task-Agnostic Baselines: JAFAR consistently improves semantic segmentation performance across datasets and output resolutions, except on ADE20K at 448 for FeatUp.The comparison includes evaluations against FeatUp and LiFT under their respective feature-extractor and backbone settings.
  • C Additionnal Comparisons With Task-Agnostic Baselines: The evaluation compares JAFAR with FeatUp using FeatUp’s original feature extractor and the official JBU checkpoint on DINOv2 ViT-S/14.
  • C Additionnal Comparisons With Task-Agnostic Baselines: The evaluation compares JAFAR with LiFT using LiFT’s original feature extractor and its checkpoint on DINO ViT-S/16.

C.2 LoftUp

Against LoftUp, JAFAR performs better at lower upsampling resolutions and comparably at higher resolutions, while using a substantially smaller single-stage design without external annotations. LoftUp’s self-distillation is complementary and can be integrated into JAFAR.

  • C.2 LoftUp: JAFAR outperforms LoftUp at resolutions 56 and 112 and delivers comparable performance at resolutions 224 and 448.The comparison uses LoftUp’s official codebase and a DINOv2 ViT-S/14 backbone.
  • C.2 LoftUp: JAFAR uses a simpler single-stage strategy that operates entirely at low resolution and does not rely on external annotations.LoftUp uses segmentation masks from SAM during training and a two-stage pipeline with self-distillation.
  • C.2 LoftUp: LoftUp’s self-distillation mechanism is complementary to JAFAR and can be integrated into its pipeline, producing a clear boost at resolutions 224 and 448.The reported gains are minimal at lower resolutions.
  • C.2 LoftUp: JAFAR has 700K parameters compared with LoftUp’s 4.3M parameters.

D Additional Analysis

Additional analysis finds that JAFAR maintains strong segmentation performance across increasingly extreme upsampling ratios, while RoPE becomes increasingly important for spatial alignment at higher resolutions.

  • D Additional Analysis: JAFAR sustains strong linear-probing segmentation performance as the upsampling ratio grows, including the extreme 82 → 8962 range.This supports generalization beyond the training range.
  • D Additional Analysis: RoPE plays a critical role in maintaining consistent spatial alignment between queries and keys across varying resolutions.Its benefits become more pronounced at higher resolutions.

E Performance

The performance analysis compares runtime, memory, and computational cost across resolutions and reports that JAFAR improves BeV segmentation accuracy with a modest computational overhead.

  • E Performance: Runtime and memory usage are compared across methods and target resolutions using batch size 1, input resolution 448, and a single A100 GPU.
  • E Performance: JAFAR delivers 6%−15% higher IoU in BeV segmentation while requiring only 3%−10% more GFLOPs than the version without upsampling.The comparison highlights an accuracy–computation trade-off.

F Limitations

JAFAR’s global attention provides a large receptive field, but its cost grows with the number of low-resolution keys and can become prohibitive at high input resolutions.

  • F Limitations: Global attention becomes computationally and memory intensive as the number of keys grows, especially for key sets of 64 × 64 or higher.Each query attends to every key, so larger input resolutions increase the key set and associated costs.
  • F Limitations: Localized attention is identified as a promising direction for improving efficiency while retaining a large receptive field.The paper notes that attending to every key is not strictly necessary and cites localized alternatives as potentially offering orders-of-magnitude efficiency improvements.
  • F Limitations: The limitation reflects a trade-off between JAFAR’s desirable large receptive field and the prohibitive cost of attending to all keys.
Loading 2506.11136v3…