PaperlayerView exact revision on arXiv

Source-linked AI summary

UPLiFT: Efficient Pixel-Dense Feature Upsampling with Local Attenders

Matthew Walmer, Saksham Suri, Anirud Aggarwal, Abhinav Shrivastava

arXiv:2601.17950v2cs.CV

TL;DR

Dense features from visual backbones are useful but costly because increasing token density can make attention scale quadratically. UPLiFT uses iterative convolutional upsampling and a locally defined Local Attender to produce dense features efficiently, achieving state-of-the-art performance across predictive and generative tasks while lowering inference costs.

  • Problem

    Visual backbones down-sample spatial dimensions, while denser features are often needed and increasing token density makes self-attention computationally expensive.

  • Method

    UPLiFT is an iterative convolutional feature upsampler using a Local Attender that performs attentional pooling through local relative positions instead of Query-Key-Value attention.

  • Results

    UPLiFT achieves state-of-the-art performance across predictive and generative tasks with lower inference costs than comparable methods.

  • Takeaways & Limitations

    UPLiFT provides an efficient approach for creating pixel-dense features from low-resolution visual or VAE features across predictive and generative applications.

Abstract

from arXiv · show

The space of task-agnostic feature upsampling has emerged as a promising area of research to efficiently create denser features from pre-trained visual backbones. These methods act as a shortcut to achieve dense features for a fraction of the cost by learning to map low-resolution features to high-resolution versions. While early works in this space used iterative upsampling approaches, more recent works have switched to cross-attention-based methods, which risk falling into the same efficiency scaling problems of the backbones they are upsampling. In this work, we demonstrate that iterative upsampling methods can still compete with cross-attention-based methods; moreover, they can achieve state-of-the-art performance with lower inference costs. We propose UPLiFT, an architecture for Universal Pixel-dense Lightweight Feature Transforms. We also propose an efficient Local Attender operator to overcome the limitations of prior iterative feature upsampling methods. This operator uses an alternative attentional pooling formulation defined fully locally. We show that our Local Attender allows UPLiFT to maintain stable features throughout upsampling, enabling state-of-the-art performance with lower inference costs than existing pixel-dense feature upsamplers. In addition, we apply UPLiFT to generative downstream tasks and show that it achieves competitive performance with state-of-the-art Coupled Flow Matching models for VAE feature upsampling. Altogether, UPLiFT offers a versatile and efficient approach to creating denser features.

1. Introduction

UPLiFT addresses the cost–density trade-off in visual features with an efficient iterative upsampler and Local Attender. It achieves strong predictive and generative performance while reducing inference costs.

  • Motivation: ViT backbones down-sample spatial dimensions, while denser features improve many applications but increase computational cost through quadratic self-attention.Feature upsampling offers a task-agnostic route from coarse backbone features to high-resolution representations.
  • UPLiFT: UPLiFT combines iterative feature upsampling with a Local Attender that replaces standard Query-Key-Value attention with locally defined relative-position operations.The design targets stable feature semantics without cross-attention’s scaling costs.
  • Results: UPLiFT achieves state-of-the-art performance on several dense predictive tasks with lower inference costs than comparable recent methods.The contribution is positioned as evidence that iterative upsampling remains competitive with cross-attention-based approaches.
  • Applications: UPLiFT applies to predictive and generative tasks including semantic segmentation, monocular depth estimation, image super-resolution, and efficient text-to-image generation.The task set spans both dense prediction and generative applications.

2. Related Works

Prior feature upsamplers progressed from iterative modules to cross-attention resampling, while generative feature upsampling remains less explored. UPLiFT is positioned against these approaches for efficient predictive and generative feature construction.

  • Feature Upsampling: Feature upsampling methods create denser features from pre-trained backbones, offering an alternative to task-specific feature adapters.The broader goal is to retain pretrained visual representations while increasing spatial density.
  • Iterative Methods: FeatUp uses either a fast Joint Bilinear Upsampler with lower quality or per-image Implicit Networks with sharper features and higher inference cost.The two methods expose a speed–quality–cost trade-off.
  • Iterative Methods: LiFT learns 2× upsampled features with self-supervision and can be iteratively applied for larger upsampling.The passage identifies iterative application as LiFT’s route toward pixel-scale density.
  • Cross-Attention Methods: LoftUp, JAFAR, and AnyUp use query-key-value cross-attention to combine high-resolution image queries with low-resolution backbone keys and values.These methods directly support flexible upsampling and help maintain similarity between output and input feature distributions.
  • Efficiency Comparison: AnyUp-v2 uses Neighborhood Attention for linear scaling, whereas UPLiFT uses fixed directional offsets and a high-resolution guide map without QKV attention.UPLiFT is less flexible because it uses fixed-step upsamplers.
  • Generative Tasks: Generative feature upsampling has received less attention than predictive applications, despite the use of VAE features in latent image generation.Upsampled VAE features can be decoded for image super-resolution and efficient text-to-image generation.
  • Generative Tasks: Coupled Flow-Matching is described as the state-of-the-art method for upsampling VAE latent features using a conditioned Flow-Matching model.It predicts high-resolution VAE features from low-resolution VAE features.

3. UPLiFT Approach

UPLiFT combines iterative 2× decoding with a shallow dense encoder and Local Attender to produce pixel-dense features while preserving feature consistency and efficient scaling.

  • Architecture Overview: UPLiFT addresses semantic drift from iterative upsampling with a decoder architecture designed to preserve feature-distribution consistency.The decoder repeatedly applies 2× upsampling, while Local Attender integrates the original backbone feature into the final decoder step.
  • Architecture Overview: A single compact DUPLiFT performs 2× upsampling repeatedly, guided by dense EUPLiFT features computed once from the input image.EUPLiFT uses a shallow, pixel-density-preserving encoder rather than rerunning an image encoder at every upsampling step.
  • Local Attender: Local Attender uses guide features to predict weights over fixed directional neighborhoods and linearly resamples value features.A 1×1 convolution produces one weight per neighborhood offset; weighted offset feature maps are summed to form the output.
  • Local Attender: The operator has O(nT) memory and compute cost, scaling linearly with the number of spatial tokens while avoiding quadratic QKV attention.Here n is the neighborhood size and T is the number of spatial tokens in the guide feature.
  • Local Attender: Local Attender supports upsampling by grouping high-resolution guide tokens into cells corresponding to lower-resolution value tokens.Each guide token attends to a neighborhood around its corresponding value token, preserving a fixed neighborhood size during spatial expansion.
  • UPLiFT Training: Training applies reconstruction losses across intermediate upsampling steps, with shared decoder weights and multiple training depths.The primary configuration uses d ∈ {1, 2, 3}, and intermediate targets are obtained by downsampling the input before comparing features with L2 distance.

4. UPLiFT for Predictive Tasks

UPLiFT achieves state-of-the-art semantic segmentation performance across four datasets while maintaining faster inference than high-performing pixel-dense upsamplers. It also remains competitive for monocular depth estimation despite using only local feature pooling.

  • Segmentation: UPLiFT achieves the highest mIoU and Accuracy scores on all four semantic segmentation datasets, surpassing cross-attention feature-pooling methods.The datasets are COCO-Stuff, Pascal VOC, ADE20k, and Cityscapes.
  • Efficiency: UPLiFT is faster than all high-performing pixel-dense upsamplers, while UPLIFTFAST nearly doubles its inference speed.Nearest, Bilinear, and LiFT-2× are faster but have inferior performance.
  • Depth Estimation: UPLiFT ties AnyUp-v1 for the lowest RMSE in monocular depth estimation and is surpassed only by AnyUp-v1 and AnyUp-v2 on δ1.The evaluation uses thresholded accuracy δ1 and Root Mean Square Error (RMSE) on COCO-Stuff.
  • Depth Estimation: Local-only feature pooling still produces competitive depth-estimation performance, indicating that the upsampled features derive sufficient global information from backbone features.The Local Attender gathers information locally while relying on the backbone features for broader context.
  • Efficiency: Cross-attention-based upsamplers face quadratic time and memory scaling with input-token count, whereas UPLiFT achieves faster inference at 448×448 resolution with 1024 visual tokens.The comparison is measured for patch size 14 on a 448×448 image.

5. UPLiFT for Generative Tasks

UPLiFT is extended to VAE latent features for efficient high-resolution generation and image super-resolution. It delivers strong or competitive quality with substantially lower computational requirements than CFM and only two iterative upsampling steps in super-resolution.

  • Generative Setup: UPLiFT uses a larger model for generative tasks to preserve feature distributions, yet requires only one half or one sixth the parameters of comparable CFM models.The model is trained for 5 epochs on Unsplash-Lite’s 25k images, compared with CFM’s training on over 5 Million Unsplash images.
  • High-Resolution Image Generation: UPLiFT achieves a lower FID than CFM in a single COCO upscaling step while reducing latency by 41%.It also approaches SDXL’s FID while operating at faster speeds than that native 1024×1024 generator.
  • High-Resolution Image Generation: On reLAION, UPLiFT’s 40-step configuration achieves the best CLIP, FID, and patch-FID while maintaining significantly lower latency than CFM.The comparison evaluates zero-shot text-to-image upscaling from 512×512 to 1024×1024.
  • Image Super-Resolution: UPLiFT achieves competitive 4× image super-resolution quality using two iterative steps and an inference time of 271 milliseconds per image.It is 13% slower than bilinear latent-space upsampling while delivering an order-of-magnitude improvement in super-resolution quality.

6. Conclusion

UPLiFT creates high-resolution feature maps from low-resolution backbone features through efficient iterative convolutional upsampling and a Local Attender. It achieves high performance across predictive and generative tasks while reducing inference costs.

  • UPLiFT creates high-resolution feature maps from low-resolution features of pre-trained visual backbones.
  • UPLiFT uses iterative convolutional upsampling and an efficient Local Attender instead of expensive cross-attention.The Local Attender maintains semantic consistency with minimal extra computational cost.
  • UPLiFT achieves state-of-the-art performance across predictive and generative tasks with lower inference costs than comparable methods.
  • The authors position UPLiFT as a way to improve the efficiency and practicality of deep models for dense visual tasks.

A. Additional Details for Predictive Tasks

The evaluation compares UPLiFT with iterative and cross-attention-based task-agnostic feature upsamplers under a common pixel-dense feature setting. The protocol uses DINOv2-S/14 and requires 14× upsampling.

  • The comparison includes FeatUp, LiFT, LoftUp, JAFAR, and AnyUp as task-agnostic feature upsamplers.
  • FeatUp performs 16× iterative upsampling with four modified Joint Bilinear Upsamplers.
  • LiFT is evaluated both as four iterative 2× applications for 16× upsampling and as a single 2× application followed by another method.
  • LoftUp and JAFAR directly upsample backbone features by 14× using cross-attention-based approaches.
  • AnyUp modifies JAFAR with a feature-agnostic layer intended to generalize across backbones at inference time.
  • The main evaluation uses DINOv2-S/14 and produces pixel-dense features requiring 14× upsampling for every method.For 16× methods, features are over-upsampled and then downsampled to pixel resolution.

A.3. Training Cost Comparison

UPLiFT’s training takes longer than some comparison methods but remains feasible on a single moderate-strength GPU. The section also details implementation choices for Local Attender and VAE feature upsampling.

  • Training Cost Comparison: UPLiFT training with DINOv2-S/14 takes ∼9 hours for one epoch on one A5000 GPU.
  • Training Cost Comparison: JAFAR takes ∼3 hours on one A5000, while AnyUp reports ∼5 hours on one H100 GPU.
  • Training Cost Comparison: UPLiFT training takes longer than JAFAR and AnyUp, but all three methods can be trained overnight with one moderate-strength GPU.
  • Local Attender: The Local Attender implementation predicts attention maps with a 1×1 convolution, applies softmax, offsets the Value feature, and combines the results.
  • VAE Feature Upsampling: Small UPLiFT models with roughly 2.8M parameters produce blurry, low-quality VAE upsampling, indicating that generative tasks require greater capacity.
  • Local Attender: Local Attender uses a high-resolution Guide feature to derive attention over a fixed-offset local neighborhood for Value feature upsampling.
  • VAE Feature Upsampling: Removing the Refiner Block produces significant blocky artifacts in upsampled images.
  • VAE Feature Upsampling: VAE upsampling additionally uses noise inputs, latent-space losses, color correction, and Layer Normalization for the larger model.

B.2. Experimental Methods for Generative Tasks

The generative evaluation measures diffusion upsampling and super-resolution using specified datasets, diffusion models, metrics, and latency protocols. It also tests Local Attender neighborhood designs and training-depth configurations.

  • Diffusion Upsampling: Diffusion upsampling is evaluated on COCO 2014 and reLAION-400M using randomly sampled 5k examples from each dataset.
  • Diffusion Upsampling: The experiments prioritize official Stable Diffusion 1.5 latents generated at 512×512 resolution.
  • Diffusion Upsampling: Evaluation reports FID, patch-FID, and CLIP score for diffusion upsampling.
  • Super-Resolution: Super-resolution is evaluated on FacesHQ and LHQ after 1024×1024 center crops are downsampled to 256×256.
  • Latency Timing: Latency is measured on one NVIDIA A100-SXM4-80GB with specified batch sizes, torch.compile, three warm-up batches, and ten averaged batches.
  • Local Attender Ablations: Local Attender experiments test neighborhood sizes and shapes up to 5×5, while an additional ablation removes the module.
  • Training Ablations: Training-depth configurations are evaluated on COCO and VOC segmentation using individual and concurrent depth settings.

C.1. Training Depth

UPLiFT performs best with multi-depth training that combines d∈{1,2,3}; adding d=4 harms performance, while shallower depths work better in single-depth settings.

  • C.1. Training Depth: The ablation evaluates individual and concurrent training depths using semantic segmentation on COCO and VOC with pixel-dense features.The training depth d controls both the downsampling level and the number of upsampling steps.
  • C.1. Training Depth: Single-depth training favors shallower configurations: VOC performs best at d=1, while COCO performs slightly better at d=2.Performance drops slightly at d=3 and substantially at d=4.
  • C.1. Training Depth: Concurrent training with d∈{1,2} improves performance on both COCO and VOC, and adding d=3 enhances it further.The multi-depth configuration d∈{1,2,3} yields the best results.
  • C.1. Training Depth: Adding d=4 to multi-depth training is detrimental because severe input downsampling hinders learning.The final predictive-task configuration uses d∈{1,2,3}.

C.2. Local Attender Neighborhood

The Local Attender is central to UPLiFT’s feature-upsampling quality, with neighborhood expansion improving results and outperforming windowed cross-attention across tested settings and backbones.

  • C.2. Local Attender Neighborhood: Removing the Local Attender significantly harms performance, showing that it is essential for effective multi-step feature upsampling.The ablation compares neighborhood patterns up to a maximum 5×5 size.
  • C.2. Local Attender Neighborhood: Expanding the neighborhood from n=5 to n=9, 13, and 17 progressively improves performance on both evaluated datasets.The n=5 neighborhood contains the current token and immediately touching neighbors.
  • C.2. Local Attender Neighborhood: Larger encoder and decoder modules double parameters and slow inference while providing minimal performance gains, whereas smaller modules underperform.The final design balances performance against computational cost.
  • C.2. Local Attender Neighborhood: UPLiFT with Local Attender outperforms all tested Windowed Cross-Attention variants across both metrics and datasets.WCA was evaluated with 1, 4, 8, and 12 attention heads; 12 heads was detrimental.
  • C.2. Local Attender Neighborhood: UPLiFT improves both metrics on ResNet-50 and ConvNeXt-S compared with bilinear feature upsampling, extending its results beyond ViT backbones.The SD1.5 VAE backbone used elsewhere is also primarily convolution-based.
  • C.2. Local Attender Neighborhood: With DINOv2 and DINOv3 backbones, UPLiFT achieves the best semantic-segmentation performance across both metrics and datasets.No official DINOv3 LoftUp upsampler or training code was available for comparison.
  • C.2. Local Attender Neighborhood: PCA visualizations show UPLiFT and recent methods preserve key-object semantics, while LiFT exhibits visual degradation and LoftUp removes some background artifacts.The practical benefit of artifact removal is uncertain because it shifts features from the original backbone distribution.
  • C.2. Local Attender Neighborhood: UPLiFT maintains linear inference-time and memory scaling, while several alternatives show quadratic scaling; UPLIFTFAST extends the maximum processing size further.On H100, UPLIFTFAST processes resolutions over 3500×3500 and nearly 50000 visual tokens.

D.5. Semantic Stability with UPLiFT

UPLiFT preserves semantic content through iterative upsampling, whereas LiFT’s intermediate features become increasingly degraded and distorted. Its stable features also scale linearly in time and memory.

  • D.5. Semantic Stability with UPLiFT: LiFT’s iterative upsampling causes intermediate features to become increasingly faded, murky, and distorted.Local regions with consistent semantic content do not retain similar features across steps.
  • D.5. Semantic Stability with UPLiFT: UPLiFT maintains linear time and memory scaling as visual token counts increase.The scaling comparison covers NVIDIA A5000 and H100 GPUs.
  • D.5. Semantic Stability with UPLiFT: UPLiFT’s Local Attender maintains local object-region features across all upsampling stages, unlike LiFT’s semantic drift.Consistent coloration in the PCA visualization indicates consistent features.
  • D.5. Semantic Stability with UPLiFT: UPLiFT maintains consistent features across upsampling steps, with sharper object edges that match the original image.The comparison uses joint PCA over backbone features and all intermediate upsampling steps.

D.6. Additional Visualization for Generative Tasks

UPLiFT provides efficient latent-feature upsampling for generative applications, producing higher-fidelity super-resolution images with modest or low end-to-end latency.

  • D.6. Additional Visualization for Generative Tasks: UPLiFT is only 8.47% slower than latent-space bilinear upsampling for 4× super-resolution while producing substantially better visual fidelity.The comparison upsamples 512×512 inputs to 2048×2048 outputs.
  • D.6. Additional Visualization for Generative Tasks: UPLiFT generates 512×512-to-2048×2048 outputs from Stable Diffusion 1.5 latents using a 4× configuration.The latents were generated with 50 diffusion steps.
  • D.6. Additional Visualization for Generative Tasks: For 2× upsampling from 512×512 to 1024×1024, end-to-end latency is 2.75 seconds on an NVIDIA A100, with UPLiFT itself taking 104 milliseconds.The model is applied to Stable Diffusion 1.5 VAE latents generated with 50 diffusion steps.
  • D.6. Additional Visualization for Generative Tasks: For FacesHQ, UPLiFT upsamples 256×256 images to 1024×1024 in 270.9 milliseconds end-to-end on an NVIDIA A100.The VAE UPLiFT model is trained only in latent space and is not fine-tuned for super-resolution.
  • D.6. Additional Visualization for Generative Tasks: On the more diverse LHQ dataset, a generalist UPLiFT model still shows good performance without dataset-specific fine-tuning.The comparison notes that another method uses a fine-tuned model with three times as many parameters.
Loading 2601.17950v2…