PaperlayerView exact revision on arXiv

Source-linked AI summary

RaysUp: Ultra-light Universal Feature Upsampling via Geometry-Aware Ray Representation

Yuchuan Ding, Linfei Li, Lin Zhang, Ying Shen

arXiv:2606.22749v1cs.CV

TL;DR

Vision Foundation Models produce low-resolution features, while existing upsampling methods can lose semantic information or require retraining and inference optimization. RaysUp reconstructs features at arbitrary resolutions using geometry-aware ray representations, achieving superior dense-prediction performance with approximately 16% of AnyUp’s parameters and 7× faster inference.

  • Problem

    VFM outputs are often too spatially coarse for pixel-level tasks, while existing upsamplers may lose semantic information or require VFM-specific retraining or inference optimization.

  • Method

    RaysUp reconstructs arbitrary-resolution feature maps using direction-aware guidance, resolution-flexible cross-attention, and RayPE-based geometry-aware neighborhood aggregation.

  • Results

    RaysUp achieves superior performance across multiple dense prediction tasks using approximately 16% of AnyUp’s parameters and approximately 7× faster inference.

  • Takeaways & Limitations

    RaysUp offers a practical accuracy-efficiency trade-off for task-agnostic and VFM-agnostic feature upsampling.

  • Takeaways & Limitations

    Experiments use DINOv2-S as the default backbone for upsampler training and downstream feature extraction.

Abstract

from arXiv · show

Pre-trained Vision Foundation Models (VFMs) have become central to modern computer vision due to their powerful semantic representations and strong generalization ability. However, their patchified or pooled outputs are inherently low-resolution, limiting their effectiveness in tasks requiring fine-grained, pixel-level reasoning. Existing feature upsampling approaches either degrade semantic fidelity or rely on VFM-specific retraining and heavy architectures, hindering efficiency and scalability. To address these challenges, we propose RaysUp, an ultra-lightweight, task-agnostic, and VFM-agnostic feature upsampling framework that reconstructs high-resolution feature maps at arbitrary resolutions. Unlike conventional 2D interpolation or attention-based schemes, RaysUp lifts feature reconstruction into a geometry-aware ray domain. Specifically, we introduce a Spatially Decoupled Guidance Encoder for direction-aware guidance encoding, an Any-Resolution Cross-Attention mechanism for resolution-flexible reconstruction, and a novel Ray Positional Encoding (RayPE) that injects implicit 3D geometric priors via 6D Plucker ray coordinates. Finally, a Geometry-Aware Neighborhood Attention module further ensures content-adaptive bilateral aggregation while preserving geometric consistency. Extensive experiments across diverse dense prediction tasks demonstrate that RaysUp achieves state-of-the-art performance while using only 16% of the parameters of AnyUp and delivering approximately 7x faster inference. These results highlight a substantially improved accuracy-efficiency trade-off and establish RaysUp as a practical and scalable solution for universal feature upsampling. Code is available at https://github.com/MAP-RaysUp/RaysUp.

1 Introduction

RaysUp is an ultra-lightweight, task-agnostic, and VFM-agnostic framework that reconstructs foundation-model features at arbitrary resolutions. It combines geometry-aware ray representations with adaptive guidance and attention to preserve semantic and geometric fidelity efficiently.

  • Framework: RaysUp reconstructs features from arbitrary Vision Foundation Models at any target resolution through an ultra-lightweight, task-agnostic, and encoder-agnostic design.The framework is intended for universal feature upsampling across multiple dense prediction tasks.
  • Adaptive Guidance: A Spatially Decoupled Guidance Encoder provides direction-aware, multi-scale spatial semantics, while Any-Resolution Cross-Attention adaptively reconstructs features at arbitrary output resolutions.The guidance features provide more informative structural priors than pixel-level color differences.
  • Geometry-Aware Reconstruction: Ray Positional Encoding injects implicit 3D geometric priors through 6D Plücker ray coordinates, extending joint bilateral upsampling into geometry-aware reconstruction.This design enhances boundary fidelity and mitigates structural drift.
  • Adaptive Aggregation: Geometry-Aware Neighborhood Attention performs localized, content-adaptive bilateral aggregation between high-resolution guidance and low-resolution features while preserving geometric consistency and locality.It balances geometric fidelity, semantic consistency, and computational efficiency.
  • Results: RaysUp achieves superior dense-prediction performance to AnyUp using approximately 16% of its parameters and delivering approximately 7× faster inference.These gains substantially reduce computational costs while improving the accuracy-efficiency trade-off.

2 Related Work

Feature upsampling for dense prediction is organized into discrete reconstruction and continuous mapping paradigms, each offering scale flexibility but facing limitations in adaptivity, detail preservation, or encoder generalization. RaysUp reformulates joint bilateral upsampling with lightweight, geometry-aware feature interaction, supporting arbitrary-resolution universal upsampling with 0.14M parameters and approximately 7× faster inference than AnyUp.

  • Related Work: Feature upsampling methods comprise discrete reconstruction and continuous mapping paradigms for recovering high-resolution representations in dense prediction tasks such as segmentation and depth estimation.Discrete methods explicitly reconstruct features through interpolation, filtering, or convolutional decoding, whereas continuous methods formulate resolution-agnostic function approximation or cross-scale interaction.
  • Discrete Reconstruction and Continuous Mapping: Classical discrete methods are efficient and scale-flexible but lack content adaptivity, causing boundary blurring and loss of fine details.Continuous approaches improve expressiveness and scale generalization, yet typically require retraining for specific visual encoders.
  • RaysUp: RaysUp revisits joint bilateral upsampling by replacing fixed 2D neighborhoods and RGB-based range modeling with a spatially decoupled guidance encoder and attention-based cross-scale interaction.This unifies filter-style local aggregation with adaptive feature interaction while improving structural fidelity and semantic expression.
  • RaysUp: 0.14M parameters enable RaysUp to integrate with arbitrary vision encoders and perform universal feature upsampling at arbitrary resolutions.The lightweight architecture follows the attention-based universal-upsampling trajectory while avoiding encoder-specific integration constraints.
  • RaysUp: Approximately 7× faster inference than AnyUp accompanies substantially reduced memory consumption in RaysUp.The comparison is reported for the lightweight universal upsampling architecture.

3 RaysUp

RaysUp reconstructs arbitrary-resolution high-resolution features from VFM outputs by combining direction-aware guidance, resolution-decoupled cross-attention, and geometry-aware ray representations. RayPE and neighborhood aggregation align feature transfer with shared 3D ray geometry while reducing attention complexity.

  • Spatially Decoupled Guidance Encoder: The spatially decoupled guidance encoder models orthogonal directional components in parallel, concatenating branch features to produce direction-aware guidance representations.Its branches divide Dg evenly and use directional convolutions with shallow residual structures.
  • Efficiency: 69.4% lower parameter overhead reduces the guidance encoder from 27Dg parameters for a standard 3 × 3 convolution to 8.25Dg, while local neighborhood aggregation reduces attention complexity to O(HanyWany· k2).The latter replaces global attention complexity O(HanyWany·HlrWlr) while preserving geometric consistency under a local planar approximation.
  • Any-Resolution Cross-Attention: High-resolution guidance queries and low-resolution keys are adaptively pooled from the guidance encoder, while VFM features serve directly as values for flexible cross-resolution propagation.This enables arbitrary target resolutions without explicit interpolation.
  • Ray-Based Reconstruction: RaysUp lifts feature reconstruction from the 2D image grid into a projective ray domain, transporting features from coarse backbone rays to densely sampled target-resolution rays.This addresses perspective cases where image-space proximity does not guarantee 3D geometric consistency.
  • Ray Positional Encoding: RayPE encodes 6D ray descriptors derived from camera geometry and multi-scale harmonic frequencies, extending RoPE to align cross-resolution features on a shared 3D ray manifold.The representation is resolution-independent and explicitly binds aggregation to 3D ray geometry rather than pixel-plane proximity.

4 Experiments

Across dense prediction benchmarks, RaysUp delivers consistently strong task-agnostic and VFM-agnostic performance while preserving geometric consistency. Its lightweight design and component ablations show that decoupled guidance encoding and RayPE are central to its accuracy-efficiency trade-off.

  • Experimental Setup: RaysUp uses ImageNet training with AdamW for 100,000 iterations and evaluates semantic, geometric, video-object, and open-vocabulary prediction using established datasets and metrics.DINOv2-S is the default backbone; semantic evaluation uses mIoU and pixel accuracy, while geometric evaluation includes RMSE, angular-threshold accuracy, and δ1.
  • Task-agnostic Performance: RaysUp achieves the best or second-best performance across most task-agnostic metrics, with particularly strong surface-normal and depth-estimation results.Qualitative results show better geometric consistency than AnyUp, JAFAR, FeatUp, and LoftUp, which exhibit holes or blurred boundaries.
  • VFM-agnostic Performance: RaysUp consistently outperforms AnyUp across tested VFMs—DINOv2, DINOv3 [47], SigLIP2 [51], and PE Spatial [5]—and ViT-S, ViT-M, and ViT-L scales.AnyUp is the only baseline with VFM-agnostic capability and is therefore the primary comparison.
  • Ultra-lightweight Performance: 0.14M parameters, 10.17 GFLOPs, 1.26GB GPU memory, and 55 FPS at 224×224 demonstrate RaysUp’s computational advantage over existing upsampling methods.The comparison covers parameters, GFLOPs, GPU memory, and inference FPS across multiple upsampling resolutions.
  • Effectiveness of Spatially Decoupled Guidance Encoder: 82.17% average performance with 0.14M parameters makes the Decoupled-Branch guidance encoder superior to Single-Branch, Dual-Branch, and Multi-Branch designs across four VFMs.The result supports decoupled modeling of spatial guidance features for multi-scale spatial-semantic information and lightweight design.
  • Effectiveness of RayPE: RayPE achieves the highest performance across all four VFMs without additional parameters, whereas omitting positional encoding reduces average performance to 81.05%.RayPE injects implicit 3D geometric priors; DA3 pose information can help in some cases, but Identity pose is adopted for efficiency.

5 Conclusion

RaysUp is an ultra-lightweight, task-agnostic, and encoder-agnostic framework for reconstructing backbone features at arbitrary resolutions. It combines direction-aware guidance, resolution-flexible cross-attention, and RayPE to inject implicit 3D geometric priors into high-resolution feature reconstruction.

  • 5 Conclusion: RaysUp reconstructs backbone features at arbitrary resolutions through a universal feature upsampling framework.The framework is described as ultra-lightweight, task-agnostic, and encoder-agnostic.
  • 5 Conclusion: Its design integrates a Spatially Decoupled Guidance Encoder, Any-Resolution Cross-Attention, and Ray Positional Encoding.These components provide guidance encoding, resolution-flexible reconstruction, and geometric representation, respectively.
  • 5 Conclusion: RayPE injects implicit 3D geometric priors to support geometry-aware high-resolution feature reconstruction.The conclusion identifies geometric priors as central to the reconstruction strategy.

— Supplementary Material — · A Motivation of Spatially Decoupled Guidance Encoder · B Implicit Geometric Injection in RayPE

The supplementary material motivates spatially decoupled guidance by diagnosing discontinuous feature behavior in conventional convolutions and presents RayPE as a geometry-aware positional encoding for scale-consistent upsampling. Together, these designs improve spatial continuity and replace planar pixel-distance reasoning with angular consistency on normalized camera rays.

  • A Motivation of Spatially Decoupled Guidance Encoder: The Spatially Decoupled Guidance Encoder raises central weights to 1.00, enabling channel interactions analogous to a 1×1 convolution.Independent horizontal 1×3, vertical 3×1, and corner branches preserve peripheral spatial sensitivity.
  • A Motivation of Spatially Decoupled Guidance Encoder: Its branch-wise design maintains peripheral weights of 0.89−0.99 while improving spatial consistency and semantic continuity during cross-attention.The encoder combines strong central mixing with directional and corner-sensitive processing rather than relying on one conventional kernel.
  • A Motivation of Spatially Decoupled Guidance Encoder: Conventional 3×3 kernels can assign lower weights to centers and higher weights to corners and edges, potentially creating holes and discontinuities in upsampled features.This behavior is motivated by visualized JAFAR kernels and the structural sparsity patterns reported for ACNet and RepVGG.
  • B Implicit Geometric Injection in RayPE: RoPE defines positional phase on a planar isotropic 2D Euclidean grid using pixel coordinates (i, j), whereas RayPE uses normalized camera rays to encode 3D spatial position.For RayPE, the phase depends on the focal length f and the normalized vector [i,j,f]^T / sqrt(i^2+j^2+f^2), assuming identity camera extrinsics.
  • B Implicit Geometric Injection in RayPE: RayPE avoids the additional alignment interpolation required by RoPE as coordinates shift during upsampling, preserving geometric feature consistency under identical viewing directions.This gives the representation natural scale equivariance under the stated viewing-direction condition.
  • B Implicit Geometric Injection in RayPE: By extending positional encoding from a 2D image grid to a unit viewing sphere, RayPE enables attention to model angular consistency rather than pixel distance.The geometric construction supplies an implicit 3D prior through normalized viewing directions.

C Further Implementation Details · C.1 Hyperparameter Settings

The implementation uses fixed guidance and RayPE hyperparameters, including a 256-dimensional guidance feature space and five frequency bands with distinct wavelength ranges for ray origins and directions.

  • C.1 Hyperparameter Settings: Dg = 256 and N = 5 set the guidance-feature dimensionality and total RayPE frequency bands.These values define the feature width and positional-encoding frequency capacity.
  • C.1 Hyperparameter Settings: Ray-origin wavelengths span λmax = 4.0 to λmin = 4.0/81, determining the lowest and highest angular frequencies.The wavelength range controls the angular-frequency bounds for ray-origin encoding.
  • C.1 Hyperparameter Settings: Ray-direction wavelengths span λmax = 8.0 to λmin = 8.0/81.The direction encoding uses a separate wavelength range from the ray-origin encoding.

C.2 Task Settings … C.3 Baselines

The experiments specify task-specific evaluation protocols for semantic segmentation, depth and surface normal estimation, video object segmentation, and open-vocabulary segmentation, while comparing against established feature upsamplers with differing resolution and training constraints.

  • C.2 Task Settings: Semantic segmentation retained pre-trained parameters, trained linear classifiers on extracted features, and used 448 × 448 inputs and targets with AdamW at 5×10−4 under cosine annealing.Optimization used cross-entropy loss across all datasets.
  • C.2 Task Settings: Depth and surface normal estimation used NYUv2, with AdaBins configured for 256 bins over 0–10 meters and Bae et al. ’s robust loss for normals.Depth optimization combined scale-invariant sigmoid and gradient-matching losses; normals predicted orthogonal unit vectors plus uncertainty.
  • Video object segmentation.: Video object segmentation was evaluated on DAVIS 2017 validation [40] at 480p using 2× upsampled features and local label propagation from the initial and seven preceding frames.Feature matching was restricted to a local spatial neighborhood of 12 pixels.
  • Open-Vocabulary Segmentation.: Open-vocabulary segmentation followed ProxyCLIP but replaced bilinear interpolation with RaysUp at 4× resolution before cosine-similarity classification in a shared vision-language space.Patch labels were reshaped and upsampled to form the final image segmentation map.
  • C.3 Baselines: FeatUp was evaluated only with its pretrained feedforward JBU variant because official weights for the implicit MLP were unavailable, limiting enhancement to fixed integer scales.Both variants were originally supervised with multi-view consistency loss.
  • C.3 Baselines: LoftUp used coordinate-based cross-attention with high-resolution image queries and coordinates, low-resolution VFM keys and values, and training based on SAM-generated masks followed by EMA-teacher self-distillation.The method supported only fixed-scale upsampling matching the original input image.
  • C.3 Baselines: JAFAR restored features to arbitrary resolutions through cross-attention using guidance-derived asymmetric queries and keys, with SFT modulation and cosine-plus-L2 training losses.Its training used multi-resolution image views at low upsampling factors.
  • C.3 Baselines: AnyUp used channel-canonicalizing feature-agnostic convolutions and feature-size-relative local-window attention, trained with crops and evaluated using officially released pretrained weights.Its crop-based training procedure required approximately 5 hours.

D Additional Ablation Study … Image Pose.

The ablations examine guidance-encoder structure, feature dimensionality, positional-encoding implementations, and identity-pose training conditions. They identify 256-dimensional guidance features as preferable to the tested alternatives and document the evaluated architectural and training setups.

  • Guidance Encoder.: The guidance-encoder ablation compares single-, dual-, and multi-branch convolutional designs, using 1×1, parallel 1×1/3×3, or four 1×1/1×3/3×1/3×3 branches.Outputs from parallel branches are concatenated channel-wise, and the single-branch design follows AnyUp.
  • Guidance Feature Dimensionality.: The 256-dimensional guidance representation consistently and substantially outperformed the 128-dimensional baseline on downstream tasks.The passage identifies 256 dimensions as sufficient relative to the tested alternatives.
  • Guidance Feature Dimensionality.: The 768-dimensional setting underperformed the 512-dimensional configuration, while neither notably improved over the 256-dimensional configuration.These comparisons were reported in downstream-task ablations.
  • Positional Encoding.: RoPE was implemented according to protocols established by JAFAR and AnyUp in the positional-encoding ablation.The implementation follows the cited prior protocols rather than introducing a separate procedure in this passage.
  • Positional Encoding.: For SinRays, ray origins and directions were concatenated with image-encoder features, sine-activated, and projected to the target hidden dimensionality with a convolutional layer.The sine activation was used to capture high-frequency details.
  • Image Pose.: Under the identity-pose setting, the extrinsic matrix was identity and intrinsics followed a simplified pinhole camera model.This defines the camera configuration used for the identity-pose experiments.
  • Image Pose.: Training DA3-Small, DA3-Base, and DA3-Large took approximately 5, 9, and 55 hours, respectively.The reported durations correspond to the identity-pose setting.

D.2 Ablation Study on Training Strategy · D.3 Upsampling from Any to Any Resolution.

RaysUp uses a lightweight local-crop training strategy with frozen encoder supervision and a joint cosine-L2 reconstruction loss. It generalizes across arbitrary input-target resolution pairs, achieving the best or second-best performance across Pascal-VOC evaluations while avoiding high-resolution memory failures.

  • D.2 Ablation Study on Training Strategy: RaysUp followed existing feature upsampling methods by training with lightweight randomly sampled local image crops.The ablation experiments evaluated this training strategy on semantic segmentation tasks.
  • D.3 Upsampling from Any to Any Resolution.: RaysUp achieved the best or second-best performance across a wide range of arbitrary input-target resolution pairs on Pascal-VOC.The comparison appears in Table 6; baselines were aligned to target resolutions using bilinear interpolation where necessary.
  • D.2 Ablation Study on Training Strategy: Training pairs a downsampled full image with a same-sized random high-resolution crop, extracting source and supervisory target features using a frozen vision encoder.RaysUp reconstructs the full high-resolution feature map from the low-resolution source features, then crops the corresponding aligned subregion.
  • D.2 Ablation Study on Training Strategy: The reconstruction loss jointly measures cosine dissimilarity and L2 distance between the aligned reconstructed crop and target features.Approximately 4 hours of training produced consistent performance improvements in Table 5, supporting the strategy’s effectiveness and scalability.
  • D.3 Upsampling from Any to Any Resolution.: For the any-to-any evaluation, FeatUp’s JBU variant was constrained to fixed 16× upsampling, while LoftUp only supported the original image resolution.Bilinear interpolation was applied to align both methods’ outputs with the target resolutions.
  • D.3 Upsampling from Any to Any Resolution.: JAFAR was competitive mainly at lower resolutions, but encountered out-of-memory failures at higher resolutions and could not be evaluated at the largest scales.The reported lower-resolution examples were 16 →112 and 16 →448.

E Further Qualitative Experiments

Additional qualitative experiments show that RaysUp preserves feature structure during upsampling and produces sharper, more coherent predictions across depth estimation, surface normal estimation, semantic segmentation, and video segmentation.

  • E Further Qualitative Experiments: Only RaysUp, AnyUp, and JAFAR produce distinctly sharp PCA projections while maintaining the underlying feature space during ImageNet upsampling.All methods use a uniform PCA basis for consistent visualization.
  • E Further Qualitative Experiments: On NYUv2 depth estimation, RaysUp preserves sharper object boundaries and produces predictions more closely aligned with ground truth than baseline methods.
  • E Further Qualitative Experiments: On NYUv2 surface normal estimation, RaysUp can produce more coherent predictions than ground truth, while competing methods introduce artifacts in challenging regions.This behavior is illustrated in rows 2 and 5, whereas artifacts appear in row 6 for competing methods.
  • E Further Qualitative Experiments: On VOC semantic segmentation, RaysUp produces more complete masks and sharper boundaries than AnyUp, sometimes revealing finer details than the ground truth annotations.These improvements are observed in the third, sixth, and second rows, respectively.
  • E Further Qualitative Experiments: On DAVIS video semantic segmentation, RaysUp yields sharper object boundaries and robust inter-frame temporal consistency compared with existing methods.The visualization covers frames 8, 11, 24, 28, 37, and 43 of the dogs-jump sequence.
Loading 2606.22749v1…