PaperlayerView exact revision on arXiv

Source-linked AI summary

MuRF: Unlocking the Multi-Scale Potential of Vision Foundation Models

Bocheng Zou, Mu Cai, Mark Stanley, Dingfu Lu, Yong Jae Lee

arXiv:2603.25744v2cs.CV

TL;DR

Vision Foundation Models can process varying resolutions during training, yet inference commonly uses one scale, limiting access to complementary global and fine-grained information. MuRF addresses this gap by fusing features from multiple resolutions produced by a frozen VFM. The authors report consistent improvements across dense prediction, multimodal reasoning, and unsupervised anomaly detection.

  • Problem

    VFM inference typically uses a single fixed scale, overlooking the complementary global recognition and fine-grained refinement provided by different resolutions.

  • Method

    MuRF processes an image at multiple resolutions with a frozen VFM and fuses the resulting features into a unified representation for downstream task heads.

  • Results

    MuRF consistently improves VFM-based systems across dense prediction, multimodal reasoning, and unsupervised anomaly detection.

  • Takeaways & Limitations

    Multi-resolution aggregation is presented as a general principle for enhancing frozen VFM representations across diverse vision tasks.

  • Takeaways & Limitations

    The VQA experiments keep the visual token sequence fixed at 576 to avoid creating an LLM computational bottleneck.

Abstract

from arXiv · show

Vision Foundation Models (VFMs) have become the cornerstone of modern computer vision, offering robust representations across a wide array of tasks. While recent advances allow these models to handle varying input sizes during training, inference typically remains restricted to a single, fixed scale. This prevalent single-scale paradigm overlooks a fundamental property of visual perception: varying resolutions offer complementary inductive biases, where low-resolution views excel at global semantic recognition and high-resolution views are essential for fine-grained refinement. In this work, we propose Multi-Resolution Fusion (MuRF), a simple yet universally effective strategy to harness this synergy at inference time. Instead of relying on a single view, MuRF constructs a unified representation by processing an image at multiple resolutions through a frozen VFM and fusing the resulting features. The universality of MuRF is its most compelling attribute. It is not tied to a specific architecture, serving instead as a fundamental, training-free enhancement to visual representation. We empirically validate this by applying MuRF to a broad spectrum of critical computer vision tasks across multiple distinct VFM families - primarily DINOv2, but also demonstrating successful generalization to contrastive models like SigLIP2.

1. Introduction

MuRF addresses the mismatch between flexible multi-resolution training and rigid single-scale inference by fusing complementary low- and high-resolution VFM features. The method is presented as a universal, frozen-backbone enhancement validated across diverse vision tasks.

  • Motivation: Single-scale inference forces a trade-off between globally coherent recognition at low resolution and fine-grained boundary refinement at high resolution.High-resolution representations can also make object interiors noisy, risking incomplete segmentation.
  • Contribution: MuRF processes the same image across a resolution pyramid and fuses the resulting features into one unified representation without modifying the VFM backbone.The backbone remains a frozen feature extractor, avoiding expensive multi-scale backbone training.
  • Universality: MuRF is designed as an architecture- and task-independent enhancement to visual representation, evaluated with frozen VFM backbones across a broad spectrum of computer vision tasks.The paper emphasizes universality rather than dependence on one architecture or task.
  • Empirical validation: MuRF improves dense prediction, multimodal understanding, and unsupervised anomaly detection by combining global scene context with fine-grained visual details.Reported applications include semantic segmentation, depth estimation, visual question answering, and anomaly detection.
  • Conclusion: The results suggest that multi-resolution aggregation is a general principle that consistently improves VFM-based systems.The authors characterize MuRF as a simple, unified, and effective implementation of this principle.

2. Related Work

Related work spans fixed- and variable-resolution VFM processing, classical image and feature pyramids, and input-level strategies for high-resolution multimodal vision. MuRF revisits image pyramids at inference time while addressing limitations associated with tiling and learned feature upsampling.

  • Resolution handling: Recent VFMs support varying image sizes during training, but standard inference protocols typically revert to a single fixed scale.This preserves flexibility during training without exploiting multiple scales at inference.
  • Resolution handling: Single-scale inference overlooks the complementary roles of low-resolution global recognition and high-resolution fine-grained refinement.MuRF explicitly aggregates these views while preserving both global context and local details without retraining the backbone.
  • Multi-scale representations: Classical image pyramids and modern feature pyramids construct multi-scale representations, but older image-pyramid processing was computationally prohibitive and FPNs require in-network construction.These approaches provide historical context for multi-scale representation learning.
  • Input-level strategies: High-resolution MLLM strategies commonly tile images and optionally add a low-resolution global view, but tiling can break continuity across object boundaries.Objects spanning tile boundaries become difficult for models to reason about.
  • Feature upsampling: VFM feature upsampling can recover high-frequency details but does not inherently add information absent from the original single-scale forward pass and may require training.The cited training requirement may create generalizability concerns.
  • MuRF: MuRF modernizes the image-pyramid idea for frozen VFMs by fusing multi-resolution views in feature space, avoiding tiling boundary artifacts while supporting lightweight task-specific heads.Its overview uses resized views, a frozen DINOv2 encoder, shared spatial resolution, and downstream heads.

3. Method

MuRF builds a unified, scale-robust representation by extracting features from multiple image resolutions with a frozen VFM, aligning and concatenating them, then adapting the result with lightweight task-specific heads.

  • Multi-Resolution Feature Fusion: Low-resolution features provide global recognition context, while high-resolution features supply fine-grained detail for refinement.MuRF explicitly combines these complementary visual signals rather than relying on a single resolution.
  • Multi-Resolution Feature Fusion: MuRF resizes each input into multiple scales and processes every resulting image with a frozen VFM encoder.The method creates an input pyramid, then extracts patch-level feature maps from each resolution.
  • Multi-Resolution Feature Fusion: Each scale-specific feature map is upsampled to a common spatial resolution and concatenated along the channel dimension to form ℱMuRF.The fused representation has channel dimension D = ∣Sres∣× d.
  • Task-Specific Adaptation: The resulting MuRF tensor is a frozen, task-agnostic representation adapted through lightweight task-specific heads.Dense prediction uses a head that projects D-dimensional features to task outputs, with only head parameters updated during training.
  • Multi-Resolution Feature Fusion: Channel-wise concatenation preserves scale-specific activations by projecting them into a higher-dimensional representation instead of blending them through addition or averaging.The design addresses potential destructive interference between semantically different features across scales.
  • Task-Specific Adaptation: For anomaly detection and multimodal language modeling, MuRF uses task-specific downstream processing while keeping the feature extractor frozen.Anomaly scores are fused across resolutions, whereas MLLMs project MuRF features into the LLM word-embedding space and train only designated adaptation parameters.

4. Experiments

Experiments evaluate MuRF across semantic segmentation, depth estimation, VQA, and anomaly detection using frozen VFM backbones and lightweight task-specific heads. Across these tasks, MuRF improves over single-scale baselines while maintaining a generally reasonable computational trade-off.

  • Experimental Scope: MuRF is evaluated across semantic segmentation, depth estimation, visual question answering, and anomaly detection to test its universality.Experiments primarily use frozen DINOv2, with SigLIP2 included for VQA.
  • Semantic Segmentation: For semantic segmentation, MuRF significantly outperforms the single-scale DINOv2 baseline on ADE20K and PASCAL VOC.The evaluation uses linear probing and reports mIoU, with qualitative comparisons across input resolutions.
  • Efficiency: MuRF requires approximately 1.3× the training time of single-resolution segmentation while retaining comparable VQA VRAM usage and latency.For VQA, pretraining takes 72 versus 71 minutes and finetuning takes 274 versus 270 minutes for the single-resolution baseline.
  • Depth Estimation: For depth estimation, MuRF achieves substantially lower RMSE than single-scale DINOv2 by combining global scene geometry with object-boundary information.Experiments cover in-domain NYU Depth V2 and transfer evaluation on SUN RGB-D.
  • Visual Question Answering: MuRF improves multimodal understanding in LLaVA with both DINOv2 and SigLIP2 vision encoders while preserving the visual-token budget.All experiments use 576 visual tokens, avoiding extra LLM sequence-length computational cost.
  • Unsupervised Anomaly Detection: MuRF achieves competitive, sometimes state-of-the-art, anomaly-detection performance without parameter tuning across anomalies with widely varying scales.Its fused representation combines robust anomaly identification from low-resolution views with sharper localization from high-resolution views.

4.6. Qualitative comparison

The qualitative and ablation results show that MuRF balances complementary coarse and fine visual information, with multi-resolution fusion consistently outperforming single-scale alternatives across depth and anomaly detection.

  • Qualitative comparison: High-resolution features sharpen segmentation boundaries but can lose global context, whereas low-resolution features produce smoother regions; MuRF balances both behaviors.The qualitative comparison describes sharper boundaries and local detail at high resolution, smoother interiors at low resolution, and improved balance with MuRF.
  • Feature aggregation comparison: MuRF and multi-layer feature concatenation provide complementary benefits, and combining them achieves the best depth-estimation performance.The controlled comparison contrasts three fused resolutions with three encoder layers and reports that the combined approach performs best.
  • Depth estimation ablation: 0.361 RMSE is achieved by full three-scale MuRF on NYU Depth V2, improving over the best single-scale baseline at 0.389.Two-scale variants already reduce RMSE to 0.373–0.366, while the full 0.5×–1.0×–1.5× configuration reaches 0.361.
  • Depth estimation ablation: Performance improves monotonically as more resolutions are fused, with diminishing but consistent returns indicating genuine multi-scale complementarity.Adding the low-resolution view improves the result even after combining two stronger scales.
  • Anomaly detection ablation: 57.32 AU-PRO0.05 is obtained by five-scale MuRF for anomaly detection, exceeding the best single-resolution result of 55.39.Three-scale fusion reaches 56.60, four-scale fusion reaches 57.29, and the full five-scale configuration reaches 57.32.
  • Anomaly detection ablation: Anomaly detection benefits from covering coarse-to-fine views rather than selecting one magic resolution, because coarse scales localize broadly while fine scales sharpen boundaries.Overlapping multi-scale subsets perform similarly, and the qualitative findings attribute their complementarity to localization versus boundary refinement.

5. Conclusion

MuRF builds a feature pyramid from multiple input resolutions and fuses the representations to combine global context with fine-grained detail. Across dense prediction, multimodal reasoning, and unsupervised anomaly detection, the unified inference-time approach yields consistent performance gains and supports multi-resolution aggregation as a general principle for frozen visual encoders.

  • 5. Conclusion: MuRF fuses representations from a feature pyramid to combine low-resolution global context with high-resolution fine-grained detail.The strategy operates at inference time and enhances representations without changing the underlying visual encoder.
  • 5. Conclusion: A single MuRF approach improves performance across dense prediction, multimodal reasoning, and unsupervised anomaly detection.The conclusion describes these gains as consistent and significant across the listed task categories.
  • 5. Conclusion: The results support multi-resolution aggregation as a general principle for unlocking frozen pretrained visual encoders rather than a task-specific trick.The conclusion frames the approach as a unified way to enhance pretrained visual representations across tasks.

A.1. Semantic Segmentation

The semantic-segmentation appendix evaluates MuRF on ADE20K and PASCAL VOC 2012 using frozen DINOv2 features, dataset-specific resolution sets, and task-specific training and evaluation procedures.

  • A.1. Semantic Segmentation: Semantic segmentation is evaluated on ADE20K and PASCAL VOC 2012 using frozen DINOv2-Base features and a pixel-wise cross-entropy objective.The segmentation head is optimized with AdamW; the appendix specifies the learning rate, weight decay, momentum parameters, and training duration.
  • A.1. Semantic Segmentation: MuRF uses resolutions {266, 518, 784} for ADE20K and {140, 266, 518} for PASCAL VOC 2012 because the latter images generally have lower resolution.The selected resolution sets differ across datasets to accommodate their image-resolution characteristics.
  • A.1. Semantic Segmentation: Testing resizes ADE20K images to scale 784 without sliding windows, whereas PASCAL VOC 2012 uses 512 × 512 sliding windows with stride 341 × 341.Both evaluations use the original validation sets.

A.2. Depth Estimation

The depth-estimation appendix trains and evaluates frozen DINOv2 features with single-layer and multi-layer extraction configurations, multi-scale resolution inputs, and a composite depth-and-gradient objective.

  • A.2. Depth Estimation: Depth estimation trains on NYU Depth V2 and evaluates zero-shot on SUN RGB-D using resolutions {0.5s, 1.0s, 1.5s}.Here, s denotes the original image resolution, and training uses mixed-precision arithmetic with AdamW.
  • A.2. Depth Estimation: The appendix compares final-layer features with fusion from transformer layers {3, 6, 9, 12}, concatenating layer-specific global and spatial tokens before decoding.The multi-layer representation concatenates upsampled features across the selected layers along the channel dimension.
  • A.2. Depth Estimation: Depth prediction uses 256 uniform bins over [1 × 10^-3, 10] and minimizes a composite scale-invariant log loss plus multi-scale gradient matching term.The balancing coefficients are λ_depth = 1.0 and λ_grad = 0.5; test-time augmentation averages direct and horizontally flipped predictions.

A.3. LLaVA-style MLLM training

The MLLM experiments adapt LLaVA 1.5 by replacing its vision encoder with DINOv2 or SigLIP2 and evaluating single-resolution baselines alongside MuRF configurations.

  • LLaVA 1.5 experiments replace the vision encoder with DINOv2 and SigLIP2.
  • The DINOv2 baseline uses 336 × 336 inputs, whereas MuRF combines 224 × 224 and 336 × 336 resolutions.The 224-resolution feature embedding is resized to 24 × 24 before fusion.
  • The SigLIP2 baseline uses 384 × 384 inputs, whereas MuRF combines 256 × 256 and 384 × 384 resolutions.The selected 384 × 384 input produces 576 tokens, matching the DINOv2 and original CLIP variants.
  • Token-wise concatenation doubles the feature length per visual token while preserving the total number of visual tokens.The projection layer input dimension is changed accordingly, while its output dimension and the LLM token count remain unchanged.
  • The experiments follow LLaVA 1.5’s two-stage training pipeline with the same hyperparameters.

A.4. Anomaly Detection

The anomaly-detection setup applies MuRF through multi-resolution, multi-layer feature extraction and nearest-neighbor scoring, with score maps fused after spatial upsampling.

  • MuRF anomaly detection extracts patch-level feature maps from multiple resolutions and intermediate encoder layers.Each feature map combines a spatial scale with a semantic level, using layer outputs whose spatial dimensions depend on resolution.
  • At inference, each feature vector is scored by its L2 distance to the nearest neighbor in a dedicated memory bank.
  • Individual anomaly score maps are upsampled to the original image dimensions and aggregated by element-wise averaging.
  • The experiments use layers 7, 9, and 11 with resolutions 0.3, 0.4, 0.5, 0.6, and 0.7.Images are resized to the nearest multiple of the 14-pixel patch size before entering the model.
  • The anomaly-detection implementation uses Faiss IndexIVFFlat search with nlist = 512 and nprobe = 32.
  • A full run on one MVTec AD v2 category takes less than 3 hours on the reported hardware.

B. Additional Visualization

Additional visualizations are provided for semantic segmentation, depth estimation, and PCA qualitative comparisons.

  • Additional semantic-segmentation visualizations are provided in Figure 7.
  • Additional depth-estimation visualizations are provided in Figure 8.
  • Additional PCA qualitative comparisons are provided in Figure 9.
Loading 2603.25744v2…