PaperlayerView exact revision on arXiv

Source-linked AI summary

ViT-Up: Faithful Feature Upsampling for Vision Transformers

Krispin Wandel, Jingchuan Wang, Hesheng Wang

arXiv:2606.14024v1cs.CV

TL;DR

ViTs provide powerful features but operate on coarse token grids that limit dense prediction. ViT-Up predicts features at continuous image coordinates from ViT representations, outperforming prior upsamplers across dense prediction and semantic correspondence.

  • Problem

    ViTs commonly produce coarse patch-token grids because global self-attention scales quadratically, limiting dense prediction tasks that require precise spatial features.

  • Method

    ViT-Up predicts features at arbitrary continuous image coordinates using queries constructed from ViT representations while maintaining alignment with the backbone feature space.

  • Results

    +2.07 mIoU on Cityscapes with DINOv3-S+, while ViT-Up shows gains across segmentation, depth, and semantic correspondence.

  • Takeaways & Limitations

    ViT-Up provides an effective task-agnostic feature upsampler for dense visual prediction and fine-grained correspondence, with performance gains across tasks.

  • Takeaways & Limitations

    Post-hoc upsampling cannot fully recover structures absent from the backbone hidden states, reflecting an information bottleneck.

Abstract

from arXiv · show

Vision Transformers (ViTs) have become a dominant architecture for visual representation learning, providing exceptionally strong and broadly reusable backbone features. However, ViTs are commonly operated on relatively small patch-token grids due to the quadratic cost of global self-attention, which creates a persistent bottleneck for dense prediction tasks such as semantic segmentation and depth estimation. This has motivated the development of task-agnostic feature upsamplers. While recent state-of-the-art methods produce visually sharp dense representations, their reliance on shallow image encoders for guided upsampling can introduce feature leakage, fragmentation, and blur. We introduce ViT-Up, an implicit feature upsampling framework that replaces external image guidance with layer-wise query construction from intermediate ViT hidden states. This enables feature prediction at arbitrary continuous image coordinates while preserving alignment with the backbone feature space. Experiments demonstrate that ViT-Up consistently outperforms state-of-the-art image-guided upsamplers across dense prediction and semantic correspondence. On DINOv3-S+, ViT-Up improves over prior methods by up to +2.07 mIoU on Cityscapes and +4.17 PCK@0.10 on SPair-71k. With the larger DINOv3-B backbone, these gains increase to +3.36 mIoU and +8.09 PCK@0.10, demonstrating that ViT-Up scales favorably with backbone capacity.

I. INTRODUCTION · II. RELATED WORK · A. Task-Dependent Feature Upsamplers for Dense Prediction

ViTs provide strong reusable features but operate on coarse token grids because global self-attention scales quadratically, limiting dense prediction. ViT-Up addresses this mismatch with task-agnostic, coordinate-conditioned upsampling from intermediate ViT states, avoiding leakage-prone image guidance and improving dense prediction and semantic correspondence.

  • I. INTRODUCTION: ViTs offer strong, reusable semantic features, but quadratic global self-attention encourages operation at coarse patch-token resolutions.This creates a resolution mismatch for dense prediction tasks requiring spatially detailed feature maps.
  • I. INTRODUCTION: Higher-resolution backbone evaluation can degrade feature quality out of distribution, while specialized decoders require task-specific training and additional computation.These limitations motivate reusable task-agnostic feature upsamplers.
  • I. INTRODUCTION: Image-guided upsamplers produce visually sharp maps but can suffer feature leakage because shallow image encoders lack semantic capacity comparable to the foundation backbone.The resulting leakage can mix adjacent concepts when disentangling compressed patch-token grids.
  • I. INTRODUCTION: ViT-Up constructs dense query features at arbitrary continuous image coordinates by progressively integrating hidden states from the ViT hierarchy.This preserves alignment with the backbone feature space while producing features beyond the native token grid.
  • I. INTRODUCTION: +2.07 mIoU on Cityscapes is the improvement ViT-Up achieves over NAF and UPLiFT on DINOv3-S+.The same passage reports a +0.55 increase in δ1 on COCO depth over those prior state-of-the-art methods.
  • I. INTRODUCTION: ViT-Up consistently outperforms prior feature upsamplers across dense prediction and semantic correspondence while scaling effectively with backbone capacity.Its richer intermediate representations support scaling, unlike external guidance signals that are largely decoupled from the ViT backbone.
  • I. INTRODUCTION: ViT-Up uses multi-scale teacher supervision to recover fine spatial detail while maintaining consistency with the backbone feature space across scales.The student receives a downscaled, fixed-resolution input and is queried densely over the visible image region.
  • A. Task-Dependent Feature Upsamplers for Dense Prediction: Task-dependent dynamic operators such as CARAFE, DySample, FADE, and SAPA improve upsampling within downstream encoder–decoder or feature-pyramid architectures.These methods optimize upsampling as one component of a task-specific dense prediction model rather than as a reusable foundation-feature upsampler.

B. Image Super-Resolution and Local Implicit Functions · C. Implicit Feature Upsampling · D. Guided Feature Upsampling

The paper traces feature upsampling from local implicit functions and per-image implicit fields to guided methods, highlighting their limitations for high-dimensional semantic features. ViT-Up instead decodes features at continuous coordinates from intermediate ViT hidden states, preserving connection to the backbone feature space beyond the native token grid.

  • B. Image Super-Resolution and Local Implicit Functions: Local implicit functions replace fixed-grid prediction with continuous coordinate-based decoding, allowing representations to be evaluated at arbitrary resolutions and sampled at arbitrary locations or densities.This formulation motivates applying image super-resolution ideas to low-resolution latent feature maps.
  • B. Image Super-Resolution and Local Implicit Functions: LIIF predicts arbitrary-coordinate RGB values from nearby latent features and relative offsets, relying on the closest latent cell to contain sufficient local reconstruction information.Its locality assumption is suited to RGB reconstruction because the encoder is trained for local photometric prediction.
  • B. Image Super-Resolution and Local Implicit Functions: The locality assumption may fail for feature upsampling because the target shifts from three-channel RGB values to a high-dimensional semantic feature space.Unlike RGB reconstruction, semantic feature prediction may require information beyond nearby latent cells.
  • C. Implicit Feature Upsampling: FeatUp’s optimization-based variant fits an implicit neural feature field to each input image by iteratively optimizing an MLP through multi-view consistency and extensive input augmentations.Its representation is adapted directly to each input, producing high-quality dense features, but the optimization must be repeated for every input.
  • C. Implicit Feature Upsampling: LoftUp provides an end-to-end alternative that discards restrictive locality and lets high-resolution queries access the full set of low-resolution feature tokens through cross-attention.It adds sinusoidal positional encodings to RGB values and low-resolution feature tokens.
  • D. Guided Feature Upsampling: Guided upsamplers use high-resolution signals to densify lower-resolution targets, with methods including joint bilateral upsampling, guided filtering, FeatUp, JAFAR, AnyUp, and NAF.These approaches use image-derived guidance, cross-attention, or feature-agnostic projections to support dense or zero-shot upsampling.
  • D. Guided Feature Upsampling: Recursive methods such as LiFT and UPLiFT progressively densify features across scales, but repeated stages can degrade features, motivating local attenders for image-guided aggregation.LiFT fuses ViT features with CNN image features at corresponding scales, while UPLiFT mitigates recursive degradation.
  • D. Guided Feature Upsampling: Guidance improves spatial sharpness through boundary and texture cues, yet semantic quality remains constrained by the guidance encoder and can merge visually similar but semantically distinct regions.ViT-Up avoids guidance-based token reassembly by decoding features at arbitrary continuous coordinates from intermediate ViT hidden states, keeping predictions tied to the backbone feature space.

III. METHOD · A. Problem Formulation · B. Architecture

ViT-Up formulates feature upsampling as predicting dense, spatially aligned ViT representations from low-resolution hidden states. Its architecture constructs continuous-coordinate queries, refines them layer by layer with backbone context and sub-token detail, and adapts the backbone with LoRA.

  • A. Problem Formulation: Given an input of size (H, W), a ViT with patch size p produces hidden states H_l ∈ R^{h×w×C}, where h = H/p and w = W/p.The formulation includes L layers, including the embedding layer, and model dimension C.
  • A. Problem Formulation: The goal is to predict substantially denser last hidden states H_up ∈ R^{h*×w*×C}, with h*, w* ≫ h, w, while preserving ViT feature-space structure and adding spatial detail.The dense representation is obtained by increasing spatial resolution without abandoning alignment with the backbone feature space.
  • B. Architecture: ViT-Up predicts dense features at arbitrary continuous query coordinates x_q ∈ R^2 from low-resolution ViT hidden states, with high-resolution grids obtained by querying those coordinates.This makes the upsampler implicit rather than tied to a single discrete output grid.
  • B. Architecture: For each query coordinate, ViT-Up initializes q_0 ∈ R^C and progressively refines it through T blocks using the query coordinate and layer-specific low-resolution hidden states.The blocks produce intermediate query representations q_1, …, q_T ∈ R^C and follow the layer-wise organization of the ViT backbone.
  • B. Architecture: The query embedding reuses the ViT patch-embedding convolution with kernel size p, stride p, input dimension 3, and output dimension C to retain backbone alignment.Reusing the patch-embedding weights also supplies the initial query representation with backbone-compatible features.
  • B. Architecture: Each ViT-Up block aligns queries to the current backbone layer, aggregates local token context through cross-window attention with continuous two-dimensional RoPE, and fuses the resulting outputs.NATTEN accelerates the cross-window attention operation.
  • B. Architecture: FeatX recovers sub-token detail by selecting the nearest patch-token feature and coordinate, encoding the relative offset, applying position-conditioned FiLM modulation, and extracting a query-specific feature.The relative offset is expressed in token-grid units and embedded with a sinusoidal positional encoding before FiLM modulation.
  • B. Architecture: LoRA adapts the patch embedding and ViT attention projections, adding upsampling capacity without fully fine-tuning the backbone; the main setting uses rank r = 16, scale α = 32, and dropout 0.05.The adapted projections include the query, key, value, and output projections of the attention blocks.

C. Training

ViT-Up is trained with multi-scale student-teacher distillation, supervising dense predictions against teacher features across resolutions and refinement stages. Its objective combines target-normalized L2, angular, and relational losses, using ImageNet-1K training with one epoch and cosine-annealed optimization.

  • Multi-scale Feature Supervision: Multi-scale student-teacher distillation supervises ViT-Up using teacher features from the same image processed at multiple square resolutions.This avoids expensive or degraded ultra-high-resolution teacher features while providing high-resolution supervision.
  • Multi-scale Feature Supervision: The student receives an image scaled by s ∼ U(0.1, 1.0), pasted at a random position within a black 448×448 canvas.A regular 56 × 56 query grid is sampled over the pasted image region, matching the finest teacher token-grid resolution.
  • Multi-scale Feature Supervision: Dense query predictions are supervised at the finest teacher resolution and after average pooling to coarser token-grid resolutions across all selected layers and refinement stages.The supervision compares student predictions with corresponding teacher feature maps for n ∈ N and t = 0, . . . , T.
  • Losses: The training objective combines target-normalized L2, angular alignment, and pairwise relational losses.In the main configuration, λL2, λcos, and λrel are all set to one.
  • Dataset and optimization: Training uses ImageNet-1K for one epoch with batch size 24, initial learning rate 2 × 10−4, and cosine annealing.This setup follows prior feature upsampling work and UpLiFT’s one-epoch training protocol.

IV. EXPERIMENTS · A. Evaluation Setup

The evaluation uses DINOv3 ViT backbones and compares ViT-Up against bilinear interpolation and four recent feature upsampling methods. DINOv3 is selected for its relevance to feature upsampling and cleaner intermediate representations than DINOv2.

  • A. Evaluation Setup: ViT-Up is evaluated primarily with the DINOv3 backbone family, a widely used ViT backbone for feature upsampling.
  • A. Evaluation Setup: DINOv3 is technically suited to ViT-Up because it provides cleaner intermediate representations than earlier DINOv2 features.
  • A. Evaluation Setup: The setup avoids making dense upsampling primarily an artifact-suppression problem caused by systematic position-dependent artifacts in backbone feature maps.
  • A. Evaluation Setup: ViT-Up is compared with standard bilinear interpolation and four recent methods: JAFAR, AnyUp, NAF, and UpLiFT.
  • A. Evaluation Setup: JAFAR and UpLiFT use publicly available DINOv3 checkpoints released by their authors.
  • A. Evaluation Setup: AnyUp uses a multi-backbone checkpoint trained on DINOv2, CLIP, SigLIP, DINOv2 with registers, and an ImageNet-supervised ViT-B.

B. Qualitative Analysis

Qualitative analyses on vegetable-store and traffic scenes show that ViT-Up produces more coherent dense features than competing upsamplers. Its similarity maps remain aligned with semantically matching structures, whereas visually sharp alternatives exhibit leakage, noise, or fragmented responses.

  • Qualitative comparisons: ViT-Up produces more coherent feature representations than competing methods in both vegetable-store and traffic-scene examples.The analysis uses 448×448 inputs and matching 448×448 output feature maps, with PCA projections and query-based similarity maps.
  • Vegetable-store scene: ViT-Up separates neighboring vegetables, preserves consistent shelf-label features, and limits similarity leakage into surrounding vegetables.A shelf-label query receives high similarity at other shelf labels while remaining concentrated on label-like structures.
  • Vegetable-store scene: NAF, AnyUp, and JAFAR produce visually sharp maps but leak around shelf labels and spread similarity into nearby vegetables.The selected green label is visually similar to the vegetables behind it, exposing this failure mode; visual sharpness alone does not ensure coherent dense features.
  • Traffic scene: NAF, AnyUp, and JAFAR show fragmented high-frequency noise around vehicles and background structures despite appearing sharp, while UpLiFT is much blurrier.The observed variations do not correspond to stable semantic regions.
  • Traffic scene: ViT-Up selectively highlights other traffic lights, including small distant ones, whereas competing methods are diffuse, leak into backgrounds, or weaken responses to distant lights.The traffic-light query similarity maps reinforce ViT-Up’s stronger semantic coherence.

C. Dense Linear Probing

Dense linear probing evaluates frozen upsampled features with lightweight task-specific heads, showing that ViT-Up consistently improves semantic segmentation and depth estimation over all baselines. Qualitative results further associate its gains with more coherent dense features and cleaner outputs, avoiding leakage, fragmentation, and blur.

  • Evaluation setup: Dense linear probing freezes each backbone and upsampler while training a lightweight prediction head for semantic segmentation and monocular depth estimation.Segmentation uses VOC, COCO, ADE20K, and Cityscapes; depth uses COCO images with pseudo-depth targets from Depth Anything V2.
  • Quantitative results: 64.09 mIoU on COCO, 87.47 mIoU on VOC, 44.72 mIoU on ADE20K, and 65.41 mIoU on Cityscapes are ViT-Up’s semantic-segmentation results.These improve over the best baseline by +0.23, +1.63, +0.49, and +2.07 mIoU, respectively.
  • Quantitative results: +0.16 pixel accuracy on COCO, +0.42 on VOC, +0.35 on ADE20K, and +0.60 on Cityscapes are ViT-Up’s corresponding gains over the best baselines.The passage reports these gains alongside ViT-Up’s semantic-segmentation mIoU results.
  • Interpretation: ViT-Up’s largest gains occur on Cityscapes, VOC, and COCO depth, with Cityscapes improvements linked to extracting fine spatial detail from backbone representations.Cityscapes includes small objects and thin structures such as pedestrians, poles, traffic signs, and traffic lights.
  • Qualitative analysis: Qualitative examples show leakage and fragmentation in NAF, AnyUp, and JAFAR, blur in UpLiFT, and more coherent dense features and cleaner outputs from ViT-Up.These artifacts correspond to incorrect local segmentation or degraded depth estimates around small and overlapping structures.

D. Correspondence Estimation

ViT-Up substantially improves semantic correspondence on SPair-71k and achieves the best geometric correspondence results on NAVI. Qualitative evidence attributes these gains to finer-grained feature preservation and reduced feature leakage in challenging correspondence cases.

  • Evaluation Protocol: Correspondence evaluation tests whether dense feature similarity preserves semantic and geometric structure, complementing dense linear probing.The study evaluates semantic correspondence on SPair-71k using PCK and geometric correspondence on NAVI using 3D PCK.
  • Semantic Correspondence: +4.17 PCK at threshold 0.10 is ViT-Up’s gain over the strongest baseline on SPair-71k.ViT-Up obtains 55.44, 39.07, and 7.30 PCK at thresholds 0.10, 0.05, and 0.01, with gains of +4.17, +5.11, and +3.47 points, respectively.
  • Semantic Correspondence: ViT-Up preserves fine-grained feature structure needed for subclass- and part-level discrimination, avoiding leakage and over-smoothing that can damage local feature geometry.On bird and horse image pairs, ViT-Up produces finer-grained features and more accurate matches, while NAF and UpLiFT mix object features with background or occluding obstacles.
  • Geometric Correspondence: 80.81 PCK at threshold 0.10 is ViT-Up’s NAVI result, exceeding AnyUp’s 80.31.At thresholds 0.05 and 0.01, ViT-Up obtains 51.59 and 33.83 PCK, compared with strongest-baseline results of 51.18 and 33.58 from bilinear interpolation.

E. Feature Preservation · F. Backbone Scaling

ViT-Up preserves semantic organization for frozen dense predictors while scaling favorably from DINOv3-S+ to DINOv3-B. Larger-backbone gains are strongest on Cityscapes and SPair-71k, whereas coarse-token interpolation and NAF can degrade in correspondence.

  • E. Feature Preservation: ViT-Up evaluates feature preservation by applying heads trained on native DINOv3-S+@28 features to frozen high-resolution upsampled features.For segmentation, this protocol tests whether upsampling changes the semantic organization expected by the predictor.
  • E. Feature Preservation: ViT-Up substantially outperforms prior upsamplers on VOC and Cityscapes, even surpassing competing methods’ finetuned variants with a low-resolution-trained head.The result indicates preservation of semantic feature organization rather than merely sharper visual features.
  • E. Feature Preservation: Frozen depth probing requires more caution because the head is trained on coarse native tokens, while depth regression is sensitive to feature-map resolution and local smoothness.Unlike segmentation, depth estimation is continuous and the native tokens aggregate information over relatively large image regions.
  • F. Backbone Scaling: ViT-Up scales to DINOv3-B by doubling its internal dimension to match the backbone’s doubled feature dimension while keeping the architecture unchanged.The comparison includes bilinear interpolation and NAF; UpLiFT is omitted because no public DINOv3-B checkpoint is available.
  • F. Backbone Scaling: 3.36 mIoU is ViT-Up’s Cityscapes margin over NAF on DINOv3-B, up from 2.07 mIoU, while the larger backbone also widens its SPair-71k correspondence advantage.The Cityscapes result is associated with recovering fine detail in thin and small structures.
  • F. Backbone Scaling: 1.11 points is ViT-Up’s VOC margin over NAF on DINOv3-B, down from 1.69 points, although ViT-Up still substantially outperforms NAF.The smaller margin is consistent with VOC’s foreground-background object segmentation setting, which favors image-guided aggregation.
  • F. Backbone Scaling: 50.01 is DINOv3-B’s SPair-71k PCK@0.1 with bilinear interpolation, down from 51.27 for DINOv3-S+, while NAF decreases from 48.68 to 47.19.These results show that larger backbone capacity does not automatically improve correspondence at the coarse native token resolution, whereas ViT-Up can use the increased feature dimension directly from ViT hidden states.

G. Ablation Studies · H. Runtime and Memory

ViT-Up’s ablations show that its architectural components, refinement depth, and output resolution jointly improve semantic structure and spatial precision, while runtime remains competitive through query-wise computation and controllable memory use. The six-layer base configuration balances accuracy and runtime, and 112×112 querying offers an efficient operating point.

  • G. Ablation Studies: Removing cross-attention or FeatX substantially degrades frozen probing, while their combination produces features that are both coherent and detailed.Without FeatX, features retain spatial consistency but lose fine detail; without cross-attention, they preserve local texture but introduce pixelation artifacts.
  • G. Ablation Studies: The full model is stronger at the stricter PCK@0.01 threshold, although removing FeatX slightly improves coarse SPair-71k PCK@0.10.The authors attribute the coarse-threshold effect to fusion between cross-attention output and FeatX sub-token features.
  • G. Ablation Studies: 0.11 Cityscapes mIoU is the marginal improvement from disabling LoRA, but removing LoRA or KL regularization slightly reduces SPair-71k performance overall.The full model provides the best overall trade-off across probing and correspondence.
  • G. Ablation Studies: Disabling either the decoder or transition MLP consistently lowers performance, with the decoder ablation particularly harmful on Cityscapes.The decoder appears to organize and utilize feature channels even though the latent and output feature dimensions are equal.
  • G. Ablation Studies: 1.95 mIoU, 0.82 mIoU, 2.90 points, 4.56 points, and 2.47 points are the gains from increasing refinement layers from one to twelve on Cityscapes, VOC, and SPair-71k PCK levels 0.10, 0.05, and 0.01, respectively.The largest gains occur on spatially precise metrics, validating refinement depth as critical for localized, discriminative high-resolution features.
  • H. Runtime and Memory: ViT-Up’s six-layer base configuration captures most refinement gains while offering a better accuracy–runtime trade-off than twelve layers.Runtime and peak-memory measurements use bfloat16 inference on a single H100 SXM GPU, with query chunking controlling the memory–runtime trade-off without changing output resolution.
  • G. Ablation Studies: 56×56 yields the largest initial resolution improvement, 112×112 adds a further clear gain, and performance saturates at full 448×448 resolution.The results indicate that most recoverable semantic and spatial information is captured before full image resolution.
  • H. Runtime and Memory: 4 ms slower than UpLiFT at full resolution, ViT-Up remains in the fastest-baseline runtime regime while using low memory and outperforming full-resolution prior upsamplers at 112×112.At 448×448, ViT-Up is on par with JAFAR and AnyUp but slower than NAF and UpLiFT; independent output queries enable flexible processing.

V. LIMITATIONS AND OUTLOOK · VI. CONCLUSION

ViT-Up is limited by information absent from backbone hidden states and by tight coupling to each backbone, while higher-resolution features and joint training offer potential remedies. The framework reconstructs dense features through hierarchical intermediate-state queries and improves dense prediction and semantic correspondence over existing upsampling methods.

  • V. LIMITATIONS AND OUTLOOK: All post-hoc upsamplers are bounded by information encoded in backbone hidden states, so unrepresented structures cannot be fully recovered.ViT-Up mitigates this bottleneck by exploiting intermediate representations to recover sub-token detail and improve spatial precision beyond the native token grid.
  • V. LIMITATIONS AND OUTLOOK: 28×28 hidden states are used in the main setting, while 56×56 remains effective and feature quality degrades only at substantially higher resolutions such as 112×112.At 56×56, feed-forward and projection operations still dominate over the quadratic attention term.
  • V. LIMITATIONS AND OUTLOOK: ViT-Up is trained separately for each backbone because its transition MLP and FeatX depend on backbone-specific internal representations.The transition MLP projects between skipped hidden layers, while FeatX extracts sub-token information from intermediate ViT representations.
  • V. LIMITATIONS AND OUTLOOK: Jointly training ViT-Up with the backbone could use high-resolution reconstruction losses to preserve local detail and improve coordinate-conditioned query modulation.Current backbones may discard spatial detail unnecessary for their native training objectives.
  • VI. CONCLUSION: ViT-Up predicts vision transformer features at arbitrary continuous image coordinates through implicit feature upsampling.The framework reconstructs dense feature maps rather than restricting predictions to the native token grid.
  • VI. CONCLUSION: ViT-Up progressively refines queries from patch embeddings using low-resolution intermediate hidden states, aligning dense prediction with the backbone’s layer hierarchy.This avoids relying only on the final hidden state and keeps reconstruction consistent with the backbone’s internal representation hierarchy.
  • VI. CONCLUSION: ViT-Up shows significant gains over existing feature upsampling methods across segmentation, depth, and semantic correspondence, supporting dense visual prediction and fine-grained correspondence.The conclusion reports gains across linear probing for segmentation and depth as well as semantic correspondence.
  • VI. CONCLUSION: ViT-Up provides an effective and faithful way to obtain dense feature maps from vision transformers and motivates backbones with native continuous, high-resolution feature querying.The authors present this capability as a direction for future vision backbone design.

APPENDIX A COMPARISON TO HIGHER NATIVE TOKEN RESOLUTIONS · APPENDIX B UPSAMPLING VERSUS ARTIFACT SUPPRESSION

Appendix A shows that higher native token resolution improves dense prediction but can degrade semantic correspondence, while ViT-Up produces dense features from standard-resolution features without a denser backbone run. Appendix B distinguishes faithful reconstruction from artifact suppression: image-guided methods can clean DINOv2 artifacts, whereas ViT-Up preserves feature geometry and performs strongly on DINOv3.

  • APPENDIX A COMPARISON TO HIGHER NATIVE TOKEN RESOLUTIONS: Higher native token resolution improves segmentation probing and semantic correspondence, but DINOv3-S+@112 performs worse than DINOv3-S+@56 across all SPair-71k PCK thresholds.The comparison treats higher-resolution variants as references requiring the full ViT backbone on denser token grids, not as feature upsampling baselines.
  • APPENDIX A COMPARISON TO HIGHER NATIVE TOKEN RESOLUTIONS: Starting from standard 28×28 DINOv3-S+ features, ViT-Up produces dense target-resolution features without running the full backbone on a denser token grid.Against DINOv3-S+@56, it improves COCO, VOC, and ADE20K segmentation mIoU and all SPair-71k PCK thresholds, except Cityscapes.
  • APPENDIX A COMPARISON TO HIGHER NATIVE TOKEN RESOLUTIONS: ViT-Up’s largest advantage over DINOv3-S+@56 occurs at the strict SPair-71k PCK@0.01 threshold, while Cityscapes is the sole reported exception.The passage reports improvements on COCO, VOC, ADE20K, and all SPair-71k PCK thresholds, with DINOv3-S+@56 remaining stronger on Cityscapes.
  • APPENDIX B UPSAMPLING VERSUS ARTIFACT SUPPRESSION: DINOv2 benchmarking measures both high-resolution feature recovery and removal of artifacts already present in the backbone feature field.The recurring artifact modes are grid-like position-encoding artifacts and spatial feature leakage across object boundaries.
  • APPENDIX B UPSAMPLING VERSUS ARTIFACT SUPPRESSION: Image-guided pathways can suppress target-map artifacts by providing an additional image-aligned spatial prior, whereas ViT-Up treats those artifacts as part of the supervised target representation.Removing artifacts can therefore be penalized when they are present in the supervision signal.
  • APPENDIX B UPSAMPLING VERSUS ARTIFACT SUPPRESSION: On DINOv2, NAF and UpLiFT suppress visible position artifacts and spatial leakage more effectively than ViT-Up, which preserves the target feature field more directly.This makes image-guided methods favorable for tasks benefiting from artifact suppression, while artifact suppression and feature faithfulness remain distinct objectives.
  • APPENDIX B UPSAMPLING VERSUS ARTIFACT SUPPRESSION: 53.75 PCK@0.1 is ViT-Up’s SPair-71k semantic-correspondence score with DINOv2, exceeding UpLiFT at 53.08 and NAF at 50.46.The result shows that preserving feature geometry can remain favorable for semantic correspondence even when artifacts are not suppressed.
  • APPENDIX B UPSAMPLING VERSUS ARTIFACT SUPPRESSION: DINOv3 produces cleaner dense features than DINOv2, and its rotary position embeddings encode relative spatial relations rather than fixed positional vectors added to token representations.This reduces the need for feature vectors themselves to carry additive position-dependent offsets.
Loading 2606.14024v1…