PaperlayerView exact revision on arXiv

Source-linked AI summary

SAGE: Saliency-Annealed Gated Encoder for Universal Vision Recognition

Le Yang, Yucheng Huang, Junyu Chen, Qi Dang, Xin Kong, Haifeng Huo, Qiang Zhao

arXiv:2610.36719v1cs.CV

TL;DR

Adapting self-supervised ViTs for dense prediction remains challenging because their saliency, global context, and detail information may be underused. SAGE adds saliency-guided feature enhancement, annealing, and a focal-to-VFL training transition; it reports state-of-the-art or competitive results across multiple tasks, including 62.8 box AP on COCO object detection.

  • Problem

    Self-supervised ViT adaptation to dense prediction presents distinct architectural challenges, and existing adapters may underuse multi-layer representations and register-token context.

    Source: §I · p. 1

  • Method

    SAGE is an enhanced ViT-Adapter backbone combining saliency-guided feature modulation, annealing-based training, and a two-stage focal-to-VFL strategy.

    Source: §I · pp. 1–2

  • Results

    SAGE reports state-of-the-art or competitive results across multiple vision tasks, including 62.8 box AP on COCO object detection.

    Source: §I · p. 2, §V · p. 8

  • Takeaways & Limitations

    The same enhanced backbone is used across tasks, with only the downstream task head changed.

    Source: §I · p. 2

Abstract

from arXiv · show

When adapting self-supervised Vision Transformers (ViTs) to dense prediction, existing methods often treat the ViT as a passive feature extractor and may underuse the rich priors encoded in its representations, including saliency, global context, and high-frequency details. We propose SAGE (Saliency-Annealed Gated Encoder), an enhanced ViT-Adapter backbone that explicitly exploits these priors through three complementary innovations: (1) a saliency-guided architecture that extracts DINO-style class-token (CLS) saliency to guide feature gating and high-frequency enhancement, together with dynamic multi-layer aggregation and a local-detail branch; (2) an annealing-based training paradigm designed to improve convergence and feature quality during early training while reducing feature drift later in training; and (3) a two-stage, quality-aware training mechanism that smoothly transitions from focal loss to Varifocal Loss (VFL) within a single training run. Across the evaluated tasks (Fig.~\ref{fig:fig_com}), SAGE achieves state-of-the-art (SOTA) or competitive results on COCO (62.8 box AP and 55.3 mask AP), COCO keypoint estimation (78.5 AP), ADE20K (60.3 mIoU for semantic segmentation and 44.9 mask AP for instance segmentation), TinyImageNet classification (94.76\% Top-1 accuracy), and DIOR-R oriented object detection (78.8 AP$_{50}$). The same backbone is used across tasks, with task-specific decoders. To the best of our knowledge, SAGE is the first framework to achieve multi-task state-of-the-art results on Huawei Ascend NPUs. Code and pretrained models will be released after internal review.

I. INTRODUCTION

SAGE extends ViT-Adapter to explicitly use self-supervised ViT priors for dense prediction, addressing limited foreground discrimination, fine-detail access, and use of multi-layer and register-token representations. It combines architectural changes with annealing and a focal-to-VFL training transition.

I. INTRODUCTION

  • Self-supervised ViT adaptation to dense prediction faces architectural challenges, including limited fine-detail access, explicit foreground discrimination, and dynamic use of multi-layer and register-token information.ViT patchification limits fine-grained edge and texture representation, while adapters may rely on final or fixed-index layers and use register tokens only implicitly.Source: §I · p. 1
  • SAGE extracts DINO-style CLS saliency to guide feature modulation and high-frequency enhancement, with multi-depth aggregation and a 2 × 2 sub-crop branch for local detail.The method also extracts register-token context to guide feature enhancement.Source: §I · pp. 1–2, A. DETR-Based Detection and ViT… · p. 2
  • SAGE uses a three-stage proportion-based cosine annealing schedule to reduce enhancement modules toward identity mapping and transitions from focal loss to VFL within one run.Saliency-guided query initialization and semantic-consistency loss accompany the focal-to-VFL transition.Source: §I · p. 2
  • SAGE achieves state-of-the-art or competitive results across multiple vision tasks under single-scale inference without TTA, using the same enhanced backbone with task-specific heads.The proposed feature-level enhancements were evaluated across tasks, with only the downstream task head changed.Source: §I · p. 2

A. DETR-Based Detection and ViT Adaptation

  • ViT-Adapter uses an SPM and MSDeformAttn for bidirectional feature interaction, while several related adapters lack explicit extraction of CLS saliency, register context, or high-frequency detail.DEIMv2 uses fixed layer indices without learnable aggregation; SAGE builds on this line by computing DINO-style saliency maps and extracting register-token context.Source: A. DETR-Based Detection and ViT… · p. 2

B. Training Strategies and General-Purpose Backbone Design

SAGE coordinates enhancement modules, loss scheduling, and representation stability in a task-independent feature backbone. Its architecture combines SPM and ViT pathways with local-detail features, then sends fused features to task-specific decoders.

B. Training Strategies and General-Purpose Backbone Design

  • SAGE coordinates a focal-to-VFL transition intended to mitigate cold start with annealing that gradually reduces enhancement modules toward identity mappings.Applying VFL from training’s start may be unstable when early predicted IoUs are noisy; coordinated design of loss schedules and representation stability has received less attention.Source: B. Training Strategies and… · p. 2
  • SAGE’s enhancement modules operate at the feature-representation level and are designed to be independent of downstream tasks, with the same enhanced backbone evaluated using task-specific decoders.The evaluated tasks include detection, instance segmentation, classification, oriented detection, pose estimation, and semantic segmentation.Source: B. Training Strategies and… · p. 2

A. Overall Architecture

  • The overall architecture combines SPM multi-scale features, all-layer ViT representations, and a 2 × 2 local-detail branch, then fuses enhanced features for task-specific decoding.CLS–patch similarity supplies saliency, sampled ViT layers are dynamically aggregated, and local-detail features are fused at high-resolution scales.Source: A. Overall Architecture · pp. 2–3, Figure 2 · p. 3

B. Saliency-Guided Enhanced ViT-Adapter Architecture

  • Saliency, high-frequency detail, and register-token global context motivate complementary feature-modulation modules for foreground discrimination, edge precision, and semantic consistency.Source: B. Saliency-Guided Enhanced… · p. 3

1) DINO-style Saliency Computation:

SAGE computes a foreground saliency prior from intermediate-layer CLS–patch similarity and uses it to modulate ViT features. Register tokens provide bounded global-context modulation at the deepest scale.

1) DINO-style Saliency Computation:

  • The saliency prior uses cosine similarity between normalized patch tokens and the CLS token from a two-thirds-depth ViT layer, followed by per-image standardization and sigmoid mapping.The selected layer balances semantic abstraction for foreground-background discrimination with spatial detail for boundary localization.Source: 1) DINO-style Saliency Computation · pp. 3–4

2) Scale-Adaptive Residual Soft Gating:

  • A residual gate centered at 1.0 uses saliency and learnable per-scale weights to enhance foreground features by at most 20% and suppress, but not zero, background features.The SPM baseline remains ungated, while the ViT residual features provide foreground-background discrimination.Source: 2) Scale-Adaptive Residual Soft Gating · p. 4

3) Register-Token FiLM Modulation:

  • Register-token FiLM modulation mean-pools the tokens and uses a two-layer MLP to produce bounded modulation parameters applied only at the deepest scale.The bounds keep γ between 0.90 and 1.10 and β between −0.10 and 0.10, limiting modulation magnitude during long training schedules.Source: 3) Register-Token FiLM Modulation · p. 4

4) Laplacian High-Frequency Enhancer:

The Laplacian enhancer adds bounded edge-related residuals to shallow feature scales, while dynamic multi-layer aggregation learns how Transformer depths contribute to four scale-specific representations.

4) Laplacian High-Frequency Enhancer:

  • A fixed four-neighborhood Laplacian extracts edge responses, added as a bounded residual with learnable strength to res2 and res3 for small-object and boundary detail.The scale factor is bounded by 0.20 and initialized to 0.10; SiLU is intended to attenuate weak responses while retaining stronger edges.Source: 4) Laplacian High-Frequency Enhancer · p. 4

5) Dynamic Multi-Layer Aggregation:

  • Learned weights aggregate Transformer-layer features into four scales, with depth-centered initialization and eight sampled layers instead of twelve.The eight-layer selection reduces sampled layers by 33%.Source: 5) Dynamic Multi-Layer Aggregation · p. 4

6) Local Detail Branch and Difference-Aware Gated Fusion:

A local-detail branch processes image crops and fuses their features with global features through difference-aware gating; a three-stage cosine schedule withdraws enhancement contributions later in training.

6) Local Detail Branch and Difference-Aware Gated Fusion:

  • The 2 × 2 crop branch processes four non-overlapping regions with the final one or two Transformer blocks, then stitches their features into a global token map for gated fusion.Each crop yields an 8 × 8-pixel patch representation; the gate uses local features and their absolute difference from global features, and fusion is applied at res2 and res3.Source: 6) Local Detail Branch and… · pp. 4–5

C. Annealing-enhanced Training Paradigm

  • The three-stage cosine schedule ramps enhancement modules up, holds them at full strength, then reduces them toward identity mappings while freezing aggregation-weight gradients at annealing onset.The schedule is intended to support early representation formation and reduce reliance on auxiliary pathways near the end of training.Source: C. Annealing-enhanced Training Paradigm · p. 5

D. Two-Stage Quality-Aware Training

Quality-aware training transitions from focal loss to IoU-labeled Varifocal Loss, while backbone saliency guides decoder-query initialization.

1) Two-Stage VFL Switch:

  • A linear ramp transitions from focal loss to IoU-soft-label Varifocal Loss, using focal loss first to support stable early training amid unstable predicted-box IoUs.The switch begins at VFL START ITER and interpolates the losses over VFL RAMPUP ITERS steps.Source: 1) Two-Stage VFL Switch · p. 5

2) Saliency-Guided Query Initialization:

3) Semantic Consistency Loss:

The training objective adds a semantic-consistency constraint to the task losses, with a Hungarian matching cost combining classification, mask, dice, box, and generalized-IoU terms.

3) Semantic Consistency Loss:

  • A semantic-consistency constraint addresses possible drift in query representations after nine decoder layers.The detach operation pulls predicted features toward initialized features without allowing the initialization target to change under this loss.Source: 3) Semantic Consistency Loss · p. 6

E. Training Objective

  • The total training loss sums classification, box, mask, dice, denoising, and weighted semantic-consistency losses.Hungarian matching combines classification, mask, dice, box, and generalized-IoU costs.Source: E. Training Objective · p. 6

IV. EXPERIMENTS

The experiments evaluate SAGE on four benchmarks and compare its COCO detection and segmentation results with listed methods, including same-scale and larger alternatives.

A. Datasets and Evaluation Metrics

  • The evaluation covers COCO detection, instance segmentation, and keypoints; ADE20K semantic and instance segmentation; TinyImageNet classification; and DIOR-R oriented detection.Results use single-scale inference without test-time augmentation.Source: A. Datasets and Evaluation Metrics · p. 6

B. Implementation Details

  • Training uses 100K iterations on 64 Huawei Ascend 910B1 NPUs, with AdamW, a 10^-4 learning rate, 0.05 weight decay, and WarmupPolyLR.The SPM produces four feature scales at strides 4, 8, 16, and 32; MSDeformAttn uses four attention heads.Source: B. Implementation Details · p. 6

1) Comparison with SOTA Methods:

  • 62.8 box AP and 55.3 mask AP on COCO val2017 are the strongest results among the methods listed for the ViT-L configuration.The comparison is reported in Table I.Source: 1) Comparison with SOTA Methods · p. 6
  • SAGE ViT-B improves box AP by 1.00 and mask AP by 4.50 over RF-DETR-2XL in the listed same-scale, same-pretraining-data comparison.The passage cautions that other training and implementation details should also be matched before attributing the gains solely to architecture.Source: 1) Comparison with SOTA Methods · p. 6
  • SAGE-B has fewer than half Co-DETR with Swin-L’s backbone parameters and uses about 2M versus 14M pretraining images, while achieving 0.40-point higher box AP.Differences in training and implementation mean the comparison does not isolate any single design choice.Source: 1) Comparison with SOTA Methods · p. 7

2) Ablation Study: Module Contributions:

Ablations report contributions from the enhancement modules and training schedules, while cross-task evaluations show competitive performance and identify limits on strict comparisons.

2) Ablation Study: Module Contributions:

  • The full model gains 3.40 box AP over baseline; saliency modulation adds +1.20, annealing +1.17, layer/detail fusion +0.84, and VFL/consistency +0.19 box AP.The DINO saliency prior supplies an annotation-free foreground/background signal; layer/detail fusion aggregates layer features and incorporates fine edges and textures through 2 × 2 sub-crops.Source: 2) Ablation Study: Module Contributions · p. 7

3) Ablation Study: Annealing Strategy:

  • Three-stage cosine annealing reaches 60.91 box AP versus 60.15 without annealing, outperforming the two-stage 59.21 and linear 60.09 settings.The results support the tested schedule but do not by themselves establish the proposed internalization mechanism.Source: 3) Ablation Study: Annealing Strategy · p. 7

4) Ablation Study: Two-Stage VFL:

  • Annealing raises AP by 0.76 under two-stage VFL but lowers it by 0.60 when VFL starts from scratch, yielding a +1.36 AP interaction.The estimated main effects are +0.08 AP for annealing and +0.14 AP for VFL strategy.Source: 4) Ablation Study: Two-Stage VFL · p. 7

D. Universality of the Enhanced Backbone

  • SAGE-L reaches 78.5 AP on COCO keypoints versus ViTPose-H’s 79.1, while SAGE-B’s 58.7 mIoU exceeds ViT-Adapter-L + UPerNet’s 58.0 on ADE20K.SAGE-L reports 60.3 mIoU versus UPerNet’s 59.9 on ADE20K; the latter uses a substantially larger backbone, and cross-paper configurations differ.Source: D. Universality of the Enhanced Backbone · pp. 7–8

V. CONCLUSION

  • SAGE-L reports strong results across six task settings, including 62.8 box AP on COCO detection and 60.3 mIoU on ADE20K semantic segmentation.The conclusion also reports 55.3 mask AP on COCO, 94.76% Top-1 accuracy on TinyImageNet, 44.9 mask AP on ADE20K, 78.5 AP on COCO keypoint estimation, and 78.8 AP50 on DIOR-R.Source: §V · p. 8

SUPPLEMENTARY MATERIAL

  • Supplementary comparisons report SAGE-L as strong or competitive across tasks, but differing backbones, pretraining data, schedules, and task components prevent strict control.Reported parameter counts and pretraining configurations provide context, not proof that comparisons are controlled.Source: SUPPLEMENTARY MATERIAL · p. 10

VI. COCO OBJECT DETECTION AND INSTANCE SEGMENTATION

SAGE reports strong results across COCO detection and instance segmentation, ADE20K semantic and instance segmentation, and DIOR-R oriented detection. The comparisons also show gains across backbone scales, with differing degrees of comparability to competing methods.

VI. COCO OBJECT DETECTION AND INSTANCE SEGMENTATION

  • 62.8 box AP and 55.3 mask AP exceed the strongest listed COCO baselines by 2.1 and 3.2 points, respectively.SAGE-B also improves box AP from 60.1 to 61.1 against RF-DETR-2XL; scaling to ViT-L raises box AP by 1.7 points and mask AP by 0.9 points.Source: §VI · p. 10

VII. SEMANTIC SEGMENTATION ON ADE20K

  • 60.3% mIoU from SAGE-L exceeds InternImage-H's 59.9% by 0.4 points, with reported backbone sizes of 304M and approximately 1.08B parameters, respectively.SAGE-B achieves 58.7% mIoU; differing pretraining datasets mean the results indicate empirical competitiveness rather than a controlled architecture comparison.Source: §VII · p. 10

VIII. INSTANCE SEGMENTATION ON ADE20K

  • 44.9 mask AP from SAGE-L exceeds the strongest listed ADE20K baseline, OneFormer + ViT-P with additional COCO data (40.7), by 4.2 points.Scaling from ViT-B to ViT-L improves mask AP by 2.0 points; unavailable competitor parameter counts and full pretraining configurations prevent a direct parameter-efficiency comparison.Source: §VIII · p. 10

IX. ORIENTED OBJECT DETECTION ON DIOR-R

  • 78.8 AP50 from SAGE-L exceeds RiO-DETR-X's 77.43 AP50 by 1.37 points, while SAGE-B also exceeds all listed baselines.Scaling from ViT-B to ViT-L improves AP50 by 0.5 points; differing backbone sizes, pretraining data, and task-specific components prevent a strictly controlled parameter-efficiency comparison.Source: §IX · p. 10

X. IMAGE CLASSIFICATION ON TINYIMAGENET

SAGE reports competitive results on TinyImageNet classification and COCO keypoint detection. The comparisons include a strongest reported classification result and keypoint performance close to a larger-backbone baseline, but training and pretraining differences limit strict attribution.

X. IMAGE CLASSIFICATION ON TINYIMAGENET

  • 94.76% Top-1 accuracy from SAGE-L exceeds CLAdapter's 94.21% by 0.55 percentage points on TinyImageNet.SAGE-L also reports 99.11% Top-5 accuracy, compared with 98.50% for Swin-L/4; SAGE-B reports 91.46% Top-1 accuracy with 86M parameters.Source: §X · p. 10
  • The OCD classification result uses five models, and differences in pretraining data and training procedures prevent attributing gains solely to the backbone.The collected classification results identify SAGE-L as strongest, but the ensemble is not directly comparable to single-model results.Source: §X · p. 11

XI. KEYPOINT DETECTION ON COCO

  • 78.5 AP on COCO keypoint detection from SAGE-L is close to ViTPose-H's 79.1 AP, whose backbone has 631M parameters versus SAGE-L's 304M.SAGE-B achieves 76.0 AP, exceeding the 75.8 AP reported for both ViTPose-B and RTMPose with CSPNeXt-L.Source: §XI · p. 11
  • Keypoint baselines differ in pretraining data and parameter count, so these results do not establish an advantage under strictly matched conditions.SAGE-L approaches the reported ViTPose-H result with roughly half as many backbone parameters.Source: §XI · pp. 11–12

Related papers

Loading 2610.36719v1…