PaperlayerView exact revision on arXiv

Source-linked AI summary

ProgResViT: Progressive Resolution and Width for Adaptive Vision Transformers

Ali Hojjat, Janek Haberer, Olaf Landsiedel

arXiv:2609.03216v1cs.CV

TL;DR

ViTs typically spend the same computation on every image, although images differ in difficulty and required spatial detail. ProgResViT progressively increases resolution and width, reusing features and conditioning shared blocks on stage. It improves accuracy–compute trade-offs across reported tasks, with the distilled model reaching 84.9% top-1 accuracy.

  • Problem

    Fixed-resolution, fixed-width ViTs apply the same cost despite variation in image difficulty and required spatial detail, while existing adaptive methods generally vary one axis or use separately optimized operating points.

    Source: §1 · p. 1

  • Method

    ProgResViT progressively processes images at higher resolutions and widths, reuses projected features across rounds, and uses PSG to condition shared-block operations on inference stage.

    Source: §1 · p. 1, Progress-Conditioned Soft Gating (PSG) · p. 3

  • Results

    ProgResViT improves ImageNet-1K accuracy–compute trade-offs over adaptive-width, adaptive-depth, and dynamic-token baselines, reaches 84.9% top-1 accuracy when distilled, and transfers effectively to DINO representations and ADE20K segmentation.

    Source: §6 · p. 7

  • Takeaways & Limitations

    A single trained model provides a tunable accuracy–compute trade-off by varying the inference threshold rather than requiring separate models for different operating points.

    Source: §1 · p. 1

    Entropy-based routing may spend extra computation on uncertain inputs that remain misclassified after the final round, including cases where uncertainty reflects ambiguity or distribution shift rather than insufficient computation.

    Source: §5 · p. 7

Abstract

from arXiv · show

Vision Transformers (ViTs) typically process every image using a fixed input resolution and model width, even though many images can be classified with substantially less computation. We introduce ProgResViT, an input-adaptive ViT that performs inference progressively across multiple rounds. The first round processes a low-resolution image with a narrow subnetwork. Inference terminates when the prediction is sufficiently confident; otherwise, the model reuses the representations produced in the current round and proceeds with a higher-resolution input and a wider subnetwork to refine its prediction. As all rounds share a single backbone, we propose Progress-Conditioned Soft Gating (PSG), which conditions token fusion and layer outputs on the current round, block, and input resolution. On image classification, applying ProgResViT to DeiT yields better accuracy-compute trade-offs than adaptive-width, adaptive-depth, and dynamic-token baselines. With knowledge distillation, a DeiT-based ProgResViT achieves 84.9% top-1 accuracy, slightly exceeding the reported DeiT-III-S accuracy under a comparable evaluation setting. We show that the same design also provides favorable accuracy-compute trade-offs for self-supervised DINO representations and downstream semantic segmentation. Code is available at https://github.com/ds-kiel/ProgResViT.

1 Introduction

ProgResViT addresses fixed-cost ViT inference by progressively increasing image resolution and model width, reusing features across rounds. Its shared backbone is conditioned on the inference stage, and the reported results improve accuracy–compute trade-offs across adaptive inference baselines and other vision tasks.

Motivation

  • ViTs use fixed resolution, token grid, depth, and width despite differences in image difficulty and the spatial detail needed for correct predictions.Clear, centered objects may be recognized coarsely, whereas cluttered or fine-grained examples require denser tokens and greater capacity.Source: §1 · p. 1

Prior work

  • Prior adaptive ViTs generally vary one computational axis at a time or use separately optimized operating points, leaving joint resolution-and-capacity adaptation in one shared model open.ThinkingViT progressively widens subnetworks but retains fixed input resolution and does not explicitly condition shared blocks on stage.Source: §1 · p. 1

ProgResViT

  • ProgResViT progressively increases resolution and active width, stopping when confidence is sufficient or reusing projected tokens with fresh higher-resolution embeddings to refine the prediction.Varying the inference threshold gives one trained model tunable accuracy–compute operating points without separate models.Source: §1 · p. 1

Results Overview

  • 82.21% top-1 accuracy at 6.27 GMACs becomes 82.18% at 4.47 average GMACs with confidence routing, and the schedule outperforms adaptive-width, adaptive-depth, and dynamic-token baselines.For the distilled 160 →384 schedule, accuracy is 84.90% at 16.15 GMACs and 84.87% at 11.12 average GMACs.Source: §1 · pp. 1–2

ProgResViT

  • PSG conditions token fusion and shared layer outputs on round, block, and resolution metadata so the reused backbone is informed of its current stage.The shared blocks otherwise process different token grids, resolutions, and active widths without an explicit stage indication.Source: Figure 1 · p. 2

Contributions

  • The paper evaluates the design in classification, distribution-shift tests, self-supervised DINO representations, and semantic segmentation.The reported design maintains frontier accuracy–compute trade-offs for DINO representation learning and semantic segmentation as well as ImageNet-1K inference.Source: §1 · p. 2

2 Related Work

Prior adaptive Transformers vary width, resolution, depth, or token processing in different ways, while ProgResViT combines progressive full-image resolution and width changes in one shared backbone. The paper reports comparable-compute gains across several tasks and a distilled classification result.

Adaptive-width Transformers

  • Adaptive-width methods embed subnetworks of different capacities in a shared parameter set, while ThinkingViT adds progressive wider subnetworks and confidence-based early termination.ProgResViT extends this paradigm with progressive resolution and width, cross-round feature projection, and stage-conditioned modulation.Source: §2 · p. 2

Resolution-flexible Transformers

  • Resolution-flexible Transformers vary patch sizes, resolutions, or aspect ratios, while adaptive multi-resolution methods allocate more spatial computation to uncertain inputs.Other approaches refine informative patches, localize a region for higher-resolution processing, or select coarse and fine token scales by region.Source: §2 · p. 2

Main results

  • Across DINOv1 representation learning, ADE20K segmentation, and ImageNet-1K adaptive-width inference, ProgResViT outperforms corresponding baselines at comparable compute.The distilled DeiT-S-based model reaches 84.90% top-1 accuracy, slightly exceeding reported DeiT-III-S accuracy under a comparable evaluation setting.Source: Figure 2 · p. 3

Early exit and token-adaptive inference

  • Early-exit methods adapt depth, whereas token-adaptive methods prune, halt, or selectively process spatial tokens, with AdaViT also selecting patches, heads, and blocks per input.Unlike these approaches, ProgResViT preserves the complete token grid at each round, changing token count through input resolution and computation per token through width.Source: §2 · p. 3

3 ProgResViT

ProgResViT runs shared-backbone rounds with progressively higher resolution and wider subnetworks, reusing aligned representations and conditioning block computations on inference progress. Confidence-based early stopping lets samples exit before later rounds.

ProgResViT

  • Each round traverses all D blocks of the same ViT backbone, with resolution, active attention heads, and embedding width specified per round.The first round produces a complete prediction; the second uses a higher-resolution input and wider operating point.Source: Adaptive-Width Progressive Rounds · p. 3
  • Later rounds resize and re-embed the image, then process a wider channel prefix after mixing in tokens from previous rounds.The shared positional embedding’s spatial component is interpolated to each round’s patch grid.Source: Adaptive-Width Progressive Rounds · p. 3
  • A cross-resolution token projector aligns earlier tokens to the next round’s token grid and width, adjusts the class token, and fuses them with fresh higher-resolution embeddings.The projector addresses the differing token counts and embedding widths across rounds.Source: Adaptive-Width Progressive Rounds · p. 3
  • PSG conditions shared Transformer blocks on inference progress by applying channel-wise multipliers to attention updates, MLP updates, and block outputs.Each block serves subnetworks with different resolutions and active widths across successive rounds.Source: Progress-Conditioned Soft Gating (PSG) · p. 3
  • The 192→240 resolution schedule offers the best trade-off among the schedules shown in Figure 3a.The figure compares two-round variants using 3→6 heads with a three-round 2→4→6-head variant.Source: Figure 3 · p. 4
  • Entropy routing exits samples whose top-10 class-probability entropy falls below a threshold; varying the threshold traces an accuracy–compute frontier without retraining.Training jointly optimizes all rounds on every sample; the authors favor entropy routing over a learned router whose accuracy gain is at most 0.21 percentage points at matched continuation rates.Source: Training and Inference · p. 4

4 Experiments Setup

Experiments evaluate ProgResViT’s accuracy–compute trade-offs, routing, and transfers beyond supervised classification. Results include stronger comparisons with adaptive baselines and analyses of progressive refinement and model construction.

Resolution and width scheduling

  • The tested 192→240 resolution and 3→6 head schedule provides the strongest overall trade-off; the 160→384 schedule requires more computation but gives the highest final accuracy.The study uses DeiT-S on ImageNet-1K and also evaluates the same model without fine-tuning on five ImageNet variants.Source: Accuracy–Efficiency Trade-offs for… · pp. 4–5, Setup · p. 4

Accuracy–Efficiency Trade-offs for classification

  • 84.90% top-1 accuracy at 16.15 GMACs slightly exceeds DeiT-III-S’s reported 84.80% at 15.5 GMACs; 192→240 entropy routing retains 82.18% at 4.47 GMACs.The 192→240 full two-round result is 82.21% at 6.27 GMACs; routing reduces computation by 28.7%.Source: Accuracy–Efficiency Trade-offs for… · p. 5

Distribution shifts

Adaptive-width baselines

  • ProgResViT outperforms adaptive-width baselines and ThinkingViT across operating points, with gains up to 0.74 percentage points over ThinkingViT; it also provides the strongest high-accuracy frontier among dynamic-token and early-exit baselines.The paper also reports higher accuracy at comparable throughput on one NVIDIA L40 at batch size 512.Source: Accuracy–Efficiency Trade-offs for… · p. 6, Figure 6 · p. 6

Transfer Beyond Supervised Classification

  • ProgResViT outperforms fixed-resolution DINO ViT-S/16 baselines under 20-NN evaluation and outperforms ThinkingViT and separately trained DeiT-Tiny, Small, and Base segmentation models.The segmentation setup uses one shared linear decoder across rounds.Source: Transfer Beyond Supervised… · p. 6

Progressive Refinement Ablation of ProgResViT’s construction

  • Adding resolution progression raises full-path accuracy from 81.44% to 81.74%, and adding PSG raises it to 82.20% at nearly the same GMACs.The ablation compares a fixed-resolution width baseline, resolution progression, and then PSG.Source: Progressive Refinement · p. 6

Round 1 Provides Useful Context for Round 2

  • At approximately 87% round-1 blurring, round-2 accuracy drops by 1.74%; replacing or shuffling patch tokens reduces round-2 accuracy and correct-class confidence.Replacing only the class token has the smallest effect, especially when taken from an image of the same class.Source: Progressive Refinement · p. 7

Progressive Representation Refinement Across Rounds

  • Round 2 refines round-1 representations: CLS similarity continues rising, and class-mean CLS trajectories separate further after token fusion.Patch-token similarity briefly decreases at the transition as fusion introduces fresh higher-resolution tokens; PSG multipliers also vary across channels, depth, and rounds.Source: Analysis of Progress-Conditioned Soft… · p. 6, Progressive Refinement · p. 7

5 Limitations

The entropy-based router can spend additional computation on uncertain inputs that remain misclassified after the final round.

Limitations

  • Uncertainty from ambiguity or distribution shift can trigger extra computation without correcting the prediction; rejection mechanisms could reduce this, but were excluded to keep baseline comparisons fair.The limitation concerns uncertain inputs that remain misclassified after the final round.Source: §5 · p. 7

6 Conclusion

ProgResViT progressively increases resolution and active width while reusing features across rounds, with a projector aligning representations between stages. The conclusion reports improved classification trade-offs and transfer to DINO and semantic segmentation.

6 Conclusion

  • ProgResViT improves ImageNet-1K accuracy–compute trade-offs over adaptive-width, adaptive-depth, and dynamic-token baselines; its distilled model reaches 84.9% top-1 accuracy and the design transfers to DINO and ADE20K segmentation.The model reuses features across rounds while progressively increasing input resolution and active width.Source: §6 · p. 7

A Token Projector Pipeline

  • The token projector resizes patch tokens to the next spatial grid, refines and expands their channels, and separately expands the class token before concatenating the tokens.For the default 192²→240² model, it maps a 12×12 grid with 192 channels to a 15×15 grid with 384 channels.Source: A Token Projector Pipeline · p. 11

B PSG Architecture

PSG conditions reused Transformer blocks on inference progress and resolution, producing separate channel-wise gates for round-transition fusion and block pathways.

B PSG Architecture

  • Five metadata values encode round, progress, current and previous resolutions, and their ratio for a shared conditioning encoder.The encoder produces a conditioning vector used by five branch-specific heads.Source: B PSG Architecture · p. 11
  • Separate gates scale the new-image and projected previous-round streams during fusion, and modulate attention, MLP, and complete block outputs.The code also applies gates to attention and MLP updates within Transformer blocks.Source: B PSG Architecture · p. 11
  • PSG’s conditioned heads and attention and MLP scales are initialized as identity transformations.The conditioned heads’ final linear layers are zero-initialized, while PSG-specific attention and MLP scales are initialized to ones.Source: B PSG Architecture · p. 11

C Entropy Routing Versus a Learned Router

The supplementary experiments compare entropy routing with a learned router and examine deployment, distillation, and baseline operating points. Learned routing offers localized gains, while ProgResViT retains near-full accuracy at reduced computation.

C Entropy Routing Versus a Learned Router

  • 0.214 percentage points is the learned router’s largest measured gain when 20% of images continue; gains shrink at higher continuation rates.At 50% continuation the gain is 0.036 points, at 60% it is 0.012 points, and there is no consistent advantage thereafter; entropy requires no auxiliary training set or learned parameters.Source: C Entropy Routing Versus a Learned… · p. 11, Table A1 · p. 12

D Fixed-Resolution Deployment Context

E Knowledge-Distillation Details

  • Distilled variants train for 300 epochs using hard-logit distillation from a frozen DeiT-III-B/384 teacher, which supplies hard pseudo-labels.The teacher receives no gradients, and the hard class-label target uses no temperature parameter.Source: E Knowledge-Distillation Details · p. 12

F Near-Lossless Routing Points

  • Routed operating points retain accuracy within 0.03 percentage points of full-path inference while reducing average computation by 21.9–31.1%.The selected point is the lowest-compute entropy threshold meeting that accuracy condition.Source: F Near-Lossless Routing Points · p. 12, Table A2 · p. 12

H Deployment Memory

  • Figure S3 reports peak allocated GPU memory measured at batch size 128 on a single NVIDIA L40, using the same execution setup as the throughput measurements.Source: H Deployment Memory · p. 12

I Dynamic-Token and Early-Exit Baselines

  • The dynamic-token and early-exit comparisons include multiple named baselines, with DeiT-S as a fixed reference for the dynamic-token comparison.Connected points represent operating points from one trained model, whereas isolated markers represent separately trained configurations.Source: I Dynamic-Token and Early-Exit Baselines · pp. 12–13, Table A3 · p. 13

G Comparison with Resolution-Flexible Inference

  • 83.766% at 4.462 GMACs for the 192²→240² model and 84.902% at 16.152 GMACs for the 160²→384² model are reported against FlexiViT’s 83.2% at 15.39 GMACs.The comparison varies ProgResViT entropy thresholds and uses supplied FlexiViT resolution-flexible operating points.Source: G Comparison with Resolution-Flexible… · p. 12, Figure S2 · p. 13

J DINO Linear Probing

ProgResViT is evaluated with frozen DINO encoders and with entropy-based routing for semantic segmentation.

K ADE20K Segmentation Routing

  • For segmentation, the image-level routing score averages the entropies of the most uncertain 10% of patch predictions.Images above the threshold continue to round 2 for higher-resolution, wider processing; varying the threshold controls the share receiving round-2 computation.Source: K ADE20K Segmentation Routing · p. 13

J DINO Linear Probing

  • At comparable compute, ProgResViT outperforms separately trained fixed-resolution ViT-S/16 baselines at 224² and 240² in DINO linear probing.ProgResViT and DINO baselines are trained for 100 epochs.Source: J DINO Linear Probing · p. 13, Figure S4 · p. 14

L Activation-Replacement Accuracy

Activation-matched Gaussian-noise replacement tests how round-1 representations contribute to round-2 predictions. Replacing those activations lowers accuracy and correct-class confidence.

L Activation-Replacement Accuracy

  • 82.206% accuracy without replacement falls to 79.610% at full replacement of round-1 block-12 activations.The intervention replaces reused activations with Gaussian noise matched to their mean and standard deviation, isolating the information carried by the representation.Source: Figure S5 · p. 14, L Activation-Replacement Accuracy · p. 14

M Activation-Replacement Confidence

  • Increasing replacement strength shifts the round-2 correct-class confidence distribution downward.This complements the accuracy reduction and indicates that removing information from the reused representation weakens prediction certainty.Source: Figure S6 · p. 14, M Activation-Replacement Confidence · p. 14

N Analytical Compute Decomposition

The shared path’s conditioning adds little computation, while PSG varies across branches and depth. Three-round progression broadens routing options, and DINO attention maps show shared heads refining alongside newly activated heads.

N Analytical Compute Decomposition

  • 0.002 GMACs for block PSG in round 1; the cross-round transition adds 0.02 GMACs, and cumulative two-round inference costs 6.27 GMACs.The ViT path accounts for 0.91 of round 1’s 0.91 GMACs total; the transition costs 0.02 GMACs for token projection and 0.0001 for fusion gates, while round 2 costs 5.34 GMACs.Source: N Analytical Compute Decomposition · pp. 14–15

O PSG Specialization Across Depth

P Three-Round Progression

  • 82.130% top-1 accuracy at 7.097 GMACs for three-round inference; entropy routing spans 0.188–7.097 average GMACs and 57.626–82.130% accuracy.Its peak is slightly below the default two-round model’s 82.206%, but the extra exit supports finer routing when input difficulty is heterogeneous.Source: P Three-Round Progression · p. 15, Table A5 · p. 15

Q Attention Maps Across Progressive Rounds

  • Figure S8 shows shared heads refining their round-1 attention at higher resolution while heads 4–6 capture complementary patterns.Round 1 uses the first three heads at low resolution; round 2 retains that ordered prefix and activates three more heads.Source: Q Attention Maps Across Progressive… · p. 15, Figure S8 · p. 16

Related papers

Loading 2609.03216v1…