PaperlayerView exact revision on arXiv

Source-linked AI summary

Sliced Recursive Transformer

Zhiqiang Shen, Zechun Liu, Eric Xing

arXiv:2111.05297v3cs.CVcs.AIcs.LG

TL;DR

Vision transformers face a parameter and computational-cost trade-off, motivating better parameter utilization without increasing model size. SReT shares weights through recursive operation and approximates global self-attention with sliced group self-attentions, achieving competitive or superior results with fewer parameters while reducing recursion-related cost.

  • Problem

    Vision transformers require many parameters and substantial computation, creating a challenge in balancing accuracy with model complexity.

    Source: §1 · p. 2

  • Method

    SReT shares transformer weights across recursive layers and uses sliced group self-attentions to approximate global self-attention with lower recursion cost.

    Source: §1 · pp. 2–3, §3 · p. 5

  • Results

    SReT outperforms state-of-the-art methods across evaluated scenarios while using fewer parameters.

    Source: §1 · p. 3, §6 · p. 14

  • Takeaways & Limitations

    Recursive operation offers a parameter-efficient and broadly compatible design direction for efficient transformers.

    Source: §1 · p. 3

    The FLOPs of recursive and non-recursive forms are not identical because self-attention includes additional operations that recursion multiplies.

    Source: §4 · p. 7

Abstract

from arXiv · show

We present a neat yet effective recursive operation on vision transformers that can improve parameter utilization without involving additional parameters. This is achieved by sharing weights across the depth of transformer networks. The proposed method can obtain a substantial gain (~2%) simply using naive recursive operation, requires no special or sophisticated knowledge for designing principles of networks, and introduces minimal computational overhead to the training procedure. To reduce the additional computation caused by recursive operation while maintaining the superior accuracy, we propose an approximating method through multiple sliced group self-attentions across recursive layers which can reduce the cost consumption by 10~30% with minimal performance loss. We call our model Sliced Recursive Transformer (SReT), a novel and parameter-efficient vision transformer design that is compatible with a broad range of other designs for efficient ViT architectures. Our best model establishes significant improvement on ImageNet-1K over state-of-the-art methods while containing fewer parameters. The proposed weight sharing mechanism by sliced recursion structure allows us to build a transformer with more than 100 or even 1000 shared layers with ease while keeping a compact size (13~15M), to avoid optimization difficulties when the model is too large. The flexible scalability has shown great potential for scaling up models and constructing extremely deep vision transformers. Code is available at https://github.com/szq0214/SReT.

1 Introduction

The paper improves vision-transformer parameter utilization through recursive weight sharing, then approximates recursive self-attention with sliced group attention to reduce computation. SReT targets stronger accuracy and compactness without adding parameters, reporting improvements over state-of-the-art methods with fewer parameters and FLOPs.

Results

  • SReT significantly outperforms state-of-the-art approaches while using fewer parameters and FLOPs, as illustrated by the ImageNet-1K accuracy-versus-complexity comparison.The comparison is framed as a trade-off among accuracy, model size, and complexity.Source: Figure 1 · p. 1, §1 · pp. 2–3

Motivation

  • Recursive operation reuses transformer weights across depth, strengthening feature extraction and parameter utilization without increasing model size.The motivation is that adjacent deep-transformer layers can have similar weights and activations, making reuse plausible.Source: §1 · p. 2

Method

  • Sliced group self-attention approximates global self-attention across recursive layers to reduce the computational cost of recursion while preserving representations.The method also uses spatial-pyramid design to reduce network complexity.Source: §1 · p. 3

Trade-off

  • Recursive operation introduces additional computation through repeated loops, motivating the sliced approximation for more efficient training.The paper distinguishes parameter compactness from execution efficiency.Source: §1 · p. 3

Contributions

  • The study contributes design principles and comparisons for recursive transformer variants, including computational-equivalence analysis and modified distillation.These principles are intended to inform compact-transformer design and optimization.Source: §1 · p. 3

Evaluation

  • The approach is evaluated across vision transformers, all-MLP transformer variants, and neural machine translation, outperforming state-of-the-art methods with fewer parameters.The claimed cross-scenario evaluation extends beyond image classification.Source: §1 · p. 3

2 Related Work

Prior work established transformers for language and vision, while recursive operations had been explored in both areas. This paper focuses on using recursion with grouped self-attention to build compact, efficient vision transformers.

Vision Transformers

  • Vision Transformer and its extensions established self-attention-based image classification, but many designs target accuracy and complexity through architectural or training modifications.The related methods include DeiT, spatial-pyramid, class-attention, shifted-window, and convolutional transformer variants.Source: §2 · p. 4

Recursive Operation

  • Recursive models have been studied in language and vision, but prior recursive-transformer work did not address the complexity added by recursive operation.The paper therefore approximates self-attention with multiple group self-attentions.Source: §2 · p. 4

3 Recursive Transformer

The recursive transformer reuses shared attention and feed-forward weights across depth to improve parameter utilization without adding parameters. Non-linear projections and learnable residual coefficients support nontrivial recursive representations, while sliced group attention approximates global attention with permutation.

Recursive operation

  • Weight sharing recursively applies the same transformer operations to image patch tokens across depth, increasing effective depth without additional parameters.The recursive module uses patch-token sequences as inputs and repeats shared operations across loops.Source: §3 · p. 5

Non-linear projection

  • Naive recursion can learn an identity mapping, so a non-linear projection is inserted between recursive operations to force neighboring representations to differ.The projection is implemented as an MLP-like module with a residual connection.Source: §3 · p. 5

Optimization

  • Recursive blocks use explicit backpropagation through their forward operations rather than equilibrium optimization, with loop counts controlling computation and depth.The shared recursive-block parameters receive gradients accumulated through the recursive operations.Source: §3 · p. 6

Learnable residual connection

  • Learnable coefficients on residual branches are trained jointly with the model and can improve vision-transformer performance.The coefficients are initialized to 1 and applied to the residual branches.Source: §3 · p. 6

Extremely deep transformers

  • Weight sharing enables transformers with more than 100 layers while retaining a small model and simplifying optimization at exaggerated depths.The paper presents this as an empirical property of the proposed scaling mechanism.Source: §3 · p. 6

4 Approximating Global MHSA via Multi-Group MHSA

SReT approximates global self-attention by distributing group self-attentions across recursive layers, preserving parameter count while trading FLOPs against accuracy. The analysis establishes FLOP equivalency conditions, and experiments report better accuracy than vanilla attention at similar computation.

Approximation method

  • Sliced group self-attention decomposes global self-attention across recursive layers without changing parameter count, allowing more groups to reduce FLOPs with slightly lower performance.The number of groups can vary across recursions to control the FLOPs–accuracy trade-off.Source: §4 · p. 7

Computational equivalency

  • When the number of recursions equals the number of groups, recursive group attention and vanilla attention have equivalent FLOPs under the stated simplifying assumptions.If the number of groups exceeds the number of recursions, group attention can use fewer FLOPs than vanilla attention.Source: §4 · p. 7

Empirical comparison

  • With similar computation, recursive group self-attention achieves better accuracy than vanilla self-attention in the ex-tiny evaluation.Table 2 reports the representation-ability comparison between global and group self-attentions.Source: §4 · p. 7, Table 2 · p. 8

Practical qualification

  • In practice, the FLOPs of global and group attention are not identical because softmax, scaling, and attention-value operations add recursive overhead.The theoretical equivalency abstracts from these extra operations.Source: §4 · p. 7

Optimization analysis

  • The recursive benefit is attributed to additional parameter updating within the same iteration, which makes learned weights more aligned with the loss function.The explanation is presented from an optimization perspective through enhanced gradient accumulation.Source: §4 · p. 8

5 Experiments

Experiments evaluate SReT across vision, all-MLP, and translation settings, alongside ablations and analyses of recursion, efficiency, and optimization. The results show improved accuracy–complexity trade-offs and broader architectural generalization.

5.2 Na¨ıve Recursion on Transformer

  • Internal recursion reaches 67.6% or 70.3% versus 67.0% for external recursion in the reported 12-layer comparison, so internal recursion is used by default.The alternatives are reported with different parameter counts and recursive settings.Source: §5.2 · p. 9

5.3 Ablation Studies

  • Soft distillation improves ImageNet-1K accuracy to 77.7% from the 77.1% baseline, while one recursive iteration is selected because further loops provide marginal gains and slower execution.The ablations also vary non-shared layer ratios, permutation designs, and group counts.Source: §5.3 · p. 10

5.4 Comparison with State-of-the-art Approaches

  • SReT-T outperforms DeiT-T by 3.8% with 15.8% fewer parameters and 15.4% fewer FLOPs, while SReT-ExT exceeds PiT-T by 1.0% with 18.4% fewer parameters.SReT-S also surpasses state-of-the-art Swin-T and T2T on accuracy, model size, and FLOPs.Source: §5.4 · p. 11, Table 5 · p. 12

5.5 All-MLP Architecture

  • The recursive operation also improves MLP-Mixer over its baseline under the same training protocol, supporting generalization beyond self-attention transformers.The comparison is shown in Fig. 6 (1).Source: §5.5 · p. 11, Figure 6 · p. 11

5.6 Neural Machine Translation

  • On WMT14 En-De, the method is favorably better than the vanilla transformer, while LRC makes training more stable despite slightly slower convergence.The improvement is less significant on the smaller IWSLT’14 De-En dataset.Source: §5.6 · pp. 11–12, Figure 7 · p. 13

5.7 Landscape Visualizations of DeiT and Our Mixed-depth SReT

  • Mixed-depth training uses recursive and non-shared branches with shared weights, providing parallel optimization and stronger supervision for the shallow branch.Landscape visualizations report DeiT-108 as more chaotic and harder to optimize than the mixed-depth network.Source: §5.7 · pp. 12–13, Figure 9 · p. 13

5.8 Analysis and Understanding

  • SReT’s spatial pyramid decreases feature-map resolution with depth, unlike DeiT’s uniform 14×14 resolution, while LRC coefficients shift from identity toward main-branch contributions during training.On MHSA, the residual branch remains dominant at completion, whereas the main branch exceeds it in FFN and NLL.Source: §5.8 · p. 14, Figure 10 · p. 14

6 Conclusion

The conclusion frames SReT as a compact recursive transformer that improves parameter utilization through shared weights and sliced group self-attention. Its supporting appendix analyzes FLOPs, implementations, architectures, datasets, and additional experiments.

6 Conclusion

  • SReT targets efficient parameter utilization with recursive operation and group self-attention approximation, and its experiments cover image classification, neural machine translation, and MLP-Mixer.The conclusion attributes performance to intensified intermediate-feature representations.Source: §6 · p. 14

Appendix

  • The appendix supplies a FLOPs proof, pseudocode, implementation settings, architecture details, and further ablations for permutation designs, group counts, LRC, response maps, and expanded depth.It also reports additional evaluations on ImageNet ReaL, ImageNetV2, WMT14 En-De, and IWSLT14 De-En.Source: Appendix · p. 15

A FLOPs Analysis

  • When the number of group attentions equals the recursive-operation count, group and vanilla self-attention have equivalent FLOPs; using more groups reduces group-attention complexity.The analysis assumes matching group counts and vector dimensions across recursive operations.Source: A FLOPs Analysis · p. 16

B More Results and Comparisons on ImageNet ReaL [8] and ImageNetV2 [43] Datasets

SReT models achieve consistent improvement over DeiT on ImageNetV2 across multiple network architectures and evaluation metrics.

C Ablation Results on Different Permutation Designs and Groups Numbers

The ablation studies examine permutation choices and group counts for sliced group self-attention, selecting a design that balances accuracy and computational cost.

C Ablation Results on Different Permutation Designs and Groups Numbers

  • [8,2][4,1][1,1] is selected as the final SReT structure because it provides the best accuracy-computational-cost trade-off.The design uses two recursion loops in each recursive block and specifies group counts across spatial-pyramid stages.Source: C Ablation Results on Different… · p. 17, Table 7 · p. 18
  • An inverse permutation layer preserves input-order information after sliced group self-attention, with permutation variants evaluated using SReT-T and SReT-TL.The ablation denotes permutation, inverse permutation, and no permutation in the final stage with one group.Source: C Ablation Results on Different… · p. 17, Figure 11 · p. 18

D Pseudocode for Sliced Group Self-attention

E Training Details on ImageNet-1K

The appendix specifies ImageNet-1K training, language-model settings, SReT architecture notation, and the all-MLP backbone configuration.

E Training Details on ImageNet-1K

  • ImageNet-1K experiments use conventional training, soft-label distillation, and higher-resolution fine-tuning, with settings detailed in Tables 8–10.The default batch sizes are 512/1024 and the default initial learning rate is 1e-3; a 2e-3 rate with 30 warmup epochs can improve accuracy.Source: E Training Details on ImageNet-1K · p. 18, Table 8 · p. 19, Table 9 · p. 19
  • Soft-label distillation without one-hot labels consistently outperforms DeiT's hard-distillation scheme under the described framework.RegNetY-16GF is used as the teacher for distillation from scratch.Source: E Training Details on ImageNet-1K · p. 19
  • The vision design removes the class and distillation tokens, uses global average pooling, and sets fine-tuning weight decay to 0.These modifications slightly save parameters, boost performance, and simplify the framework.Source: E Training Details on ImageNet-1K · p. 20, Figure 13 · p. 20

F Hyper-parameter Settings of Language Models

G Details of Our SReT Architectures

  • SReT architecture notation represents non-shared blocks, recursive operations, and hidden-feature dimensions through A, B, C, and D parameters.The architecture details are provided for SReT-T, SReT-TL, SReT-S, and SReT-B.Source: G Details of Our SReT Architectures · p. 21

H All-MLP Structure

  • The all-MLP Mixer backbone uses 12 layers, 16×16 patches, hidden size 768, sequence length 196, and MLP dimensions 3072 and 384.Source: H All-MLP Structure · p. 21

I Ablation Study on Different LRC Designs

The LRC ablation favors learnable coefficients on both branches, while response-map visualizations indicate that recursion and spatial pyramids produce hierarchical representations.

I Ablation Study on Different LRC Designs

J Observations of Response Maps

  • Recursive operations produce hierarchical features within blocks, with shallow layers capturing details and deeper layers focusing on semantic information.The observed organization resembles CNN-style feature hierarchies and is attributed to recursion combined with spatial-pyramid design.Source: J Observations of Response Maps · p. 22

K More Evolution Visualization of LRC Coefficients on ImageNet-1K Dataset

Coefficient visualizations show distinct training behavior for recursive vision-transformer blocks and language models. In vision, identity-related coefficients remain near one while branch contributions settle into depth-specific proportions; language-model coefficients are more stable and symmetric.

K More Evolution Visualization of LRC Coefficients on ImageNet-1K Dataset

  • Recursive vision-transformer blocks keep α near one, while identity-mapping coefficients γ and ζ remain near one during training.The identity branch therefore passes the original signal with small scaling, while branch contributions maintain a depth-specific proportion.Source: K More Evolution Visualization of LRC… · p. 23

L Evolution Visualization of LRC Coefficients on Language Model

M More Ablation Results on Directly Enlarging Depth of Baseline DeiT Model

Directly increasing DeiT depth does not improve performance, whereas parameter-sharing recursion enlarges transformer depth while obtaining additional improvement. The recursive design selectively reuses blocks and combines slicing with permutation to enable broader token interaction within a layer.

M More Ablation Results on Directly Enlarging Depth of Baseline DeiT Model

  • Deeper naïve DeiT adds no performance gain, whereas parameter-sharing recursion enlarges depth and provides extra improvement.The comparison is presented in Table 14; the recursive model avoids relying on a heavier network with harder-to-learn intermediate features.Source: M More Ablation Results on Directly… · p. 24, Table 14 · p. 24

Feed-forward Networks, Recurrent Neural Networks and Recursive

Related papers

Loading 2111.05297v3…