BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search

1. Introduction

• Manual design of well-optimized hybrid architectures becomes challenging as the number of design choices increases.NAS reduces human effort by automatically searching a predefined space of architectures.Source: §1 · pp. 1–2
• Layer-level weight-sharing spaces grow exponentially with network depth, while teacher-supervised block-wise methods can rank candidates in ways correlated with the teacher architecture.This teacher preference limits their application to search spaces with disparate candidates such as CNNs and transformers.Source: §1 · p. 2
• BossNAS uses ensemble bootstrapping to train each supernet block self-supervised, then evaluates architectures against the population probability ensemble.The method is proposed to address inaccurate predictive rankings while avoiding bias from a teacher model.Source: §1 · p. 2
• HyTra places CNN and transformer blocks of different resolutions in parallel, with searchable downsampling positions.Its search space covers pure transformers with fixed content length and CNNs with progressively reduced spatial scales.Source: §1 · p. 2
• 0.78 and 0.76 Spearman correlations on MBConv with ImageNet and NATS-Bench size with CIFAR-100, respectively, surpass state-of-the-art NAS methods.The authors report that BossNAS generalizes across three search spaces and datasets, and outperforms its supervised counterpart on HyTra.Source: §1 · p. 2
• 82.5% ImageNet accuracy for BossNet-T surpasses EfficientNet by 2.4% with comparable compute time.The paper presents HyTra as a case study for diversified hybrid architectures.Source: §1 · p. 2

2. Related Works

• Block-wise weight-sharing NAS factorizes the supernet into independently optimized blocks to reduce the weight-sharing space and inaccurate architecture ratings.Prior methods use supervised distillation, which can introduce teacher architectural bias; BossNAS instead proposes a block-wise self-supervised scheme.Source: §2 · p. 2
• BossNAS applies probability-ensemble bootstrapping to Siamese supernets, unlike prior unsupervised NAS based on pretext tasks or random labels.The related work describes BYOL and SimSiam as predicting one view's representation from another without negative examples.Source: §2 · p. 3
• HyTra uses a fabric-like network-level structure with layer-level granularity, whereas cell-based spaces search repeatable cells within a manually designed network-level architecture.Network-level spaces can also search macro structure using manually designed building blocks.Source: §2 · p. 3

3. Block-wisely Self-supervised NAS

• Block-wise factorization reduces the weight-sharing space while retaining the original search-space size, addressing inaccurate ratings from entangled, incompletely optimized shared weights.Earlier block-wise methods train blocks separately and combine their evaluation losses, but teacher-generated supervision can bias candidate ratings toward the teacher architecture.Source: §3.1 · p. 3

3.2. Training with Ensemble Bootstrapping

• Ensemble bootstrapping trains each block of Siamese supernets without a fully supervised teacher, using an ensemble of sampled EMA-supernet paths as the online paths’ common target.This common objective addresses the convergence difficulty that arises when each sampled sub-network learns only from its own moving average.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3. Searching Towards the Population Center

• Architecture rating uses a probability ensemble of the evolving population as a common target, with fixed validation views to avoid bias from variable augmentation.The method seeks an unsupervised rating because contrastive-learning losses need not represent architecture performance or convergence, and sampled target networks may be biased.Source: §3.3 · pp. 4–5
• For medium-sized blocks, such as four layers with four candidates each, all candidates can be evaluated by traversal and the block search can finish in one step.The example contains 256 architectures.Source: §3.3 · p. 5

4.1. CNN and Transformer Candidate Blocks

• HyTra combines residual bottleneck ResConv blocks with lightweight ResAtt transformer blocks based on BoTBlock and NLBlock.ResAtt replaces BoTBlock’s relative position encoding branch with a depthwise separable convolution, reducing that module’s complexity from O(CW^3) to O(CW^2).Source: §4.1 · p. 5
• ResAtt places implicit position encoding between input projection and self-attention, and the same modification in ResConv enables weight sharing across down-sampling rates.The position-encoding module also performs down-sampling; the candidate blocks are designed for comparable computational complexity.Source: §4.1 · p. 5

4.2. Fabric of Hybrid CNN-transformers

• At each choice layer, HyTra can retain spatial resolution or halve it until the smallest scale, forming a fabric that includes CNN, transformer, and hybrid architectures.Its flexible down-sampling positions accommodate CNNs’ changing spatial scales and transformers’ typically fixed sequence length.Source: §4.2 · p. 5, §4 · p. 5

5. Experiments

• Experiments cover HyTra, MBConv, and NATS-Bench SS, using ImageNet, CIFAR-10, and CIFAR-100; each supernet block is trained for 20 epochs with one warm-up epoch.Four paths are sampled per training step.Source: §5 · p. 6

5.1. Searching for Hybrid CNN-transformer

• BossNet-T0 reaches 80.5% top-1 accuracy, beating BoTNet50 by 2.2% while 1.19× faster; with SE and SiLU, it reaches 80.8%, beating EfficientNet-B1 by 1.7% while 1.14× faster.BossNet-T0 also improves by up to 6.0% over manually selected HyTra models, and gains 0.5% accuracy over DNA-T.Source: §5.1 · p. 6
• 82.5% top-1 accuracy on 256×256 input: BossNet-T1↑ surpasses T2T-ViT-19 by 0.6% and EfficientNet-B2 by 2.4% with comparable runtime.This larger-input evaluation uses no finetuning.Source: §5.1 · p. 6
• DNA-T selects 13 ResConv and three ResAtt blocks, whereas BossNet-T0 has similar numbers of each and higher accuracy; the authors attribute DNA-T’s preference to teacher supervision.Manual BoT50-style construction with the proposed blocks also beats the BoT50 prototype by 1.2% top-1 accuracy with 1.17× compute-time reduction.Source: §5.1 · p. 6

5.2. Results on MBConv Search Space

• 0.65 Kendall τ: BossNAS has the highest MBConv architecture-rating accuracy among the compared NAS methods, exceeding DNA by 0.03 τ and MnasNet by 0.07 R with 28.8× acceleration.BossNet-M1 beats SPOS and RLNAS by 1.4% and 0.6%, respectively; BossNet-M2 beats EfficientNet by 1.1%.Source: §5.2 · p. 7, Table 3 · p. 7

5.3. Results on NATS-Bench SS

• On CIFAR-100, BossNAS beats a predictor-based method trained with ground-truth architecture accuracies by 0.16 τ and 0.19 R, without ground-truth architecture accuracies or human labels.The NATS-Bench SS experiments cover CIFAR-10 and CIFAR-100 and rate all 32,768 architectures.Source: §5.3 · pp. 7–8, Table 4 · p. 7

5.4. Ablation Study

• Replacing ensemble bootstrapping with naive bootstrapping drops architecture-rating accuracy by 0.53 τ, while unsupervised evaluation beats supervised linear evaluation by 0.1 τ.The ablations compare training schemes and evaluation methods on MBConv and ImageNet.Source: §5.4 · p. 8, Table 5 · p. 8

5.5. Convergence Behavior

• On MBConv with ImageNet, architecture-rating accuracy rises rapidly early in training, converges at epoch 12, and remains stable through the 20-epoch run.The authors interpret this as stable rating ability and fast convergence.Source: §5.5 · p. 8, Figure 7 · p. 8

6. Conclusion

• 82.5% ImageNet accuracy for BossNet-T surpasses EfficientNet by 2.4% with comparable compute time.The conclusion also reports that ablations found ensemble bootstrapping and the unsupervised evaluation metric both crucial.Source: §6 · p. 8
• BossNAS addresses inaccurate architecture ratings from large weight-sharing spaces while avoiding architectural bias from supervised distillation.The method combines ensemble bootstrapping with an unsupervised evaluation metric; experiments span three search spaces.Source: §6 · p. 8
• HyTra is a fabric-like search space for hybrid CNN-transformer architectures.On this search space, the paper evaluates BossNet-T.Source: §6 · p. 8

A.1. A brief review of NAS

• NAS optimizes architectures by exploring search spaces with search algorithms and evaluating candidates through rating schemes.The review groups NAS methods by their rating scheme into multi-trial and weight-sharing approaches.Source: §A.1 · p. 12
• Multi-trial NAS trains sampled architectures from scratch, making evaluation computationally prohibitive and difficult to deploy on large datasets.The review notes that some methods instead use smaller datasets or only the first few ImageNet training epochs.Source: §A.1 · p. 12
• Weight-sharing NAS trains a supernet encoding the search space, then rates candidates using inherited weights; one-shot methods train the supernet before searching with frozen weights.The paper uses “one-shot” for two-stage weight-sharing methods, rather than the broader weight-sharing category.Source: §A.1 · p. 12

A.2. Implementation Details

• HyTra has 16 choice layers with ResConv and ResAtt candidates, searchable downsampling, and roughly 2.8×10^6 architectures.ResAtt is available only at 1/16 and 1/32 resolution; layers may keep their scale or downsample until reaching 1/32.Source: §A.2 · p. 12
• The MBConv space has 18 layers with four MobileNet-block choices per layer, yielding about 6.9×10^10 architectures.Choices combine kernel sizes 3 or 5 with reduction rates 3 or 6.Source: §A.2 · p. 12
• NATS-Bench SS has 32,768 channel configurations across five layers, with the supernet divided into three blocks by spatial size.Each layer selects from eight channel counts, 8 through 64; the second and fourth layers downsample by a rate of 2.Source: §A.2 · pp. 12–13
• ImageNet training uses unlabeled NAS-train images, with 50,000 training images reserved for NAS validation; searched models are retrained from scratch and evaluated on validation data.CIFAR splits follow NATS-Bench, and final searched-model accuracies are queried from NATS-Bench SS.Source: §A.2 · p. 13
• BossNAS trains each block for 20 epochs on ImageNet and 30 on CIFAR, sampling four paths per step for ensemble bootstrapping.Training uses one linear warm-up epoch, LARS, cosine learning-rate decay, and batch size 4096.Source: §A.2 · p. 13
• BossNet-T retraining follows DeiT with AdamW, while BossNet-M follows EfficientNet with RMSprop.BossNet-T uses batch size 1024 and weight decay 0.05; BossNet-M uses batch size 4096 and a learning rate decayed by 0.97 every 2.4 epochs.Source: §A.2 · p. 13
• For DNA, the supernet is divided into four four-layer blocks trained for 20 epochs, with pooled and projected student and teacher features used for distillation.UnNAS uses rotation prediction and three initial stride-2 convolution layers, training the supernet for two epochs.Source: §A.2 · p. 13

A.3. Additional Analysis on NATS-Bench SS

• 0.11 and 0.16 τ: BossNAS outperforms CE on architecture-rating accuracy on CIFAR-10 and CIFAR-100, respectively.The comparison evaluates 3,000 randomly sampled NATS-Bench SS architectures.Source: §A.3 · p. 13
• BossNAS architectures form denser, more spindly rating-versus-true-accuracy scatter patterns than CE on both CIFAR datasets.Figure 8 compares the rating and true accuracy of sampled NATS-Bench SS architectures.Source: §A.3 · p. 13
• 0.76 ρ: BossNAS reaches this CIFAR-100 architecture-rating accuracy within 20 epochs and remains stable for 10 more.The reported convergence is smooth and quick, with minor fluctuations during training.Source: Figure 10 · p. 14, Figure 9 · p. 14

• Figure 10 illustrates HyTra versions of ResNet50-T, ViT-T/16, and BoTNet50-T that follow their prototypes as closely as possible.Source: §A.4 · p. 14