Source-linked AI summary
DiD It in 87 Minutes: A Label-Free Softmax-to-Linear Adaptation of Vision Transformers for Object Detection
Huaiyuan Qin, Gabriel James Goenawan, Zihang Lin, Muli Yang, Hongyuan Zhu
TL;DR
The paper addresses label-free conversion of trained Softmax-attention ViT detectors to linear attention, where direct replacement and generic distillation fail to preserve detection behavior. It introduces Detector-Interface Distillation, which aligns detector-facing tensors from a frozen teacher while training only the linear backbone. On DOTA-v1.5, DiD is the strongest label-free method and reaches performance comparable to supervised linear-attention references, with adaptation completing in about 87 minutes on 4 GPUs.
Problem
Converting a trained Softmax-attention detector to linear attention without labels or retraining the entire detector requires preserving the feature tensors expected by its fixed downstream pipeline.
Method
Detector-Interface Distillation trains only the linear-attention backbone to align detector-facing interface tensors with those of a frozen Softmax teacher while keeping the downstream detector fixed.
Results
DiD is the strongest label-free method on DOTA-v1.5 for RetinaNet and Faster R-CNN, reaching performance comparable to supervised linear-attention training.
Takeaways & Limitations
The method provides a practical post-training route for reusing trained Softmax detectors as efficient linear-attention detectors.
Abstract
from arXiv · showhide
While linear attention is a compelling mechanism for high-resolution object detection due to its reduced cost for global token mixing, converting the Softmax-attention ViT backbone of a trained detector into a linear-attention one is not a trivial drop-in replacement. Directly swapping the attention operator leads to severe performance degradation, and generic label-free distillation, though effective for classification, often fails on detection tasks. We argue that the central challenge is \textit{detector-interface preservation}: the converted backbone must reproduce the exact feature tensors expected by the fixed downstream detector, rather than merely imitating internal Softmax hidden states. To address this, we introduce Detector-Interface Distillation (DiD), a label-free conversion method that exclusively trains the linear-attention backbone by aligning detector-facing interface tensors with those of a frozen Softmax teacher. On DOTA-v1.5, DiD substantially outperforms established baselines and matches supervised, fully trained linear models. Adaptation completes in roughly 87 minutes on 4 GPUs, and the linearized backbone cuts inference latency by ~62% and peak memory by ~49%. We hope our findings offer the community a simple, label-free route to reusing trained Softmax detectors as efficient linear ones, and encourage interface-aware objectives in future architecture-conversion work.
1 Introduction
The paper studies label-free conversion of trained Softmax-attention ViT detectors to linear-attention backbones without retraining the full detector. It proposes preserving detector-facing feature interfaces, enabling strong transfer with reduced inference cost and short adaptation.
- Motivation: Linear attention reduces global token-mixing cost, but converting an existing Softmax-attention detector remains an open label-free conversion problem.The paper frames whether a quadratic-attention backbone can be replaced without labels or retraining the entire detector.
- Challenge: Detector-interface mismatch is identified as a key failure mode: direct replacement collapses detection performance, while generic distillation mainly preserves classification behavior.Generic label-free distillation is insufficient for detection despite retaining classification behavior well.
- Method: Detector-Interface Distillation aligns detector-facing interface tensors while keeping the downstream detector fixed and updating only the linear-attention backbone.The method targets the feature contract consumed by the fixed detector rather than all backbone hidden states.
- Results: DiD substantially outperforms direct copy, generic self-supervised distillation, and hidden-state distillation, and is competitive with fully supervised linear-detector training.The controlled study reports detector-interface supervision as the strongest tested label-free approach.
- Practical payoff: 87 minutes on 4 GPUs is the reported adaptation time for the conservative 40k-step procedure, alongside substantially reduced latency and memory.The paper presents the conversion as a practical post-training adaptation procedure.
2 Related Work
Related work covers efficient attention architectures, label-free model conversion, and detector distillation. It positions DiD as conversion of a fixed detector through detector-facing feature supervision rather than conventional teacher-to-smaller-student training.
- Efficient Attention for Vision: Efficient vision transformers reduce attention cost through local, hierarchical, or sparse interactions, but long-range information may be propagated indirectly.These approaches have been effective for dense prediction while changing the architecture relative to standard global attention.
- Label-Free Model Conversion: Label-free conversion methods commonly match logits, representations, hidden states, attention behavior, or approximations of attention maps.The related-work framing treats Softmax-to-linear conversion as a distillation problem with varied supervision targets.
- Knowledge Distillation for Object Detection: Detection distillation often favors intermediate feature supervision because detectors rely on structured spatial features and imbalanced foreground-background signals.Conventional approaches generally transfer a larger teacher into a smaller student while training the detector pathway.
- Cross-task comparison: Table 1 compares native task scores and retention relative to each task’s Softmax reference across classification and detection.Its caption states that generic distillation nearly restores classification accuracy but recovers only half of DOTA-v1.5 detector mAP.
3 Problem Formulation and Diagnostic Study
Replacing Softmax attention with linear attention changes the trained backbone function, creating a conversion problem when the downstream detector remains fixed. Diagnostic experiments show that generic global-feature transfer and internal-state matching do not preserve the transformed multi-scale features consumed by the detector.
- 3.1 Label-Free Softmax-to-Linear Conversion: Linear attention reorders feature-map operations to avoid explicitly materializing token-to-token affinities, improving scaling with sequence length but changing the backbone function.The linear form uses a non-negative feature map, so parameters trained with Softmax attention are not guaranteed to preserve teacher behavior after replacement.
- 3.1 Label-Free Softmax-to-Linear Conversion: The conversion freezes the trained Softmax backbone and downstream neck and prediction modules while optimizing only a linear-attention student backbone on unlabeled images.The setup asks which teacher-student discrepancy must be minimized to keep the fixed downstream detector functional.
- 3.2 Why Is Generic Transfer Insufficient?: Direct attention replacement collapses performance across ImageNet, CIFAR-100, and DOTA-v1.5, confirming that operator substitution requires adaptation.This establishes the failure of direct copy before comparing label-free transfer objectives.
- 3.2 Why Is Generic Transfer Insufficient?: Generic distillation retains 97.5% and 97.2% of Softmax-reference accuracy on ImageNet and CIFAR-100, but only 54.0% of reference mAP on DOTA-v1.5.The comparison reports each task using its native metric and retention relative to its own Softmax reference.
- 3.3 Where Should Conversion Be Supervised?: The diagnostic study compares attention outputs, internal hidden states, backbone readouts, final hidden states, and post-neck/FPN multi-scale features as supervision targets.The controlled RetinaNet conversion on DOTA-v1.5 fixes reconstruction loss and optimization settings while changing only the supervised tensors.
- 3.3 Where Should Conversion Be Supervised?: Matching all transformer hidden states is the strongest detector-agnostic target, yet remains 5.5 mAP below supervision after the neck/FPN.Matching only the backbone layers supplied to the neck is also insufficient, so proximity to the detector alone does not guarantee compatibility.
- 3.3 Where Should Conversion Be Supervised?: The appropriate conversion target is the detector-facing boundary between the frozen neck/FPN and detector heads, where transformed multi-scale features are passed onward.This boundary is formalized as the target of Detector-Interface Distillation.
4 DiD: Detector-Interface Distillation
DiD converts a Softmax-attention detector backbone to linear attention by preserving the detector-facing multi-scale feature interface, while keeping downstream components fixed. It trains only the student backbone on unlabeled images using teacher-generated interface tensors.
- DiD aligns multi-scale tensors produced by identical frozen neck/FPN pathways while updating only the linear-attention student backbone.The teacher and student process the same unlabeled images, and the downstream detector remains fixed.
- DiD avoids matching complete internal computation traces because Softmax and linear attention use different token-mixing rules.Internal transformer states and final detector predictions are not used as distillation targets.
- Detector-facing interfaces are the neck/FPN outputs consumed by the detector, distinct from backbone readouts that serve as neck/FPN inputs.This interface defines the feature contract that DiD preserves.
- The weighted reconstruction objective assigns positive weights to each interface output; the five-interface RetinaNet default is (3.0, 2.5, 2.0, 1.5, 1.0).Plain MSE preserves the absolute feature scale and channel responses expected by the frozen detector.
- DiD uses no ground-truth boxes, class labels, detector-output targets, or pseudo-label assignment during adaptation.After conversion, the teacher is discarded and the original neck/FPN and prediction modules are reused without additional fine-tuning.
5 Experiments
Experiments show that DiD preserves detector-facing behavior across two detector families while recovering strong accuracy and delivering substantial efficiency gains. Ablations indicate that early-heavy interface supervision, simple MSE reconstruction, light augmentation, and in-domain conversion data are effective under the tested setting.
- Experimental Setup: DOTA-v1.5 conversion uses unlabeled training images, frozen teacher and detector components, and optimization of only the linear-attention student backbone.RetinaNet and Faster R-CNN use ViT-S backbones; DiD supervises multi-scale tensors exposed by the frozen neck/FPN.
- Main Results: Direct copy produces near-zero mAP, while generic and pseudo-label distillation recover only part of the lost detector performance.Hidden-state distillation is stronger than these baselines but remains below DiD.
- Main Results: 31.1 mAP is achieved by DiD with both ReLU and ELU on RetinaNet, compared with 28.1 and 26.7 for hidden-state distillation and 30.9 and 30.8 for supervised references.The small differences around supervised references are interpreted as comparable operating points, not evidence that label-free conversion is superior.
- Main Results: 29.3 mAP with ReLU and 28.9 with ELU are achieved by DiD on Faster R-CNN, with performance comparable to corresponding supervised linear-attention references.The same overall ranking across RetinaNet and Faster R-CNN indicates that DiD is not tied to one downstream detection architecture.
- Main Results: DiD lowers detector-facing mismatch rather than internal transformer mismatch, and reaches the supervised linear-training regime after approximately 20k-30k steps.Hidden-state distillation better matches internal states, whereas DiD better matches detector-facing tensors and achieves higher detector accuracy.
- Inference and Conversion Efficiency: 62% lower average latency and 49% lower peak memory are obtained by the linearized backbone in backbone-only inference.The comparison is between Softmax attention and the ReLU-linearized ViT.
- Inference and Conversion Efficiency: 87 minutes on four GPUs is the conservative 40k-step conversion time, while the 20k-step practical regime takes approximately 44 minutes.At fixed global batch size 8, throughput rises from 2.98 to 7.67 iterations per second when increasing GPUs from one to four.
- Ablation Studies: 30.7 mAP with early-heavy weighting outperforms 29.8 for uniform and 28.4 for late-heavy weighting across the five detector-facing tensors.The ordering is adopted as the default for the tested setting without claiming universal optimality.
6 Conclusion
The conclusion identifies detector-facing interface compatibility as the governing principle for label-free Softmax-to-linear detector conversion. DiD performs competitively with supervised linear training while reducing conversion time and inference resource use, though broader validation remains future work.
- Conclusion: DiD converts trained Softmax-attention ViT detectors into efficient linear-attention detectors without labels.The method trains the linear-attention backbone while using detector-facing tensors consumed by the frozen detector.
- Conclusion: Detector-facing compatibility lets the linear backbone compensate internally, whereas direct operator replacement and generic distillation fail.The conclusion presents interface supervision as the central explanation for successful conversion.
- Conclusion: DiD is the strongest label-free conversion method for RetinaNet and Faster R-CNN on DOTA-v1.5 and reaches an operating point comparable to fully supervised linear-attention training.The conclusion frames these as the principal experimental findings.
- Conclusion: The paper reports about 87 minutes on four GPUs together with substantial reductions in inference latency and peak memory.The conclusion connects the adaptation result with the systems benefits of the linearized backbone.
- Conclusion: The reported scope is ViT-S, with extension to larger backbones, additional detector families, and broader architecture-conversion settings left open.The conclusion presents these extensions as future directions rather than established results.
A Baseline Objectives
The baselines test whether global representations, detector outputs, or hidden states can support label-free Softmax-to-linear conversion, while noting that generic objectives do not explicitly preserve detector-facing tensors.
- Generic distillation: Generic distillation matches global representations across two stochastic augmentations of the same unlabeled image.The frozen teacher and student use global projection heads for this objective.
- Generic distillation: Generic distillation preserves high-level semantic consistency but does not explicitly constrain detector-facing interface tensors.It therefore evaluates representation transfer without directly matching the tensors consumed by the detector.
- Pseudo-label distillation: Softmax-generated pseudo-label distillation trains the student with the detection loss computed from teacher-produced pseudo detections on unlabeled images.This transfers supervision at the detector output level rather than through intermediate feature matching.
- Hidden-state distillation: Hidden-state distillation matches backbone hidden states across selected transformer layers using layer-specific weights.The objective follows a generic, detector-agnostic strategy for preserving intermediate transformer behavior.
- Comparison: The baselines collectively test whether semantic consistency, pseudo-label supervision, or hidden-state imitation suffice for object-detection conversion.The reported table organizes detector, kernel, and mAP metrics for the comparison.
B Variance Across Seeds
The experiments repeat each conversion three times with different random seeds and report mean and standard deviation for the detector and kernel combinations.
- Seed protocol: Each experiment is repeated three times with different random seeds, with Table A reporting mean and standard deviation.The evaluations cover RetinaNet and Faster R-CNN with ReLU and ELU kernels on DOTA-v1.5.
C Ablation on Teacher Initialization
DiD remains stronger than hidden-state distillation across ImageNet, DINO, and no-pretraining teacher initializations, while converted accuracy generally follows teacher quality.
- Teacher initialization: DiD remains stronger than hidden-state distillation for ImageNet, DINO, and no-pretraining teachers.The comparison covers all three tested teacher initialization settings.
- Teacher initialization: Stronger Softmax teachers generally yield stronger converted models.The final converted accuracy depends on the interface learned by the original detector.
- No pretraining: 0.3 mAP is the DiD improvement over the Softmax reference in the no-pretraining setting.The paper treats this small difference as setting-specific rather than evidence of generally improved teacher conversion.
D Generalization to Pascal VOC
A Pascal VOC study applies the DOTA-v1.5 conversion protocol to test transfer beyond aerial imagery, comparing label-free adaptation methods against a supervised reference.
- Transfer evaluation: The Pascal VOC experiment applies the same conversion protocol and baselines used in the main DOTA-v1.5 study.Its purpose is to test whether conversion behavior transfers beyond aerial imagery.
- Evaluation setup: Table C compares label-free adaptation methods with full supervised training for ViT-S conversion on Pascal VOC.Full training is the supervised reference, while the other adaptation methods use no detection labels.