PaperlayerView exact revision on arXiv

Source-linked AI summary

Weave Mamba Fusion: Global Cross-Scale Interaction for Lightweight Face Detection

Dohun Kim, Jinmyung Jung

arXiv:2610.05865v1cs.CVcs.AI

TL;DR

Local pyramid fusion cannot capture distant cross-scale context needed for difficult face detection. WMF weaves adjacent scales for global scanning, and WeaveFace reaches 91.41% mean AP on WIDER FACE with 0.34M parameters and 1.16 GFLOPs.

  • Problem

    Local feature-pyramid fusion is spatially local, limiting access to distant cross-scale context relevant to small and occluded faces.

    Source: §1 · pp. 1–2

  • Method

    WMF weaves adjacent feature maps column by column for bidirectional SS2D scanning, with partial-channel processing, and is integrated into WeaveBiFPN.

    Source: §1 · pp. 2–3

  • Results

    91.41% mean AP and 87.14% Hard AP are achieved by WeaveFace on WIDER FACE with 0.34M parameters and 1.16 GFLOPs.

    Source: §1 · p. 3

  • Takeaways & Limitations

    WMF provides global cross-scale interaction at linear cost for face detection.

    Source: §1 · p. 2

    WeaveFace has higher inference latency than summation-based baselines (23.3 ms vs. 15.1 ms), partly because selective scan is memory-bandwidth-bound.

    Source: §5 · p. 14

Abstract

from arXiv · show

Feature pyramid methods, from FPN to BiFPN, have achieved strong performance in face detection by fusing multi-scale features. However, detecting faces under unconstrained conditions, such as small scale, occlusion, and extreme pose, remains difficult, as it requires global cross-scale dependencies that local fusion cannot model. State space models such as Mamba provide global context with linear complexity by scanning features as a sequence, and therefore offer a promising direction for this problem. Nevertheless, such a scan needs the two pyramid scales combined into a single feature map, and the way they are combined determines whether cross-scale structure is preserved. Summation collapses the two scales before the scan, so the scan has no cross-scale structure to exploit, while concatenation keeps both scales but at far higher cost. To address this, we propose \textbf{Weave Mamba Fusion (WMF)}, which interleaves two adjacent pyramid scales column by column so that each step of a horizontal bidirectional SS2D scan moves from one scale to the other. With partial-channel processing and parameter-free de-weaving, WMF enables efficient cross-scale interaction while preserving feature structure. Integrating WMF into every fusion node yields \textbf{WeaveBiFPN}, the neck of our \textbf{WeaveFace} detector. On WIDER FACE, WeaveFace achieves 91.41\% mean AP with only 0.34M parameters and 1.16 GFLOPs, outperforming prior detectors under 0.5M parameters. Its largest gains are on the Hard subset, where it reaches 87.14\% AP. The code is publicly available at \url{https://github.com/dohun-mat/WeaveMambaFusion}.

1 Introduction

Unconstrained face detection remains difficult because local pyramid fusion cannot connect small or occluded faces to distant semantic context. Weave Mamba Fusion (WMF) interleaves adjacent scales for global cross-scale scanning, and WeaveFace achieves strong WIDER FACE accuracy with a lightweight model.

1 Introduction

  • Local pyramid fusion cannot reach distant cross-scale context needed to detect small or occluded faces.Such faces rely on semantic context at positions away from the face in the low-resolution pyramid level.Source: §1 · pp. 1–2
  • WMF weaves two feature maps into one 2D map for bidirectional scanning, enabling global cross-scale interaction within a single operator.Adjacent columns belong to different scales, so each scan step propagates information between them.Source: §1 · pp. 2–3
  • A partial-channel design scans only a fraction of channels, reducing WMF’s dominant cost while preserving cross-scale fusion.Applying WMF at every node of both BiFPN pathways yields WeaveBiFPN, the neck of WeaveFace.Source: §1 · p. 3
  • 87.14% Hard AP and 91.41% mean AP are achieved by WeaveFace with 0.34M parameters and 1.16 GFLOPs on WIDER FACE.It is the most accurate detector with fewer than 0.5M parameters.Source: §1 · p. 3

2 Related Work

Related work spans face detectors, feature-pyramid fusion, and state-space models. The paper positions WMF as a fusion operator that uses an SSM scan to provide global interaction between pyramid scales.

2.1 Face Detection

  • Face detectors including RetinaFace, DSFD, and TinaFace [37] achieve accuracy at the cost of tens of millions of parameters and tens to hundreds of GFLOPs.Lightweight approaches include SCRFD [9], EResFD, FDLite [1], and FeatherFace.Source: §2.1 · p. 4

2.2 Feature Pyramid Networks

  • FPN [25], PANet, and BiFPN vary pyramid pathways, but their position-wise fusion and local convolutions limit interaction to nearby positions.FPT models nonlocal pyramid interaction with attention, whose quadratic cost is ill-suited to lightweight detectors.Source: §2.2 · p. 4

2.3 State Space Models for Multi-Scale Fusion

  • Prior detection SSMs generally scan backbone features or enhance features before or after conventional fusion, whereas WMF uses the SSM as the fusion operator.WMF weaves alternating scale positions so each SS2D scan step moves between scales.Source: §2.3 · pp. 4–5

3 WeaveFace

WeaveFace combines an EResNet backbone, CBAM refinement, a WeaveBiFPN neck built around WMF, and task-specific detection heads. WMF weaves adjacent-scale features for global cross-scale interaction, while WeaveBiFPN applies it throughout both pyramid pathways.

3.1 Weave Mamba Fusion (WMF)

  • WMF aligns adjacent-level features, interleaves their columns, scans half the channels horizontally in both directions, then de-weaves and remixes the outputs.The remaining channels bypass the scan through an identity path; de-weaving restores the scale-specific spatial structure without adding parameters.Source: §3.1 · pp. 6–7

3.2 Complexity

  • WMF’s partial-channel design reduces SS2D projection cost by N^2, while total cost remains O(L · C^2), linear in token count.The final 1×1 convolution dominates the total cost.Source: §3.2 · p. 7

3.3 WeaveBiFPN: Fully Bidirectional Pyramid with WMF

  • WeaveBiFPN applies WMF at every top-down and bottom-up node, removing same-level skips so each node fuses exactly two inputs.The pathways fuse each level with an upsampled feature from above or a downsampled feature from below.Source: §3.3 · pp. 7–8

3.4 Auxiliary Modules and Detection Heads

  • CBAM refinement brackets WeaveBiFPN, while deformable-convolution CCPM and ShuffleNet heads produce classification, box, and landmark predictions.The predictions are supervised with a multi-task loss.Source: §3.4 · p. 8

4 Experimental Results and Analysis

WeaveFace pairs strong WIDER FACE accuracy with a lightweight model, while qualitative results and ablations examine its robustness, fusion design, and efficiency trade-offs.

4.1 Experimental Setup

  • The evaluation uses WIDER FACE’s Easy, Medium, and Hard subsets; its landmark branch is trained with RetinaFace’s five-point annotations because WIDER FACE lacks landmark ground truth.Training images are resized to 640 × 640, and FLOPs are calculated at 640 × 480.Source: Dataset · p. 9, Implementation Details · p. 9

4.2 Quantitative Results

  • 91.41% mean AP with 0.34M parameters and 1.16 GFLOPs makes WeaveFace the highest-AP detector under 0.5M parameters on WIDER FACE.It also exceeds EResFD by 6.73% on Hard and nearly matches TransEnc-R50 in mean AP (91.41% vs. 91.49%).Source: Table 1 · p. 9, §4.2 · p. 10
  • On AFW and PASCAL Faces without fine-tuning, WeaveFace achieves 99.50% and 98.77% AP, respectively, despite having the fewest parameters among compared methods.It exceeds SCRFD-0.5GF on both datasets while using 40% fewer parameters.Source: §4.2 · p. 10, Table 2 · p. 10

4.3 Qualitative Results

  • In the Hard-subset qualitative comparison, WeaveFace recovers most missed faces and suppresses false positives in crowded, small-face, and heavily occluded scenes.Both detectors use a 0.5 confidence threshold, 0.4 NMS IoU threshold, and single-scale inference at a 1600-pixel short side.Source: §4.3 · pp. 10–11, Figure 4 · p. 11

4.4 Ablation Study

  • On BiFPN, WMF reaches 91.41 mean AP, exceeding summation by 1.91 and cross-attention by 0.92, with 9.7× less compute than cross-attention.The comparison reports 0.344M parameters and 1.159G FLOPs for WMF versus 0.577M and 11.228G for cross-attention; their latencies are 23.3 ms and 23.7 ms.Source: §4.4 · pp. 11–12, Table 3 · p. 12
  • Across pyramid topologies, WMF improves mean AP; FPN gains +0.34 and PANet +0.43 while using fewer parameters and FLOPs.The gain is +1.91 mean AP on BiFPN, with increased parameters and FLOPs; adding BiFPN parameters without weaving yields a smaller gain.Source: §4.4 · p. 12, Table 4 · p. 12
  • As channels are reduced, weaving loses less AP than concatenation or summation and outperforms concatenation at every reduced ratio, including 91.41 vs. 90.29 at N=2.At N=2, the chosen setting comes within 1.03 AP of full-channel concatenation with one-third its parameters and less than half its FLOPs.Source: §4.4 · p. 13, Table 5 · p. 13
  • With N=2, adding a backward horizontal scan improves mean AP, while adding vertical scans provides no additional gain; the selected s=2 setting reaches 91.41 mean AP.The scan-count comparison varies s from one forward horizontal direction to four directions.Source: §4.4 · p. 13, Table 6 · p. 14

5 Conclusion

The conclusion presents WMF as a global cross-scale fusion operator for lightweight face detection and reports WeaveFace’s WIDER FACE performance. It also identifies inference latency as the primary trade-off.

5 Conclusion

  • WMF weaves adjacent feature maps and applies a 2D Selective Scan to mix scales at each step, with partial-channel processing reducing scan cost to roughly 1/N^2.The authors place WMF in every node of a fully bidirectional pyramid to form WeaveBiFPN, the neck of WeaveFace.Source: §5 · p. 14
  • On WIDER FACE, WeaveFace reaches 91.41% mean AP with 0.34M parameters and 1.16 GFLOPs, the most accurate detector below 0.5M parameters.The conclusion notes that its largest gains are on the Hard subset.Source: §5 · p. 14
  • WeaveFace has higher inference latency than summation-based baselines: 23.3 ms versus 15.1 ms, attributed primarily to the memory-bandwidth-bound selective scan.The authors identify incomplete use of hardware parallelism and propose Mamba-2 and fusing weaving into the scan kernel as future optimizations.Source: §5 · p. 14

A Generality across Backbones

The backbone comparison varies architecture while holding the WeaveBiFPN neck and detection components fixed. EResNet remains the default because MobileNetV1-0.25 exceeds the targeted parameter budget, while the neck is reported to scale across backbones without backbone-specific tuning.

Backbone comparison

  • The comparison holds WeaveBiFPN (N=2, s=2), CBAM, and CCPM fixed while varying EResNet, MobileNetV1-0.25, ResNet-50, and ResNet-152.All use ImageNet-pretrained weights and identical training settings.Source: A Generality across Backbones · p. 17, Table 7 · p. 17
  • MobileNetV1-0.25 is slightly more accurate than EResNet but uses 1.6× more parameters and exceeds the target sub-0.5M budget, so EResNet is the default.The table caption specifies that only the backbone changes.Source: A Generality across Backbones · p. 17, Table 7 · p. 17
  • ResNet-152 more than doubles ResNet-50’s parameter count without improving performance; the authors report stable training and consistent scaling across the tested backbones without backbone-specific tuning.Source: A Generality across Backbones · p. 17

B Anchor Settings

The anchor configuration assigns two 1:1 anchors to each spatial location across pyramid levels P2–P4.

Anchor Settings

  • Two 1:1 anchors are assigned at every P2–P4 location, with six sizes doubling from 16 to 512 pixels and remaining fixed across input resolutions.Label assignment and hard-negative mining follow RetinaFace’s matching and loss formulation.Source: B Anchor Settings · p. 17

C Generality to Generic Object Detection

The study tests whether WMF transfers beyond face detection by adding it to EfficientDet’s BiFPN neck for generic object detection. WMF improves results on both COCO val2017 and PASCAL VOC with modest added computation.

C Generality to Generic Object Detection

Related papers

Loading 2610.05865v1…