PaperlayerView exact revision on arXiv

Source-linked AI summary

LayoutMamba: Efficient Document Layout Analysis with Hybrid Mamba-Transformer Networks

Kehua Chen, Mingyi Chen

arXiv:2610.34879v1cs.CV

TL;DR

Document layout analysis must handle heterogeneous documents and intricate spatial structure. LayoutMamba combines Transformers for local feature learning with Mamba for long-range modeling, and adds a geometric overlap penalty. It reports state-of-the-art results on PubLayNet and D4LA with 8× fewer parameters.

  • Problem

    Document layout analysis must segment documents with complex spatial structure across heterogeneous document types and variable scan quality.

    Source: §1 · p. 1

  • Method

    LayoutMamba combines Transformer feature learning with Mamba’s long-range spatial modeling and uses Geometric Overlap Penalty to constrain bounding-box overlaps.

    Source: §1 · pp. 2–3, §5 · p. 12

  • Results

    8× fewer parameters accompany state-of-the-art results on PubLayNet and D4LA.

    Source: §1 · p. 3, §5 · p. 12

  • Takeaways & Limitations

    The approach demonstrates Mamba’s potential for efficient multimodal document understanding.

    Source: §5 · p. 12

Abstract

from arXiv · show

Document layout analysis (DLA) faces a long-standing dilemma: existing methods either capture rich local details at prohibitive computational cost (e.g., Transformer-based approaches with quadratic complexity) or achieve efficient global modeling but sacrifice fine-grained structural awareness (e.g., state-space models). This trade-off hinders deployment in resource-constrained scenarios. We present LayoutMamba, a hybrid framework that synergistically combines sparse-attention Transformers for localized feature extraction (O(n) complexity) and the recurrent state-space mechanism of Mamba for dilution-free long-range dependency modeling across the 2D document canvas. Our two-stage paradigm first uses a lightweight Transformer encoder to hierarchically distill salient local semantics; second, Mamba propagates global contextual dependencies without attention dilution, enabling precise, efficient, and scalable layout understanding. We further identify a critical yet overlooked issue in existing DLA benchmarks: the inherent low-overlap distribution bias of layout objects, which we mitigate through a novel Geometric Overlap Penalty Loss (GOPLoss), a dual-optimization objective that minimizes redundant bounding box overlaps and encourages geometrically consistent predictions. Extensive experiments on PubLayNet and D4LA show that LayoutMamba surpasses state-of-the-art AP on PubLayNet with 8x fewer parameters and maintains leading performance on D4LA, ranking top-3 among published methods, achieving an unprecedented balance between accuracy, efficiency, and scalability. Our work pioneers a cognition-inspired paradigm for document analysis, advancing practical deployment in edge computing and resource-limited environments.

1 Introduction

LayoutMamba combines Transformer feature refinement with Mamba’s efficient long-range modeling for document layout analysis. The introduction reports strong benchmark results and presents Geometric Overlap Penalty loss to reduce bounding-box overlaps.

1 Introduction

  • DLA segments document images into semantic regions, but heterogeneous document types, intricate layouts, and variable scan quality complicate the task.CNN methods favor local features, while Transformer self-attention has quadratic computational complexity that constrains high-resolution or text-dense documents.Source: §1 · pp. 1–2
  • LayoutMamba assigns Mamba long-range spatial modeling and Transformers fine-grained local feature refinement and cross-modal fusion.The proposed division of labor aims to capture distant layout relationships without self-attention’s quadratic complexity.Source: §1 · p. 2
  • D4LA evaluates generalizability on a more demanding, diverse benchmark than PubLayNet, which primarily contains scientific articles with relatively standardized layouts.The introduction describes D4LA as designed to address the homogeneity and simplicity limitations of earlier datasets.Source: §1 · pp. 2–3
  • LayoutMamba, a hybrid Transformer-Mamba architecture tailored for document analysis, matches the accuracy of prior state-of-the-art Transformer-based methods with a 30% reduction in model parameters.Source: §1 · p. 3
  • Geometric Overlap Penalty combines IoU regularization and spatial repulsion to constrain bounding-box overlap, with ablations reporting more precise localization and better detection accuracy.The loss is designed for document object detection scenarios.Source: §1 · p. 3
  • 8× parameter compression versus competitive Transformer-based baselines accompanies new state-of-the-art results on PubLayNet and D4LA.The introduction describes particularly pronounced gains on the more complex and heterogeneous D4LA benchmark.Source: §1 · p. 3

2 Related Work

Related work traces DLA from object detection and CNN–Transformer approaches toward state-space models. It identifies efficient fusion of global and local features as an open challenge for document analysis.

2 Related Work

  • DLA has used general-purpose detectors, CNN features, and Transformer self-attention; YOLO models are noted for balancing accuracy and inference speed.Examples include YOLOv7’s E-Elan structure and YOLOv10’s dual label-assignment strategy.Source: §2 · p. 3
  • Mamba-YOLO applies state-space models to detection, but efficiently fusing global and local features to improve performance remains an open question.Its LSBlock and RGBlock are described as capturing local dependencies.Source: §2 · p. 3
  • Vision Mamba uses cross-scan modules for 2D selective image scanning, while MambaVision combines Mamba and Transformer modules in its final two stages.Building on these approaches, LayoutMamba integrates the modules for document analysis and emphasizes efficient inference and global receptive fields.Source: §2 · p. 4

3 Methodology

The methodology combines state-space modeling and Mamba with a multi-stage VRSSBlock for spatial and channel-wise feature refinement, then adds multi-head attention and an overlap penalty tailored to document layouts. Its components address long-range context, feature aggregation, and the low-overlap characteristics of layout predictions.

3.1 Preliminary: SSMs and Mamba

  • Classical SSMs efficiently maintain a latent state but their linear nature limits modeling complex long-range dependencies; Mamba’s S6 adds input-dependent selection, discretization, and adaptive temporal dynamics.S6 uses a learnable time-step and zero-order hold to enable input-conditioned scanning and nonlinear state evolution.Source: §3.1 · p. 4

3.2 The Overview of the LayoutMamba

  • VRSSBlock transforms input features through dimensionality alignment, global spatial aggregation via SS2D, and channel-wise refinement via a Residual Gated Block.Its projection maps channels to a hidden dimension, while the SS2D path combines local spatial processing with long-range modeling; residual connections preserve the original representation.Source: §3.2 · p. 5
  • SS2D uses structured state space models to process 2D inputs efficiently for global context modeling.The VRSSBlock also integrates residual learning and gated MLP variants to capture global spatial context and local channel-wise nuances.Source: §3.2 · pp. 4–5
  • The Residual Gated Block splits normalized input into local and global branches, then gates their fusion to selectively amplify salient features and suppress less informative ones.The local branch uses a depthwise convolution-based Position Encoding Module; an outer residual connection adds the refined output to the input.Source: §3.2 · p. 6

3.3 Multi-Head Attention

  • Multi-head attention aggregates information across the sequence dimension, with query, key, and value projections arranged across heads and then concatenated and linearly projected.Each head has dimension dh = C/H, where H is the number of heads.Source: §3.3 · p. 7

3.4 Geometric Overlap Penalty Loss

  • Document layout datasets contain many densely arranged boxes but rarely overlapping ones, motivating GOPLoss to penalize overlap between predicted boxes.The passage contrasts this property with traditional object detection tasks, where dense prediction boxes are commonly observed.Source: §3.4 · p. 8
  • GOPLoss averages pairwise box IoU using n^2 as the denominator, where n is the number of predicted boxes, to focus on relative overlap rather than absolute counts.The authors state this normalization aims to eliminate scale variations caused by different numbers of predictions.Source: §3.4 · p. 8
  • Because GOPLoss is unsupervised and independent of ground-truth labels, it is added as an auxiliary term to the weighted box, classification, and Distribution Focal losses.Its computation is O(n^2), but the authors report negligible practical overhead because few predictions typically survive NMS.Source: §3.4 · p. 8

4 Experiments

Experiments evaluate LayoutMamba on PubLayNet and the more varied D4LA benchmark, then compare its performance and test architectural and GOPLoss ablations. The reported comparisons show strong results on both datasets, while ablations identify gains from VRSSBlock, attention, and a moderate GOPLoss weight.

4.1 Dataset

  • PubLayNet contains 335,703 training, 11,245 validation, and 11,405 test images across five layout categories; D4LA has 8,868 training and 2,224 validation images across 12 document types and 27 fine-grained categories.D4LA includes skewness, blur, and diverse layouts, making it more challenging than PubLayNet for robustness evaluation.Source: §4.1 · p. 8

4.3 Comparison to Existing Approaches

  • LayoutMamba-T reaches 95.1 mAP on PubLayNet, comparable to DoPTA-HR with 85M parameters while using 6.9M; LayoutMamba-L reaches 96.3 mAP.On D4LA, LayoutMamba-T exceeds LayoutLMv3-Base by 5.8% mAP despite nearly 20× more parameters.Source: Table 1 · p. 9, §4.3 · p. 9, Table 2 · p. 10

4.2 Implementation Details

  • The comparison experiments use PAFPN for multi-scale feature fusion; PubLayNet training runs 300 epochs on four A100 GPUs, while D4LA uses one A100 GPU.Both datasets use SGD with linear warmup for three epochs; batch sizes are 128 for PubLayNet and 16 for D4LA.Source: §4.2 · p. 9

4.4 Ablation Studies

  • At λ4 = 5.0, GOPLoss reaches 58.4 mAP on D4LA, 0.5 percentage points above the λ4 = 0.0 baseline at 57.9; at λ4 = 7.5, mAP falls to 57.6.Visual comparisons report diminished target-box overlap in dense text regions and fewer generated boxes with GOPLoss.Source: Table 3 · p. 10, §4.4 · pp. 11–12, Figure 2 · p. 11
  • 60.7% mAP at IoU [0.50,0.95] is the VSSBlock baseline with 5.8M parameters on D4LA.Replacing VSSBlock with VRSSBlock raises mAP from 60.7% to 61.2% as parameters increase to 6.2M.Source: §4.4 · p. 10, Table 4 · p. 11
  • Adding multi-head attention to VRSSBlock raises D4LA mAP from 61.2% to 63.3%, with parameters increasing from 6.2M to 6.9M.The attention module derives query, key, and value projections from VRSSBlock-processed features for adaptive recalibration across spatial and channel dimensions.Source: §4.4 · pp. 10–11

5 Conclusion

LayoutMamba combines Vision Transformers and Mamba for document layout analysis, using sparse attention for local features and Mamba for global context. The conclusion reports state-of-the-art results on PubLayNet and D4LA with 8× fewer parameters, and introduces a loss addressing low-overlap bias.

5 Conclusion

  • State-of-the-art results on PubLayNet and D4LA are reported with 8× fewer parameters.The conclusion also describes the approach as demonstrating Mamba’s potential for efficient multimodal document understanding.Source: §5 · p. 12
  • Sparse attention extracts local features while Mamba’s linear complexity supports global context modeling.The architecture combines Vision Transformers and state-space models for document layout analysis.Source: §5 · p. 12
  • Geometric Overlap Penalty Loss addresses low-overlap bias in layout predictions.Source: §5 · p. 12
Loading 2610.34879v1…