PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Spectral Feature Fusion for Multispectral Document Binarization

Yasas Savinda, Maheshi Buddhinee Dissanayake

arXiv:2610.30863v1cs.CVeess.SP

TL;DR

Historical-document binarization remains difficult when wavelength-dependent degradation obscures text, and adaptive integration of multispectral features is underexplored. This study uses targeted VIS–NIR structural guidance to regulate intermediate fusion; F3+NIR achieves the strongest overall evaluated performance, with significant gains over fixed averaging in three metrics.

  • Problem

    How to adaptively integrate visible and NIR features within deep restoration networks remains less explored, despite multispectral evidence that can help address degraded documents.

    Source: §1 · pp. 1–2

  • Method

    A multi-branch framework uses learned modality and structural gating with targeted visible–NIR edge guidance while retaining the full multispectral observation.

    Source: §1 · pp. 2–3

  • Results

    F3+NIR structural guidance achieved the strongest overall performance among evaluated configurations, with Holm-significant gains over fixed averaging in F1, pseudo-F1, and IoU.

    Source: §1 · p. 3, §7 · p. 16

  • Takeaways & Limitations

    Targeted cross-spectral structural guidance regulates intermediate multispectral fusion while preserving the complete spectral observation.

    Source: §7 · pp. 16–17

    The four-band SMADI dataset limits media, ink, and degradation diversity, so F3+NIR is not established as universally optimal.

    Source: §6 · p. 16

Abstract

from arXiv · show

Historical document binarization remains challenging when faded ink, bleed-through, and background degradation exhibit wavelength-dependent behavior across visible and near-infrared observations. This study investigates whether adaptive cross-spectral feature fusion and targeted structural guidance can improve multispectral document binarization beyond fixed fusion and generic attention mechanisms. Four registered SMADI bands at 500, 600, 700, and 1000 nm are processed using a multi-branch conditional generative adversarial network (cGAN). Visible, NIR, and structural feature streams are fused at encoder depth L = 3 using spatial- and channel-wise softmax gating, while auxiliary edge maps derived from selected VIS-NIR band pairs provide targeted structural guidance. Nine controlled fusion and structural configurations are evaluated using five-fold document-level cross-validation, DIBCO-standard metrics, and Holm-corrected statistical testing. The F3 + NIR edge-guided configuration provides the strongest overall performance among the evaluated models, achieving mean F1 = 0.7916, p-FM = 0.8044, and IoU = 0.6654. Relative to the fixed Mid AVG baseline, it improves F1 by 0.0269, p-FM by 0.0360, and IoU by 0.0321, with all three gains remaining statistically significant after Holm correction. Mean NRM and DRD are also reduced, although these reductions are not statistically significant after correction. Targeted cross-spectral structural guidance provides an effective mechanism for regulating intermediate multispectral fusion, improving primary foreground-overlap performance while retaining the complete four-band observation.

1 Introduction

Historical document binarization must recover weak strokes amid spatially uneven degradation, while multispectral methods still leave adaptive integration of visible and NIR features underexplored. This work tests targeted VIS–NIR structural guidance for adaptive intermediate fusion and reports stronger overall performance for F3+NIR guidance than the fixed baseline.

1 Introduction

  • Historical-document degradation can obscure strokes, so binarization must suppress background while recovering weak, thin, or discontinuous text without false foreground structures.Faded ink, bleed-through, stains, uneven illumination, aging, surface damage, and low contrast complicate digital analysis.Source: §1 · p. 1
  • Existing multispectral methods show the value of spectral information, but how to adaptively integrate visible and NIR features within deep restoration networks remains less explored.Prior approaches include band or subspace selection, analytical transformations, source decomposition, target detection, factorization, and direct learned mappings.Source: §1 · p. 2
  • The literature separates multimodal fusion from document-specific structural priors, leaving VIS–NIR integration under structural guidance systematically unexamined.The study asks whether a targeted VIS–NIR cue can regulate intermediate features at encoder depth L = 3.Source: §1 · p. 2
  • This journal extension replaces fixed averaging with learned modality and structural gating, adds targeted visible–NIR edge guidance and edge-aware supervision, and expands page-level validation and profiling.It evaluates nine configurations using five-fold document-level cross-validation and Holm-corrected statistical tests.Source: §1 · pp. 2–3
  • F3+NIR structural guidance achieved the strongest overall performance among evaluated configurations, with Holm-significant gains over fixed averaging in F1, pseudo-F1, and IoU.The proposed framework uses a targeted cross-spectral cue to regulate intermediate fusion.Source: §1 · p. 3

2 Related Work

Prior work develops binarization, multispectral analysis, adaptive fusion, and structural guidance largely as separate lines. This paper connects them by using a targeted cross-spectral structural representation to govern intermediate fusion while retaining the full-spectrum input.

2.1 Historical Document Binarization and Enhancement

  • Classical thresholding struggles with combined fading, bleed-through, stains, texture variation, and localized degradation; deep and adversarial methods address restoration but do not explicitly exploit wavelength-dependent multispectral information.Structural mechanisms such as GDB and RA-GAN add edge or gradient guidance, but derive it from conventional document inputs.Source: §2.1 · p. 3

2.2 Multispectral Historical Document Analysis

  • Multispectral document methods progress from explicit spectral modeling to CNN and factorized representations, while selected spectral cues and contours remain relevant to foreground-background separation.The present approach uses a selected visible–NIR pair as auxiliary guidance while preserving the complete multispectral input.Source: §2.2 · p. 4

2.3 Adaptive Multimodal and Visible–Infrared Fusion

  • Intermediate fusion preserves modality-specific feature extraction before cross-modal interaction, while GMUs, SE, and CBAM provide adaptive weighting or attention baselines.Generic attention modules recalibrate domain-agnostic feature maps rather than document-specific stroke topographies.Source: §2.3 · p. 4
  • Generic VIS–IR fusion often concerns thermal emissions, whereas historical-document MSI uses reflected NIR whose contrasts arise from wavelength-dependent absorption and scattering.Learned fusion principles remain relevant, but the physical basis of spectral complementarity differs.Source: §2.3 · pp. 4–5

2.4 Structural Guidance for Fusion and Binarization

  • Prior work supports structural constraints for fusion and stroke integrity for binarization, but existing document priors use conventional observations and multispectral cues are not dynamic controls in deep networks.This work combines targeted cross-spectral structural guidance with intermediate fusion while retaining the full-spectrum observation.Source: §2.4 · p. 5

3 Proposed Method

The method maps four registered spectral bands and an auxiliary structural prior to a continuous document representation using adaptive intermediate feature fusion. It constructs the prior from a selected visible–NIR pair while retaining all four bands as network input.

3.1 Problem Formulation

  • The input comprises registered 500, 600, 700, and 1000 nm observations, and the generator estimates a continuous representation separating foreground text from degraded background.The selected visible–NIR pair supplies an auxiliary prior without replacing any original spectral band.Source: §3.1 · p. 5
  • At encoder depth L = 3, visible, NIR, and structural features are combined by a learnable spatial- and channel-wise adaptive gating operator.The generator performs adaptive rather than fixed arithmetic fusion.Source: §3.1 · pp. 5–6
  • A global threshold converts reconstructed page predictions to binary maps, with the threshold optimized on validation pages and fixed for test evaluation.The binary convention assigns foreground text below the threshold and background at or above it.Source: §3.1 · p. 6

3.2 Multispectral Preprocessing and Structural Edge Construction

  • The structural map is built from normalized 600 nm visible and 1000 nm NIR bands, averaged equally; all four spectral bands remain network inputs.The map is generated at full-page resolution before patch extraction to avoid artificial boundary gradients.Source: §3.1 · p. 5, §3.2 · p. 6
  • CLAHE enhances IVN (clip limit 2.0; 8 × 8 grid); 3 × 3 Sobel gradients yield magnitude G, min–max rescaled to 8-bit, while aligned inputs use 256 × 256 patches at 204-pixel stride (∼20% overlap).Source: §3.2 · p. 6

3.3 Fixed Intermediate Fusion Baseline

The fixed intermediate-fusion model is the principal baseline for the adaptive mechanism. It averages independently encoded visible and NIR streams at depth L = 3 using static equal weights.

3.3 Fixed Intermediate Fusion Baseline

  • Normalized additive fusion at encoder depth L = 3 was identified as the strongest prior-work baseline and is retrained under five-fold document-level cross-validation.The earlier comparison found it stronger than early- and late-fusion alternatives.Source: §3.3 · p. 7
  • Visible and NIR streams are encoded independently through the first three stages, then element-wise averaged because their feature maps share spatial and channel dimensions.The visible stream contains F2, F3, and F4; the NIR stream contains F7.Source: §3.3 · p. 7
  • The baseline fixes visible and NIR fusion weights at 0.5 regardless of local degradation or spectral contrast, motivating dynamic content-adaptive fusion.The averaged representation passes through the remaining shared encoder stages and U-Net decoder.Source: §3.3 · p. 7

3.4 Adaptive Visible–NIR Edge-Guided Fusion

The generator retains the multispectral input and adds a visible–NIR structural stream, then adaptively fuses the three streams at encoder depth L = 3. Spatial- and channel-wise softmax gating weights these features before the decoder produces the output.

3.4 Adaptive Visible–NIR Edge-Guided Fusion

  • The generator uses visible, NIR, and structural feature branches to adaptively fuse information at encoder depth L = 3.The auxiliary visible–NIR structural map supplements rather than replaces the multispectral input.Source: §3.4 · p. 7
  • A 1 × 1 convolution maps concatenated branch features to modality logits, which receive spatial- and channel-wise softmax normalization.The weights sum to one at each channel and spatial coordinate.Source: §3.4 · p. 7
  • The gated visible, NIR, and edge features are combined element-wise to form the fused representation.After fusion, shared encoder stages and a U-Net decoder with skip connections restore spatial resolution and produce a single-channel output.Source: §3.4 · p. 8
  • Unlike fixed averaging, gating can favor visible features, shift toward NIR under severe visible-spectrum staining, or use edge features near degraded boundaries.The stated aim is to preserve stroke topology near degraded boundaries.Source: §3.4 · p. 8

3.5 Conditional Adversarial Learning Objective

The generator is trained conditionally against a PatchGAN discriminator, with adversarial and pixel-level reconstruction objectives. An additional edge-aware loss supervises structural and boundary fidelity.

3.5 Conditional Adversarial Learning Objective

  • The generator maps multispectral and structural inputs to a restored single-channel representation, while a conditioned PatchGAN discriminator distinguishes generated outputs from paired ground truth.Optimization alternates discriminator updates with end-to-end generator updates.Source: §3.5 · p. 8
  • The ℓ1 reconstruction loss supports pixel-level alignment and reduces blurring along thin text boundaries compared with squared error.It is combined with adversarial and edge-aware structural constraints in the generator objective.Source: §3.5 · p. 8

3.6 Edge-Aware Structural Loss

  • The edge-consistency loss compares Sobel gradient magnitudes of prediction and ground truth to penalize structural and boundary errors.It complements ℓ1 intensity fidelity by targeting character erosion, stroke fragmentation, and boundary degradation.Source: §3.6 · pp. 8–9

3.7 Full-Page Reconstruction and Inference

Full-page inference reconstructs pages from overlapping patch predictions and applies a global threshold. Checkpoints and thresholds are selected using validation pages, with test documents held out until evaluation.

3.7 Full-Page Reconstruction and Inference

  • Padded pages are divided into aligned 256 × 256 patches with 204-pixel stride, and their soft predictions are accumulated into page-level intensity and coverage matrices.Overlap averaging suppresses boundary seams and edge discontinuities before global thresholding.Source: §3.7 · p. 9
  • The threshold is tuned on reconstructed validation pages and applied unchanged to the test fold.The final binary map assigns foreground where the soft prediction is below the threshold and background otherwise.Source: §3.7 · p. 9

3.8 Validation-Based Model and Threshold Selection

  • Model and threshold selection use only each fold’s validation partition; held-out test documents remain unseen during optimization and are evaluated once per fold.The five-fold procedure repeats this separation independently across folds.Source: §3.8 · p. 9
  • Checkpoint selection maximizes validation text F1, using SSIM to resolve ties.The selected checkpoint is frozen before validation pages are used to optimize the binarization threshold.Source: §3.8 · p. 9
  • Threshold selection prioritizes validation pseudo-F1, with standard text F1 as a secondary tiebreaker.The internal pseudo-F1 screening metric is distinct from final p-FM evaluation.Source: §3.8 · p. 9

4 Experimental Protocol

The experimental protocol evaluates nine fusion and structural configurations on registered multispectral SMADI pages using document-level cross-validation. It combines full-page evaluation, paired statistical testing, and non-learning baselines.

4.1 Dataset and Spectral Configurations

  • Experiments use 30 paired SMADI pages with registered visible bands at 500, 600, and 700 nm and a 1000 nm NIR band.The auxiliary structural map uses the F3 and F7 pair while the full four-band input remains available to the network.Source: §4.1 · p. 10

4.2 Five-Fold Document-Level Cross-Validation

  • Five-fold document-level cross-validation assigns six documents each to test and validation, with the remaining 18 used for training in each run.The partitions are disjoint, and normalization and patch extraction occur only after assignment.Source: §4.2 · p. 10

4.3 Compared Fusion Architectures

  • Nine configurations isolate fixed versus learned fusion, structural conditioning and supervision, generic attention, and F2, F3, or F4 pairings with NIR.All use fusion depth L = 3 and the same training schedules and validation pipelines.Source: §4.3 · p. 10

4.4 Training Protocol

  • Networks are trained independently per fold for 100 epochs using Adam, with synchronized flips applied to inputs, structural maps, and targets during training.The specified optimizer settings include learning rate 2 × 10−4 and batch size 8.Source: §4.4 · p. 10

4.5 Evaluation Metrics

  • Evaluation uses full-page binary outputs and five complementary metrics, including text F1, IoU, p-FM, NRM, and DRD.p-FM is character-structure-aware and uses a DIBCO-compatible reference pipeline; lower is better for NRM and DRD.Source: §4.5 · pp. 10–11

4.6 Statistical Analysis

  • Candidate architectures are compared with Mid AVG using paired two-sided Wilcoxon signed-rank tests and Holm correction across eight comparisons per metric family.Effect sizes use matched rank-biserial correlation, with bootstrap confidence intervals and document win/tie/loss counts for the primary F3+NIR architecture.Source: §4.6 · p. 11

4.7 Classical Binarization Baselines

  • Otsu and Sauvola are evaluated as non-learning baselines on the rescaled mean of the three visible bands.Sauvola parameters are selected per fold using validation pseudo-F1.Source: §4.7 · p. 11

5 Results and Discussion

Across nine architectures, F3+NIR edge-guided fusion led on primary overlap metrics and significantly outperformed Mid AVG, while wavelength choice, edge supervision, overlap, and metric-specific trade-offs shaped the results.

5.1 Overall Comparison of Fusion Architectures

  • F3+NIR reaches F1=0.7916, p-FM=0.8044, and IoU=0.6654; versus Mid AVG, gains are 0.0269, 0.0360, and 0.0321, all Holm-significant.Mean NRM and DRD also decrease versus Mid AVG, but neither reduction remains significant after Holm adjustment.Source: §5.1 · p. 11, Table 2 · p. 12, §5.3 · p. 14

5.2 Effect of Targeted Visible–NIR Structural Guidance

  • F3+F7 (600/1000 nm) yields the highest F1, p-FM, and IoU among targeted pairs, while F2+F7 has the lowest NRM and the four-band map the lowest DRD.All configurations retain the complete four-band input; only the auxiliary structural source and edge-consistency loss vary.Source: §5.2 · pp. 12–13, Table 3 · p. 13

5.3 Statistical Significance

  • Across 30 held-out pages, paired Wilcoxon tests with Holm correction confirm F3+NIR gains on F1, p-FM, and IoU, but not its NRM or DRD reductions.The primary overlap gains have pHolm < 0.03, rank-biserial effects rrb ≥0.596, and win ratios ≥23/30.Source: §5.3 · pp. 13–14

5.4 Comparison with Classical Binarization

  • F3+NIR improves over Sauvola by +0.0480 F1 and +0.0566 IoU, while Sauvola’s F1=0.7436 exceeds Otsu’s F1=0.5400.Otsu’s NRM is 0.1175, but its weaker F1 and IoU illustrate why evaluation uses multiple metrics.Source: §5.4 · p. 14, Table 6 · p. 15

5.5 Reconstruction-Overlap Sensitivity

  • With the frozen F3+NIR model, 30% overlap gives the best reported metrics, including F1=0.7957 and IoU=0.6717, while 0% overlap produces boundary artifacts.The 20% setting remains in architecture comparisons to balance efficiency against incremental gains.Source: §5.5 · p. 14, Table 7 · p. 15

5.6 Computational Complexity and Inference Efficiency

  • The generator has 56.31 million parameters and requires 20.68 GMACs per 256 × 256 patch; T4 single-patch latency is 8.73 ± 0.57 ms, and model size motivates compression.The complexity estimates are approximate indicators, not hardware-independent runtime predictions.Source: §5.6 · pp. 15–16

5.7 Qualitative Analysis

  • Qualitatively, edge-aware supervision improves stroke continuity and suppresses isolated false positives, while targeted F3+NIR guidance better reconstructs strokes without boundary erosion or false-positive expansion.Residual errors persist where local background contrast approaches zero.Source: §5.7 · p. 16

6 Limitations and Future Work

The study’s scope and computational demands constrain deployment and generalization, while the authors identify broader evaluation and more efficient architectures as priorities for future work.

Spectral Pair Selection and Dataset Scope

  • Evaluation on 30 SMADI documents limits media, ink, and degradation diversity, so F3 + NIR is not universally optimal.The authors call for broader corpora and data-driven wavelength selection.Source: §6 · p. 16

Loss Formulation and Structural Guidance Filters

  • The edge-loss weight and Sobel extraction were tested under controlled rather than exhaustive settings, and Sobel can preserve non-text artifacts.Future work could explore text-aware descriptors, multiscale edges, or confidence-weighted losses.Source: §6 · p. 16

Computational Complexity and Full-Page Inference

  • The 56.31 × 10^6-parameter generator requires approximately 20.68 GMACs per 256×256 patch, and 20% reconstruction overlap increases page-level computation.The authors identify branch sharing, distillation, lightweight fusion, and fully convolutional page inference as deployment directions.Source: §6 · p. 16

Future Work

  • Future work will emphasize cross-dataset generalization, adaptive spatial-spectral guidance, and architectural compression.Source: §6 · p. 16

7 Conclusion

The study concludes that targeted multispectral intermediate fusion improves historical-document binarization over simple averaging and generic attention. F3 + NIR guidance leads on foreground recovery, while other wavelength pairings favor some error metrics.

7 Conclusion

  • Across 30 SMADI pages, F3 + NIR guidance achieved statistically significant F1, p-FM, and IoU gains over fixed averaging and generic CBAM/SE attention.The full F2, F3, F4, and F7 observation remains available while a dedicated F3 + NIR (600/1000 nm) pair supplies structural guidance.Source: §7 · p. 16
  • A 20% patch reconstruction overlap provides a practical deployment compromise for full-page GPU inference.Source: §7 · p. 16
  • Figure 1 compares baseline, gated, and channel-attention architectures on held-out documents using validation-selected thresholds.Source: Figure 1 · p. 17
  • Targeted F3 + NIR guidance favors foreground recovery and structural continuity, whereas alternative pairings favor NRM or DRD.Source: §7 · p. 17

Statements and Declarations

The authors report funding, contributions, competing-interest and ethics statements, and the availability of data and code. The supplied figure caption also identifies a highlighted wavelength-specific edge-guided model.

Funding

Competing interests

Author contributions

  • Yasas Savinda and Maheshi Buddhinee Dissanayake share authorship and approval, with contributions spanning methodology, analysis, and writing.Savinda is credited with software, validation, investigation, and visualization; Dissanayake with supervision and grant management.Source: Statements and Declarations · p. 17

Data availability

  • The experiments use the SMADI dataset associated with the ICDAR 2015 MultiSpectral Text Extraction benchmark, and the dataset and supplementary outputs are publicly available for non-commercial research.Source: Statements and Declarations · pp. 17–18

Figure 2

  • Figure 2 compares channel-attention and wavelength-specific edge-guided models (F2–F4 + NIR), with the proposed F3+NIR model highlighted in bold.Source: Figure 2 · p. 18

Code availability

Ethics and consent

Loading 2610.30863v1…