PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Adaptive Feature Refinement Network for VHR Remote Sensing Image Segmentation

Shuaishuai Cao, Meng Tang, Shuwei Peng, Xuan Liu, Min Huang, Jie Chen, Jiacheng Niu, Yong Chen, Edore Akpokodje, Hui Lin

arXiv:2608.15647v1cs.CVcs.AI

TL;DR

VHR segmentation must combine detailed shallow features with semantic deep features while preserving useful pretrained representations. HAFR-Net progressively refines hierarchical features and achieves strong results across four VHR benchmarks, including 55.17% mIoU on LoveDA and 67.70% on OpenEarthMap.

  • Problem

    VHR segmentation must combine spatially detailed shallow features with semantically strong deep features without unnecessarily destroying pretrained information.

  • Method

    HAFR-Net progressively fuses heterogeneity-conditioned hierarchical features, adds bounded frequency residuals, and uses structural and class-relation cues for decoding.

  • Results

    55.17% mIoU on LoveDA and 67.70% on OpenEarthMap, with the best published means on Potsdam and OpenEarthMap among compared entries.

  • Takeaways & Limitations

    Across four VHR benchmarks, adaptive conservative refinement is associated with spatially varying stage routing, improved boundary and thin-structure prediction, and reduced class-confusion errors.

  • Takeaways & Limitations

    Experiments cover RGB or NIR-RG imagery but have not been systematically extended to SAR or hyperspectral data.

Abstract

from arXiv · show

Semantic segmentation of very-high-resolution (VHR) remote sensing imagery increasingly benefits from strong pretrained hierarchical encoders, yet exploiting their multi-stage representations remains difficult. Nearby regions demand different balances between fine detail and semantic context, aggressive task-specific transformations perturb useful pretrained features, and conventional semantic supervision provides limited structural guidance. We present HAFR-Net, a progressive refinement framework that adaptively organizes and conservatively refines hierarchical representations instead of replacing them with a monolithic decoder transformation. Heterogeneity-Guided Stage-Adaptive Fusion (HG-SAF) predicts dense stage weights conditioned on local feature variation. A Frequency-Residual Adapter (FRA) then injects frequency information through a bounded, zero-initialized residual branch that keeps the fused representation as its reference. A Confusion-Aware Tri-Prior Decoder (CATP) finally regularizes the prediction with boundary, objectness, and training-derived class-relation cues. Under a matched Swin-B training and single-scale inference protocol, HAFR-Net attains 84.12%, 87.86%, 55.17%, and 67.70% mIoU on ISPRS Vaihingen, ISPRS Potsdam, LoveDA, and OpenEarthMap, improving the matched UPerNet baseline by 0.55, 0.95, 1.55, and 1.84 percentage points, respectively. Controlled analyses further show consistent spatial reweighting beyond content-only routing, improved boundary and thin-structure accuracy over matched spatial and spectral alternatives, and reduced confusion on pre-declared class pairs.

I. INTRODUCTION · II. RELATED WORK · A. Hierarchical Feature Fusion for Remote Sensing Segmentation

The paper frames VHR segmentation as progressive, adaptive refinement of hierarchical pretrained features rather than fixed aggregation or unconstrained transformation. HAFR-Net combines heterogeneity-guided fusion, conservative frequency residual refinement, and confusion-aware structural regularization, while positioning HG-SAF against established hierarchical decoder designs.

  • I. INTRODUCTION: VHR imagery contains fine-scale buildings, roads, vehicles, vegetation, and other land-cover elements, while pretrained hierarchical encoders have strengthened modern segmentation systems.The introduction identifies VHR imagery’s detailed observations and the benefits of CNN, Transformer, and state-space encoders.
  • I. INTRODUCTION: Local regions require different balances between high-resolution spatial detail and deep semantic context, limiting fixed top-down, predefined-sum, or image-level fusion strategies.Small vehicles, narrow roads, roof edges, and boundaries favor shallow stages, whereas large roofs, agricultural parcels, and homogeneous regions favor deeper context.
  • I. INTRODUCTION: Unrestricted frequency or global-context transformations can disrupt pretrained representations, motivating conservative refinement for VHR datasets that are smaller than generic pretraining corpora.The paper argues for refining rather than replacing the fused representation.
  • I. INTRODUCTION: Boundary errors, small foreground regions, and semantically similar categories motivate structural supervision and class-relation constraints as lightweight decoding regularizers.These auxiliary signals shape the final feature space while inference remains dominated by a primary segmentation pathway.
  • I. INTRODUCTION: HAFR-Net progressively refines four pretrained Swin-B feature stages using pixel-wise heterogeneity-conditioned HG-SAF, bounded residual FRA, and final confusion-aware tri-prior decoding.The framework preserves the pretrained hierarchy while applying adaptive fusion, conservative feature refinement, and tri-prior decoding.
  • I. INTRODUCTION: HG-SAF formulates VHR decoding as hierarchical adaptive refinement by varying shallow-detail and deep-context contributions across a scene through local feature heterogeneity.The introduction presents this as a departure from generic feature aggregation.
  • I. INTRODUCTION: FRA uses channel-bottlenecked spectral modulation, an explicitly bounded gate, and residual initialization to refine detail from the pretrained spatial representation.The stated design avoids an unconstrained frequency transform and begins from the pretrained representation.
  • A. Hierarchical Feature Fusion for Remote Sensing Segmentation: Established encoder–decoder systems made multi-level aggregation standard, while remote-sensing decoders differ mainly in fixed top-down laterals, global pooling, channel attention, or spatially varying weights.HG-SAF is evaluated against mean fusion, globally pooled gates, and content-adaptive pixel-wise alternatives under the same backbone to isolate the heterogeneity cue.

B. Spatial-Frequency Feature Refinement … A. Overview

HAFR-Net progressively organizes and refines pretrained hierarchical Swin-B representations through adaptive fusion, bounded frequency residual correction, and tri-prior semantic decoding. The merged sections position these components against spatial-frequency, structure-aware, and lightweight-adaptation alternatives while preserving the fused representation as the refinement reference.

  • B. Spatial-Frequency Feature Refinement: Frequency-aware remote sensing models commonly complement spatial processing through wavelet fusion, frequency-band decoupling, FFT-guided filtering, content-conditioned modulation, or frequency-guided denoising.Examples include SFFNet, FDNet, FGNet, AFENet, FreDNet, and WgANet, alongside generic Fourier mixers and operator learning.
  • B. Spatial-Frequency Feature Refinement: FRA attaches frequency modeling as a bounded residual correction inside a channel bottleneck, preserving the fused spatial features as the reference.It is compared at the same insertion point with standard spatial residuals and spectral alternatives.
  • C. Structure-Aware Semantic Decoding: CATP combines boundary, objectness, and prototype-like structural cues in a compact final decoder after hierarchical adaptive refinement.Matched controls test whether this combination improves on conventional auxiliary losses, while the individual mechanisms are established rather than claimed as novel primitives.
  • C. Structure-Aware Semantic Decoding: Transformer and Mamba backbones improve long-range context, while lightweight adapters and LoRA can adapt fixed pretrained models for multimodal remote sensing segmentation.HAFR-Net instead emphasizes how hierarchical encoder features are organized and refined for VHR scenes, retaining Swin-B as its representation backbone.
  • A. Overview: HAFR-Net progressively refines pretrained hierarchical representations instead of replacing them with an increasingly complex task-specific decoder.A pretrained Swin-B encoder produces four stage features that pass through preparation, adaptive fusion, residual refinement, and final decoding.
  • III. METHOD: HG-SAF projects stage features into a common space and adaptively aggregates them into a fused representation before subsequent refinement.The method separates hierarchical information selection, representation refinement, and tri-prior decoding into distinct functions.
  • A. Overview: The Swin-B encoder uses four stages with widths (128, 256, 512, 1024), projects them to (96, 192, 384, 768), and aligns each stage to width Ce=384 at spatial size (H/4, W/4).The encoder is pretrained on ImageNet-22k, fine-tuned on ImageNet-1k, and the aligned stages are prepared before fusion.

B. Stage-Specialized Preparation · C. Heterogeneity-Guided Stage-Adaptive Fusion

Stage-specialized preparation uses fixed SSDB and BCS-Mamba blocks before decoding, while HG-SAF adaptively fuses hierarchical stages using dense, heterogeneity-guided pixel-wise weighting. The fusion is initialized as uniform mean fusion and then learns spatially varying routing from local feature variation.

  • B. Stage-Specialized Preparation: SSDB and BCS-Mamba prepare encoder stages, while SOE operates on the refined feature before decoding; these blocks are implementation components rather than claimed contributions.They remain identical in every module-level comparison, and Section VI-E measures their contribution to the Table II margin.
  • B. Stage-Specialized Preparation: SSDB processes the two high-resolution stages with parallel spectral and spatial branches, gated residual mixing, and a convolutional feed-forward tail.The spectral branch applies a learnable per-channel complex filter to an orthonormal two-dimensional real FFT; the spatial branch uses depthwise and pointwise convolutions.
  • C. Heterogeneity-Guided Stage-Adaptive Fusion: HG-SAF uses dense stage weighting because hierarchical stages differ in resolution, semantic abstraction, and local usefulness across VHR imagery.Its routing is spatially varying rather than globally shared.
  • C. Heterogeneity-Guided Stage-Adaptive Fusion: Each stage is projected to width Ce and upsampled to (H1, W1), after which local feature heterogeneity is computed from the deepest projected representation.The heterogeneity statistic uses a k×k box filter with k=5 and ε=10^-5.
  • C. Heterogeneity-Guided Stage-Adaptive Fusion: A convolutional gate predicts four pixel-wise stage logits from mean stage context and heterogeneity, then normalizes them over the stage axis into w = (w1, . . . , w4)^⊤.The gate uses a 3×3 convolution to Ce/4 channels, normalization, GELU, and a 1×1 convolution to four logits.
  • C. Heterogeneity-Guided Stage-Adaptive Fusion: The gate’s final projection is zero-initialized, so HG-SAF begins with uniform stage weights and mean fusion of the projected stages.The stage projections are trained from scratch, while the zero-initialized gate provides a nonadaptive reference at the first forward pass.
  • C. Heterogeneity-Guided Stage-Adaptive Fusion: Matched controls retain the gate architecture and capacity while removing only heterogeneity input, isolating the statistic’s effect from pixel-wise routing capacity.The overall architecture then passes the fused feature to FRA, SOE, and CATP, with auxiliary heads used only during training.

D. Frequency-Residual Adapter · E. Confusion-Aware Tri-Prior Decoder

HAFR-Net conservatively refines fused features with a bounded frequency residual, then regularizes semantic decoding using boundary, foreground, and class-relation priors. The decoder uses these auxiliary signals during training while retaining only semantic logits at inference.

  • D. Frequency-Residual Adapter: FRA constructs a bounded residual correction from frequency modeling instead of replacing the fused representation with an unrestricted spectral transformation.The spectral branch uses a channel bottleneck with width r = max(Ce/4, 16).
  • D. Frequency-Residual Adapter: Both frequency transforms are orthonormal on the half spectrum, producing real outputs while combining log-amplitude, phase sine and cosine, and a radial prior.The gate uses α=0.5, and γ is initialized to 0.
  • D. Frequency-Residual Adapter: The bottleneck limits spectral-branch cost and residual channel rank, while the multiplier satisfies |g −1| < α to constrain relative rescaling.At inference, FRA remains a single residual correction on Ffuse without an auxiliary prediction head.
  • E. Confusion-Aware Tri-Prior Decoder: CATP adds lightweight auxiliary signals because a single semantic head supplies only region-level supervision after fusion and refinement.These signals regularize boundaries, foreground structure, and inter-class relations during training.
  • E. Confusion-Aware Tri-Prior Decoder: CATP predicts semantic logits, objectness logits, and a 64-channel prototype embedding from the refined fused features.The objectness target marks valid pixels outside dataset-specific background-like classes, while the edge target is the 4-neighbour class-difference mask.
  • E. Confusion-Aware Tri-Prior Decoder: Class-relation regularization targets pairs with substantially larger cross-class errors, using prototype similarities derived from a pilot model trained on an internal training split.The official evaluation split is never used for pair selection; prototypes are mini-batch means of Q over valid pixels and are re-normalized to unit ℓ2 norm.
  • E. Confusion-Aware Tri-Prior Decoder: With m=0.2, a relation pair stops contributing when ⟨pa, pb⟩≤−m; the six-class ISPRS label set contains four pairs, and inference uses only Psem.Pair lists for all four datasets are provided in the supplement.

F. Conservative Initialization of Refinement Branches · G. Training Objective

HAFR-Net initializes its refinement branches conservatively, preserving baseline-equivalent or residual reference behaviors during fine-tuning. Its training objective combines semantic, structural, prototype, and auxiliary supervision while preserving label-resolution semantics and disabling auxiliary heads at inference.

  • F. Conservative Initialization of Refinement Branches: HG-SAF begins with baseline-equivalent uniform stage fusion, FRA is an exact residual identity at γ=0, and CATP starts with a small residual coefficient βe.These distinct reference behaviors allow later ablations to isolate initialization from architecture.
  • F. Conservative Initialization of Refinement Branches: The initialization records establish reference behaviors before the training objective introduces semantic and auxiliary structural supervision.The next subsection defines the semantic head objective and auxiliary structural signals.
  • G. Training Objective: The valid-pixel set excludes ignore index 255, keeping unlabeled and eroded-boundary pixels out of LCE and LFocalDice.The objective is evaluated only over Ω={i: yi≠255}.
  • G. Training Objective: The semantic loss is Lsem=LCE+λdLFocalDice, using DSCc with εd=1, one-hot targets, and softmax probabilities.These components define the semantic head supervision.
  • G. Training Objective: Class-balanced cross-entropy uses Car 3× and Clutter 5× weights on ISPRS, with Ncls scored classes and κ=2.5.These settings specify the class-balancing configuration.
  • G. Training Objective: The full objective is L=Lsem+λeLedge+λoLobj+λpLproto+Laux, combining semantic, edge, objectness, prototype, and auxiliary terms.The auxiliary term attaches CE and FocalDice heads to prepared stages.

IV. EXPERIMENTAL SETUP · A. Datasets and Exact Splits

The evaluation covers four VHR remote-sensing benchmarks with dataset-specific imagery, class legends, official splits, and scoring protocols. All inputs are standardized to 512×512 with nearest-neighbor label resampling, ImageNet normalization, preserved ignore labels, and no post-processing.

  • A. Datasets and Exact Splits: 33 NIR-RG tiles at 9 cm GSD define ISPRS Vaihingen, with 16 official training tiles and 17 test tiles.The test set comprises 113 non-overlapping 1024×1024 patches used for evaluation and bootstrapping.
  • A. Datasets and Exact Splits: Five foreground classes are scored on Vaihingen, excluding Clutter and applying official 3 px boundary erosion with ignore index 255.
  • A. Datasets and Exact Splits: 38 RGB tiles at 5 cm GSD define ISPRS Potsdam, using the listed train and test tiles under the same five-class eroded protocol.
  • A. Datasets and Exact Splits: 5987 RGB images at 30 cm GSD define LoveDA, which is evaluated on its official validation split across seven classes.
  • A. Datasets and Exact Splits: 5000 OpenEarthMap images at 25–50 cm GSD are scored by mIoU over eight official land-cover classes, with unlabeled pixels ignored.OpenEarthMap has no separate Background class.
  • A. Datasets and Exact Splits: 512×512 inputs are used for every image, with ISPRS patches and benchmark tiles resampled accordingly and labels resized by nearest neighbour.ImageNet mean and standard deviation normalization is used; ignore indices are preserved, and no CRF, multi-crop merging, or other post-processing is applied.

B. Controlled Baselines and Published Comparisons … A. Controlled Comparison

The paper grounds its conclusions in a matched Swin-B decoder comparison with shared training and inference conditions, while separately documenting implementation, evaluation metrics, and uncertainty. HAFR-Net improves the controlled UPerNet baseline across all four datasets, with larger gains on heterogeneous benchmarks.

  • B. Controlled Baselines and Published Comparisons: Controlled conclusions use Swin-B models retrained with one recipe, while published CNN, Transformer, and Mamba results provide contextual rather than strictly controlled comparisons.The controlled candidates include UPerNet, FPN, DeepLabv3+, SegFormer MLP, a UNetFormer-style decoder, SE-gated fusion, and an FFT mixer.
  • C. Implementation Details: Every controlled model shares 512×512 inputs, an 80-epoch maximum, batch size 8, AdamW, cosine scheduling with warmup, weight decay, clipping, BF16, and flip/rotation augmentation.Training uses PyTorch 2.1 on one RTX 4090; head and backbone learning rates are 1.2×10−4 and 4×10−5.
  • C. Implementation Details: Seeds 42, 43, and 44 are used, with the best-mIoU checkpoint selected under one identical monitoring rule; auxiliary supervision weights compare decoding paths rather than training-signal quantity.The auxiliary weights are (0.4, 0.3, 0.2, 0.1).
  • C. Implementation Details: Controlled inference is single-scale without TTA, while ISPRS results additionally separate flip and rotation TTA gains; LoveDA alone uses stronger photometric augmentation, EMA, and early stopping.ISPRS and OpenEarthMap use none of LoveDA’s color jitter, Gaussian blur, EMA, or early-stopping additions.
  • D. Evaluation Metrics: Evaluation reports mIoU and per-class IoU, with stratified IoU, stage-weight, boundary, thin-structure, small-component, and pairwise-confusion analyses assigned to the corresponding modules.Thin-structure IoU uses local width below 8 px, small-component IoU uses area quartile Q1, and CATP evaluates pre-declared class pairs.
  • A. Controlled Comparison: Table II fixes the encoder, schedule, training, split, augmentation, checkpoint rule, and inference setting, isolating differences between rows to decoding paths.HAFR-Net adds the fixed Section III-B blocks to its three refinement stages, whose separate contribution is measured later.
  • A. Controlled Comparison: +0.55 pp, +0.95 pp, +1.55 pp, and +1.84 pp are HAFR-Net’s no-TTA improvements over Swin-B+UPerNet on Vaihingen, Potsdam, LoveDA, and OpenEarthMap, respectively.The corresponding 95% confidence intervals exclude zero, and the ordering persists across seeds; gains are larger on LoveDA and OpenEarthMap.
  • A. Controlled Comparison: The controlled comparison attributes the observed margin to decoding design rather than a dataset-specific trick.This interpretation follows the shared protocol and matched inference conditions.

B. Contextual Comparison with Published Results · VI. ANALYSIS AND ABLATION

Published results position HAFR-Net among the strongest methods across four benchmarks, while differing protocols make them contextual rather than strict comparisons. The ablation analysis instead fixes the Swin-B encoder and swaps only the component under test to examine whether each refinement stage behaves as designed.

  • B. Contextual Comparison with Published Results: HAFR-Net achieves the best or second-best reported accuracy on each of the four benchmarks in contextual comparisons.These comparisons span CNN-, Transformer-, and Mamba-based models but use different backbones, training recipes, and inference settings.
  • B. Contextual Comparison with Published Results: 84.48% mIoU on Vaihingen and 88.20% on Potsdam are reported with test-time augmentation (TTA).HAFR-Net has the highest Potsdam mean, while PyramidMamba exceeds its Vaihingen mean by 0.36 percentage points under a different backbone and TTA setting.
  • B. Contextual Comparison with Published Results: 67.70% mIoU is obtained on OpenEarthMap under the eight-class official protocol without TTA.The largest reduction in confusion mass among the pre-declared relation pairs occurs for Rangeland ↔Agriculture.
  • B. Contextual Comparison with Published Results: HAFR-Net leads the published means on Potsdam and OpenEarthMap and trails the best entries by 0.36 and 0.13 percentage points on Vaihingen and LoveDA.The published values are literature context because backbones, splits, training recipes, and inference settings differ; Table II provides the matched Swin-B comparison.
  • B. Contextual Comparison with Published Results: The urban-benchmark contextual tables score five foreground classes with Clutter excluded and apply the official 3-pixel boundary erosion.Methods marked † use TTA as reported by their sources, reinforcing that these are not matched benchmark comparisons.
  • B. Contextual Comparison with Published Results: LoveDA is evaluated on the official validation split with single-scale inference and no TTA across seven classes.Several published Transformer and Mamba entries came from the online test server and are shown only as literature context.
  • VI. ANALYSIS AND ABLATION: The ablation section asks whether each refinement stage behaves as designed rather than merely reporting overall accuracy.It keeps the Swin-B encoder fixed, swaps only the component under test, and concludes with visual examples of the identified strata.
  • VI. ANALYSIS AND ABLATION: 84.48% is the full-model ISPRS reference entry for analyses using the Table III flip-and-rotation protocol unless a table note specifies otherwise.This establishes the evaluation reference used for the controlled component analyses.

A. Stage-Adaptive Fusion · B. Frequency-Residual Adaptation

HG-SAF uses heterogeneity-guided stage routing to emphasize detail in heterogeneous regions while retaining deep semantic context. FRA adds a conservative spectral residual whose bounded low-rank parameterization improves structural metrics over matched alternatives.

  • A. Stage-Adaptive Fusion: HG-SAF decomposes its +0.20 pp gain over mean fusion into +0.04 from global pooling, +0.06 from mean-context pixel routing, and +0.14 from heterogeneity.The heterogeneity statistic carries the largest contribution under matched fusion-stage replacements.
  • A. Stage-Adaptive Fusion: +1.57 pp in the highest heterogeneity quartile and +1.52 pp on small objects exceed the +0.19 pp change for large objects.HG-SAF’s largest stratified gains occur where heterogeneity and fine detail are most pronounced.
  • A. Stage-Adaptive Fusion: HG-SAF’s heterogeneity statistic tracks object outlines and small structures, while its predicted weight field varies spatially and redistributes stage emphasis toward detail.The routing emphasizes detail in the same strata where the largest gains are reported, while pretrained deep semantic context remains dominant.
  • A. Stage-Adaptive Fusion: Fused maps retain shallow edge responses in heterogeneous regions and deeper homogeneous responses in large interiors.This qualitative pattern corresponds to the routing statistics reported for HG-SAF.
  • B. Frequency-Residual Adaptation: Table VII compares FRA with matched spatial and spectral alternatives to separate residual-branch, frequency-operator, and bounded low-rank effects.All variants are evaluated at the same fused-feature location.
  • B. Frequency-Residual Adaptation: A GFNet mixer with the same zero-init residual parameterization recovers part of the gain and ranks second-best on four of five accuracy metrics.This supports residual parameterization as a contributor to performance.
  • B. Frequency-Residual Adaptation: FRA’s bounded low-rank design further improves Boundary IoU and thin-structure IoU over the matched alternatives.The comparison indicates that the additional design, beyond residual parameterization, benefits structural accuracy.
  • B. Frequency-Residual Adaptation: The constraints apply to every coefficient alike, so FRA’s margin follows from bounded low-rank parameterization rather than preference for any spectral region.FRA is characterized as a conservative spectral residual, not a generic frequency-module replacement.

C. Tri-Prior Decoding · D. Component Complementarity and Initialization · E. Fixed Preparation Blocks

CATP improves structural prediction beyond independently added auxiliary losses by coupling boundary, objectness, and class-relation priors. Across component, initialization, and preparation analyses, HAFR-Net’s modules are complementary, conservatively initialized, and supported by fixed preprocessing blocks.

  • C. Tri-Prior Decoding: 23–28% lower pairwise confusion mass occurs on every reported pre-declared class pair under CATP.The largest reduction is for OpenEarthMap Rangeland ↔Agriculture, where the hinge applies pressure.
  • C. Tri-Prior Decoding: 0.16 pp separates CATP from the combined boundary, objectness, and pixel-contrast auxiliary-loss alternative.Each individual auxiliary loss improves the CE/FocalDice reference by at most 0.05 pp, indicating the gain comes from coupling structural signals.
  • D. Component Complementarity and Initialization: Reference initialization attains the highest mIoU among the compared initialization strategies.Table IX evaluates conservative, random, and fully active starts over the component combinations; stability is reported over three seeds.
  • D. Component Complementarity and Initialization: mIoU increases along every component-combination path on all four datasets.Each module is beneficial alone, every pair improves on both members, and the full combination is best on all four datasets.
  • D. Component Complementarity and Initialization: +0.20, +0.17, and +0.22 pp result from restoring HG-SAF, FRA, and CATP, respectively, to pairs on Vaihingen.Pairwise interactions are IHF= +0.04, IHC= −0.11, and IFC= −0.09 pp, remaining below the main effects.
  • E. Fixed Preparation Blocks: +0.09, +0.14, and +0.07 pp come from SSDB, BCS-Mamba, and SOE individually, while their joint contribution is +0.22 pp.The joint configuration is the preparation stage adopted inside HAFR-Net.
  • E. Fixed Preparation Blocks: +0.22 pp of HAFR-Net’s +0.55 pp Vaihingen gain over the reference decoder comes from fixed preparation blocks, with +0.33 pp from refinement stages.All module analyses in Tables VI–IX keep preparation blocks fixed; the comparison uses single-scale, no-TTA Vaihingen mIoU.

F. Qualitative Comparison · VII. EFFICIENCY AND LIMITATIONS · A. Efficiency

Qualitative examples localize HAFR-Net’s measured gains to small objects, thin structures, heterogeneous regions, and pre-declared class pairs rather than uniformly improving large homogeneous interiors. Under matched Swin-B re-implementations, these gains require moderate overhead, including 9.5% more parameters, 10.8% more GFLOPs, 15.2% more peak memory, and 13.7% lower throughput.

  • F. Qualitative Comparison: Qualitative predictions across four benchmarks repeatedly expose gains in small objects, thin structures, heterogeneous regions, and pre-declared class pairs.The figures do not show uniform improvement over large homogeneous interiors.
  • F. Qualitative Comparison: +1.52 pp stratified IoU improves the ISPRS small-object stratum, while vehicles remain separated from adjacent impervious surfaces and building boundaries.The examples correspond to the HG-SAF and CATP regimes analyzed separately.
  • F. Qualitative Comparison: Thin rural roads remain continuous in LoveDA despite partial occlusion, while low-texture Barren regions resist absorption into Background.These examples correspond to thin-structure IoU and the Barren–Background confusion case.
  • F. Qualitative Comparison: LoveDA’s mixed urban and rural tiles place more pixels in upper heterogeneity quartiles, where Table VI reports the largest stratified gains.This helps account for part of the dataset-level pattern.
  • F. Qualitative Comparison: OpenEarthMap examples show gradual Rangeland ↔Agriculture transitions and thin roads, matching the dataset’s pre-declared pair with the largest relative confusion-mass reduction.The transition requires selecting a boundary across a broad gradient rather than locating a sharp visible edge.
  • VII. EFFICIENCY AND LIMITATIONS: The efficiency comparison reports parameters, GFLOPs, peak memory, latency, and FPS for re-implemented Swin-B decoders under a matched protocol, excluding published mixed-protocol models.The matched comparison is intended to state accuracy gains alongside comparable cost.
  • A. Efficiency: HAFR-Net adds 9.5% parameters, 10.8% GFLOPs, and 15.2% peak memory, lowering throughput from 46.7 to 40.3 FPS by 13.7% for +1.22 pp four-dataset mean mIoU without TTA.Its 24.8 ms forward pass includes a 2.9 ms FP32 FFT cost per 512×512 tile, about 12% of the pass.
  • A. Efficiency: +1.15 pp four-dataset mean mIoU accompanies 1.3 ms and 4.0 M additional cost versus the matched FFT mixer, whose reference is 23.5 ms and 93.8 M parameters.At batch size 1, the full model still requires 24.8 ms per 512×512 tile on one RTX 4090; the next subsection addresses unresolved cases.

B. Limitations · VIII. CONCLUSION

HAFR-Net adaptively and conservatively refines pretrained hierarchical representations for VHR segmentation, achieving benchmark gains with moderate reported compute overhead. Remaining limitations concern fixed heterogeneity estimation, FP32 FFT evaluation, and non-adaptive training-derived relation sets.

  • B. Limitations: H is a parameter-free fixed functional of learned ˜F4, so feature-variable but semantically homogeneous regions can produce boundary-like responses.Its association with shallow-stage weight is weak, motivating learned heterogeneity measures as future work.
  • VIII. CONCLUSION: Future work will investigate learned heterogeneity measures, adaptive relation discovery, and additional remote sensing modalities.These directions correspond to the stated limitations and extend the proposed refinement framework.
  • B. Limitations: FRA evaluates FFTs in FP32 for numerical stability, and the reported latency reflects this conservative variant.Half-precision or windowed spatial approximations could reduce the cost, but preservation of Table VII boundary metrics remains unverified.
  • B. Limitations: The prototype relation set is derived from a training-only pilot split and does not adapt online to deployment-domain confusions.Consequently, newly confusable class pairs receive no hinge; the passage’s discussion of candidate-count limitations is truncated.
  • VIII. CONCLUSION: +0.33 to +1.52 pp mIoU gains over the strongest matched baseline were obtained across four VHR benchmarks with moderate, fully reported compute overhead.The conclusion frames HAFR-Net as organizing added decoder capacity progressively rather than adding unconstrained capacity.
  • VIII. CONCLUSION: Module-specific analyses demonstrate spatially varying stage routing, improved boundary and thin-structure prediction, and reduced structural and class-confusion errors.These findings support adaptive and conservative refinement of strong pretrained representations as an effective design principle for accurate VHR segmentation.
Loading 2608.15647v1…