Source-linked AI summary
AAMBERS-UAV: Acquisition-Aware Multimodal Backbone Evaluation and Ranking for UAV Weedy Rice Segmentation
Tarek Rahman, Nazim-E-Alam, Md Kishor Morol, Jannatun Noor
TL;DR
Image-level splits can place related UAV frames from one acquisition in both development and testing, leaving transfer to a new survey unclear. The study compares target-held-out and target-exposed development with the same target test set. SegFormer-B0’s leading modality changes from RGB under holdout to RGB+MS after exposure.
Problem
Image-level splits can put related frames from one acquisition in development and test, leaving unclear whether scores reflect transfer to a genuinely new survey.
Source: §1 · p. 1Method
The study compares target-held-out and target-exposed development while keeping the target test set fixed and quantifies protocol and modality differences on identical test images.
Source: §1 · p. 2, §3.4 · p. 5Results
Exposure increased measured performance across both evaluated backbones, while modality ranking depended on backbone; SegFormer-B0’s RGB result was 0.7317 ± 0.0201 IoU under holdout, and RGB+MS was strongest at 0.7822 ± 0.0269 after exposure.
Source: §4.1 · pp. 5–6, §7 · p. 9Takeaways & Limitations
For the evaluated early-fusion model, RGB–MS alignment matters more than moderate global radiometric scaling in the tested corruptions.
Source: §4.4 · p. 7The target differs from source acquisitions in date, location, weed prevalence, phenotype, illumination, and altitude, so the observed shift cannot be attributed to one variable.
Source: §6 · p. 9
Abstract
from arXiv · showhide
UAV image collections contain spatially and temporally related frames, yet semantic-segmentation benchmarks commonly split them at image level. Such splitting can place samples from one acquisition in both model development and testing, obscuring transfer to a genuinely new survey. Using the 734-sample WeedyRice-RGBMS-DB, we fix a 124-image target-acquisition test set and compare two protocols with identical train, validation, and test counts: target-held-out, which excludes the target acquisition from development, and target-exposed, which admits its remaining images. SegFormer-B0 is evaluated with RGB, four-band multispectral (MS), and seven-channel RGB+MS input over two fixed-split seeds. RGB is strongest under complete acquisition holdout ($0.7317\pm0.0201$ IoU), whereas RGB+MS becomes strongest after target exposure ($0.7822\pm0.0269$). A fixed-split U-Net/ResNet18 replication confirms positive exposure gains for all three inputs, but retains RGB as the best modality under both protocols. Acquisition exposure therefore increases measured performance across both evaluated backbones, while its effect on modality ranking is architecture-dependent. A supplied-split audit reveals strong near-sequential dependence, and corruption tests show that early fusion is substantially more sensitive to RGB--MS displacement than to moderate radiometric scaling. These results support acquisition-aware same-test evaluation as a necessary complement to ordinary image-level splitting in multimodal UAV benchmarks. The code and supporting the findings of this study will be publicly released upon acceptance of the paper.
1 Introduction
Image-level splits can place related frames from one UAV acquisition in both development and testing, leaving unclear whether results transfer to a genuinely new survey. The study asks whether target-acquisition exposure changes modality rankings and tests this with a same-test design.
1 Introduction
- Image-level splits can put nearby frames from one acquisition in development and test, making scores reflect within-survey interpolation rather than transfer to a new survey.Nearby frames can share field content, illumination, crop stage, altitude, sensing geometry, and sensor calibration.Source: §1 · p. 1
- The study asks whether target-acquisition exposure changes the apparent benefit and ranking of RGB, MS, and RGB+MS inputs, and whether rankings persist across backbones.The same-test design changes only whether remaining target-acquisition images are available during development.Source: §1 · p. 2
- The contributions include an audit of temporal, frame-sequence, and geospatial dependence and an equal-count comparison of target-held-out and target-exposed development.The target test set is fixed across protocols.Source: §1 · p. 2
- The study evaluates RGB, MS, and RGB+MS with two-seed SegFormer-B0 and a fixed-split U-Net/ResNet18 replication, plus interaction, coverage, and sensor-corruption analyses.The analyses distinguish acquisition-exposure effects from architecture-dependent multimodal behavior.Source: §1 · p. 2
2 Related Work
Prior work spans UAV weed mapping, multimodal segmentation, and structured validation. This study applies acquisition-level evaluation to examine dependence that image-level splits may leave untested.
UAV weed mapping and multimodal segmentation
- UAV weed-mapping studies use multispectral orthomosaics and vegetation indices, while weedNet demonstrates dense multispectral crop–weed classification from aerial imagery.Related work also covers real-time rice-weed segmentation and dense outputs for spatial management.Source: §2 · p. 2
- A recent review notes that UAV weed-mapping research still relies on relatively few benchmarks and has limited evidence of transfer across environments.The passage cites Ahmad et al. (2026).Source: §2 · pp. 2–3
Structured dependence in model validation
- WeedyRice-RGBMS-DB’s registered RGB and four-band multispectral imagery supports testing acquisition exposure across a Transformer and a conventional CNN.The paper’s objective is to isolate how exposure changes measured modality benefits, rather than propose a new fusion network.Source: §2 · p. 3
- Structured-validation work recommends blocking when deployment targets lie outside observed spatial, temporal, or hierarchical structure; this study applies that principle at acquisition level with a fixed target test set.Prior studies cited in the passage report inflated CNN scores from spatial autocorrelation and weaker transfer under spatial validation.Source: §2 · p. 3
3 Experimental Design and Methodology
The methodology audits acquisition structure, compares target-held-out and target-exposed development on a fixed test set, and evaluates three input configurations across two backbones. Full-resolution evaluation and separate uncertainty estimates support the protocol and modality comparisons.
3.1 Dataset and Acquisition Audit
- The 734 aligned image–mask pairs span four Mekong Delta surveys, whose weed coverage, acquisition size, and altitude vary jointly; the supplied split also shows near-sequential dependence.Table 1 summarizes frame and geospatial proximity statistics.Source: §3.1 · pp. 3–4, Table 1 · p. 4
3.2 Same-Test Acquisition Protocols
- Both protocols use the same 124-image target test set and 390 training plus 97 validation images, but only target-exposed development includes target-acquisition images.Splits are assigned at full-image level before cropping, preventing overlapping crops from one image from crossing partitions.Source: §3.2 · p. 4, Figure 3 · p. 4
3.3 Inputs, Architectures, and Training
- SegFormer-B0 compares RGB, four-band multispectral, and seven-channel early RGB+MS inputs, with a U-Net/ResNet18 replication testing backbone dependence.For multispectral and fused inputs, the first encoder layer is adapted while retaining ImageNet initialization for pretrained channels.Source: §3.3 · pp. 4–5
3.4 Full Resolution Evaluation and Statistical Analysis
- Full-resolution scores use overlapping tiles, validation-selected thresholds, and aggregate confusion counts; two-seed standard deviations and paired image-bootstrap uncertainty are reported separately.The interaction measure tests whether exposure changes RGB+MS’s advantage over RGB; positive values indicate an increased advantage.Source: §3.4 · p. 5
4 Results
Across the fixed target test set, acquisition exposure improves measured performance, but modality ranking varies by backbone; the supplied split also shows near-sequential dependence, and fusion is sensitive to alignment.
4.1 Primary SegFormer Results
- 0.7317 ± 0.0201 IoU makes RGB strongest under target holdout, while RGB+MS leads after exposure at 0.7822 ± 0.0269 IoU.These comparisons use the identical 124-image target test set; Table 2 reports mean ± standard deviation over two fixed-split seeds.Source: §4.1 · pp. 5–6, Table 2 · p. 6
4.2 Exposure Gains and Fusion–RGB Interaction
- All SegFormer inputs improve under target exposure in both seeds; mean gains are 0.0350 ± 0.0127 IoU for RGB, 0.0164 ± 0.0083 for MS, and 0.0808 ± 0.0430 for fusion.The fusion–RGB interaction is positive in both runs: +0.0244 for seed 17 and +0.0673 for seed 43.Source: §4.2 · p. 6
4.3 U-Net/ResNet18 Backbone Replication
- U-Net/ResNet18 gains were +0.0489 [0.0421, 0.0565] for RGB, +0.0467 [0.0290, 0.0632] for MS, and +0.0164 [0.0042, 0.0279] for fusion; RGB remained best under both protocols.Table 3 reports the fixed-split seed-17 replication.Source: §4.3 · p. 6, Table 3 · p. 6
4.4 Supplied Split Dependence and Sensor Robustness
- The supplied split has strong near-sequential dependence: median nearest same-date training-frame gap is one, and median nearest training GPS distance is 5.42–5.44 m.The audit does not support a claim of pixel-identical duplication; the concern is dependence that may overstate transfer to a new survey.Source: §4.4 · p. 6
- Moderate MS scaling changes IoU by at most 0.0014, while 4-pixel displacement costs 0.0327, 8-pixel displacement 0.1411, and complete MS loss 0.1450.The inference-only corruption analysis concerns the seed-17 held-out SegFormer RGB+MS model; Figure 5 marks clean IoU with a dashed line.Source: §4.4 · pp. 6–7, Figure 5 · p. 6
4.5 Coverage Dependence and Compute Cost
- Mean image IoU rises from 0.564 for eight low-coverage images to 0.662 for 62 medium-coverage and 0.817 for 54 high-coverage images.The low-coverage group contains only eight images, making early-infestation conclusions limited; the 12 SegFormer runs used 7.48 GPU-hours for training.Source: §4.5 · p. 7, Table 4 · p. 7
4.6 Cross-Backbone Interaction Analysis
- SegFormer shifts from RGB under holdout to RGB+MS after exposure, while U-Net retains RGB; opposite interaction signs show modality ranking depends on backbone.The seed-17 comparison finds the largest exposure gain for SegFormer RGB+MS and U-Net RGB; the authors do not establish the mechanism or a universal fusion advantage.Source: §4.6 · pp. 7–8, Figure 7 · p. 8
5 Discussion
The same-test comparison associates target-acquisition exposure with higher measured performance, while modality ranking differs across the evaluated backbones. The discussion connects these findings to deployment choices for target-informed adaptation and unseen-acquisition transfer.
5.1 What the same-test comparison reveals
- With identical target test images and equal development-set sizes, exposure gains reflect the practical value of labelled examples from the deployment acquisition.The held-out protocol tests transfer to a survey absent from development; the exposed protocol reflects performance after examples from that survey are available.Source: §5.1 · p. 8
- Modality ranking does not transfer cleanly across backbones: SegFormer shifts from RGB to RGB+MS after exposure, while U-Net favours RGB in both settings.The suggested role of SegFormer attention and pixel-aligned spectral cues is an interpretation, not a demonstrated mechanism.Source: §5.1 · pp. 8–9
5.2 Implications for multimodal deployment
- Early fusion depends on reliable RGB–MS registration and continued spectral-sensor access; the authors recommend checking alignment and band validity at deployment.They identify modality dropout, later fusion, explicit feature alignment, and separate unimodal models as directions for improving failure behaviour.Source: §5.2 · p. 9
- With labelled calibration data from a new acquisition and verified registration, fusion may help, particularly for SegFormer; without target labels, RGB was strongest in both evaluated backbones.The discussion presents RGB as a reasonable unseen-acquisition baseline, not a universal sensor ranking.Source: §5.2 · p. 9
6 Limitations and Reproducibility
The study’s conclusions are bounded by the target’s differences from source acquisitions, sparse low-coverage cases, and limited architecture and seed replication; it also documents safeguards and retained artifacts for replication.
6 Limitations and Reproducibility
- Generalization is limited by a single target domain with multiple concurrent shifts, only eight images below 0.30 weed coverage, two SegFormer seeds, and a one-seed U-Net replication.Only early concatenation is evaluated, and polygon annotations may retain spatially structured uncertainty after expert review.Source: §6 · p. 9
- The split, training-only normalization, validation-only threshold selection, shared 124 test identifiers, and paired resampling support controlled comparisons; artifacts are retained for replication.The complete 18-run experiment required approximately 11.69 T4 GPU-hours for training and 0.61 hours for full-resolution evaluation.Source: §6 · p. 9
7 Conclusion
Acquisition exposure increases measured segmentation performance across both evaluated backbones, but its effect on modality ranking depends on the backbone. The findings support acquisition-aware same-test evaluation as a complement to ordinary image-level splitting.
7 Conclusion
- Acquisition exposure consistently increases measured UAV weedy-rice segmentation performance across the two evaluated backbones.The conclusion frames exposure as a general evaluation concern in these experiments.Source: §7 · p. 9
- SegFormer shows a protocol-dependent ranking reversal, while U-Net retains RGB as the best input under both protocols.RGB is strongest for SegFormer under complete acquisition holdout; RGB+MS is strongest after target exposure.Source: §7 · p. 9
- The findings support acquisition-aware same-test evaluation as a necessary complement to ordinary image-level splitting in multimodal UAV benchmarks.Source: §7 · p. 9