PaperlayerView exact revision on arXiv

Source-linked AI summary

SULAND v2: A Refined RGB Dataset and Deep Learning Object Detection Benchmark for UAV/UGV-Based SUrface LANDmine Detection Under Domain Shift

Sagar Lekhak, Prasanna Reddy Pulakurthi, Lalit Joshi, Ramesh Bhatta, Emmett J. Ientilucci

arXiv:2607.28996v1cs.CV

TL;DR

RGB surface-mine detectors lack broad, reliable benchmarking under realistic domain shift. This paper refines SULAND and evaluates 35 detector configurations across IID and OOD settings, finding that most models perform strongly IID but vary substantially under distribution shift. The results establish SULAND v2 as a consistent benchmark for studying robustness beyond the training environment.

  • Problem

    Existing RGB mine datasets provide limited evidence about detector generalization across shifted deployment conditions, weakening comparisons of robustness under domain shift.

  • Method

    The paper audits and re-annotates SULAND v1 into SULAND v2, validates the benchmark, and compares 35 detector configurations across IID and OOD settings.

  • Results

    Most models achieved strong IID performance, whereas OOD behavior varied substantially; YOLO-family models generally led IID accuracy, while RF-DETR-Large and Faster R-CNN retained more performance under geographic and environmental shift.

  • Takeaways & Limitations

    SULAND v2 provides a more consistent basis for comparing detectors and studying robustness beyond the training environment, while RGB detection remains survey-support for visible surface targets.

  • Takeaways & Limitations

    The findings cover two surrogate or inert surface-laid target classes and transfer between Italian and USA subsets, which do not represent the full diversity of operational minefields.

Abstract

from arXiv · show

RGB imagery offers a practical, low-cost option for Unmanned Aerial/Ground Vehicle (UAV/UGV) survey support in surface-landmine detection, but object detectors remain underexplored in this safety-critical domain. Limited cross-architecture benchmarking and insufficient out-of-distribution (OOD) analysis obscure whether detectors generalize across deployment conditions. This challenge is amplified by the scarcity of public RGB landmine datasets, making SULAND a key benchmark for PFM-1 and PMA-2 detection. However, inspection reveals missing/false annotations, localization errors, inconsistent visibility criteria, visual artifacts, temporal labeling inconsistencies, and an inverted OOD class-ID convention in SULAND. We present SULAND_v2, a refined RGB surface-landmine dataset and benchmark. Preserving original images and splits, we manually revise annotations to ensure completeness, precise localization, label validity, and class consistency. SULAND_v2 contains 33,771 images and 12,433 bounding boxes. We benchmark 35 detector configurations across nine families. Annotation refinement improves YOLOv8 in-distribution (IID) test mAP@50 by 14.6-19.6 percentage points, while fixing the OOD class-ID convention increases mean YOLOv8 OOD mAP@50 by ~25 percentage points. On SULAND_v2, YOLOv12-Small achieves the highest IID mAP@50 (0.908), while RF-DETR-Large yields the strongest OOD performance (0.799 mAP@50, 0.675 recall). Our results demonstrate that high IID accuracy does not guarantee operational readiness. SULAND_v2 provides a reliable benchmark for evaluating domain-shift robustness in RGB-based mine-action survey support.

I. INTRODUCTION

Surface-landmine detection is safety-critical because demining is hazardous and real survey environments produce substantial domain shifts. This paper addresses limited RGB benchmarks and SULAND v1 quality problems by auditing the dataset, constructing SULAND v2, and evaluating detectors under IID and OOD conditions.

  • Motivation: Humanitarian demining remains slow, hazardous, and resource-intensive, motivating remote sensing and computer vision for evidence-based survey and clearance.Operational mine action proceeds through non-technical survey, technical survey when required, and targeted clearance.
  • Motivation: UAVs and UGVs offer flexible, high-resolution data collection in areas unsafe or inefficient to survey manually, with RGB imagery providing a practical sensing modality.Prior mine-action sensing has also explored thermal, multispectral, hyperspectral, radar, magnetometer, electromagnetic, and metal-detection approaches.
  • Research gap: OOD evaluation is essential because surface mines vary across environments, illumination, camera geometry, target distance, geographic region, and occlusion, while small targets resemble natural clutter.Detector performance can degrade substantially when test distributions differ from training distributions.
  • Research gap: RGB surface-landmine detection lacks broad, standardized benchmarking, while scarce public datasets and annotation errors hinder reliable comparison and interpretation of generalization failures.Relevant issues include missing labels, false positives, inaccurate boxes, inconsistent visibility criteria, class-map mismatches, and limited environmental or protocol diversity.
  • Contributions: The study audits SULAND v1, constructs SULAND v2 with consistent annotations and class conventions, and benchmarks object detectors under both IID and OOD settings.The refinement preserves the original imagery and split organization while correcting bounding boxes and harmonizing IID/OOD class definitions.
  • Contributions: The analysis compares detector ranking, class-specific precision and recall, and tradeoffs among detection accuracy, inference speed, and OOD robustness.It aims to expose limitations not apparent from IID performance alone and guide detector selection under differing computational and evaluation requirements.

II. LITERATURE REVIEW … C. UAV/UGV-Based RGB Datasets for Surface Landmine Detection

UAV/UGV platforms and RGB imagery support flexible mine-action surveying, while deep-learning detectors demonstrate feasibility for visible surface targets. However, public RGB datasets remain scarce and domain variability limits robust evaluation across deployment conditions.

  • A. UAV/UGV-Based Remote Sensing and RGB Imagery for Mine Action: UAVs and UGVs enable flexible, low-altitude, repeatable sensing over areas that are difficult or unsafe to access directly.In mine action, remote sensing supports survey, prioritization, and risk reduction rather than replacing manual or mechanical clearance.
  • A. UAV/UGV-Based Remote Sensing and RGB Imagery for Mine Action: UAV/UGV mine-action sensing spans RGB, thermal infrared, multispectral, hyperspectral, LiDAR, SAR, EMI, magnetometry, and GPR modalities.These modalities address different detection requirements, including temperature, material, soil, vegetation, metallic, and buried-target contrasts.
  • A. UAV/UGV-Based Remote Sensing and RGB Imagery for Mine Action: RGB imagery achieved the highest detection rate for visible surface objects in a standardized seeded minefield, while geophysical methods better addressed metallic or buried targets.Sensor effectiveness depended on target type, visibility, material composition, and deployment configuration.
  • B. Deep Learning in UAV/UGV-Based RGB Imagery for Surface Landmine Detection: Faster R-CNN and YOLO established practical two-stage and one-stage detection frameworks for RGB surface-landmine imagery.Aerial applications commonly involve small objects, nonuniform scale, oblique viewpoints, and cluttered backgrounds.
  • B. Deep Learning in UAV/UGV-Based RGB Imagery for Surface Landmine Detection: Deep-learning studies applied Faster R-CNN, YOLOv5 fusion, YOLOF, DETR, Sparse R-CNN, VarifocalNet, and uncertainty-aware ResNet-50 methods to mine or UXO imagery.Reported applications included PFM-1 detection, multiple mine types, drone-based surface-mine detection, and unreliable-prediction flagging.
  • C. UAV/UGV-Based RGB Datasets for Surface Landmine Detection: Public UAV/UGV-based RGB datasets for surface landmine and UXO detection remain limited relative to general remote-sensing and object-detection benchmarks.Safety requirements, restricted sites, target availability, and operational-data release limitations constrain realistic collection and sharing.
  • C. UAV/UGV-Based RGB Datasets for Surface Landmine Detection: Existing RGB datasets vary substantially in target type, platform, modality, sample size, annotation availability, and IID/OOD support.These differences produce heterogeneous scopes and evaluation settings across mine and UXO detection studies.
  • C. UAV/UGV-Based RGB Datasets for Surface Landmine Detection: MineInsight broadens mine categories and sensing modalities, but domain gaps persist across seasons, terrains, and weather conditions.Reported environments include sunny, snowy, arid, and grassy settings; simulated data likewise does not capture full real-world variability.

D. Domain Shift and Benchmarking Gaps in RGB-Based Surface Mine Detection … B. SULAND v1 Dataset Organization and Need for Refinement

RGB surface-mine detection lacks broad architecture comparisons and standardized domain-shift evaluation, while SULAND v1 provides a rare public two-class benchmark with an explicit OOD setting. An audit of its released files found organizational and annotation inconsistencies that motivate SULAND v2 refinement for reliable IID/OOD benchmarking.

  • D. Domain Shift and Benchmarking Gaps in RGB-Based Surface Mine Detection: Existing RGB mine and UXO evaluations remain limited across detector architectures and datasets, despite the availability of two-stage, one-stage, transformer-based, and open-vocabulary models.The surveyed datasets differ in target type, platform, altitude, background, annotation format, and evaluation protocol.
  • D. Domain Shift and Benchmarking Gaps in RGB-Based Surface Mine Detection: Domain shift arises from changes in soil, vegetation, illumination, shadows, scale, viewpoint, motion, terrain, occlusion, season, and geography, challenging target-background separation.These factors can encourage reliance on scene-specific correlations rather than target-relevant cues.
  • D. Domain Shift and Benchmarking Gaps in RGB-Based Surface Mine Detection: Explicit IID/OOD splits and standardized domain-generalization protocols are often absent, hindering fair algorithm comparisons and robustness assessment in unseen environments.The present work therefore refines SULAND v1 into SULAND v2 and evaluates modern detectors under standardized IID and OOD settings.
  • A. Rationale for SULAND v1 Selection for OOD Analysis: The SULAND v1 OOD subset shifts geography from Italy to the United States and adds variation in context, vegetation, slope, viewpoint, distance, occlusion, size, color, and appearance.This structure supports studying RGB surface-landmine detection under shifted deployment conditions.
  • A. Rationale for SULAND v1 Selection for OOD Analysis: SULAND v1’s original OOD analysis qualitatively reported degraded YOLOv8 performance but did not provide quantitative OOD metrics or detailed experimental settings.The study independently assessed the split using detector-independent ImageNet-pretrained embeddings and t-SNE projections of 3000 randomly selected samples from each subset.
  • B. SULAND v1 Dataset Organization and Need for Refinement: SULAND v1 remains valuable as one of few public RGB surface-landmine datasets with an explicit OOD evaluation setting, while the audit tests annotation consistency for reliable benchmarking.The audit is framed as an assessment of benchmark suitability rather than a dismissal of the original dataset’s contribution.
  • B. SULAND v1 Dataset Organization and Need for Refinement: Annotation inconsistencies in SULAND v1—including missing or incorrect labels, mislocalized boxes, and inconsistent criteria—can create conflicting supervision and undermine localization-sensitive metrics.Because the dataset derives from video sequences, adjacent frames with similar targets may receive different annotation decisions, motivating SULAND v2 construction.

C. Audit Protocol and Annotation Issues in SULAND v1 Dataset

A systematic, two-pass manual audit evaluated SULAND v1 images and YOLO annotations frame-by-frame and organized defects into seven major categories. The audit identified annotation omissions, labeling inconsistencies, localization and artifact problems, quality degradation, and reversed IID/OOD class-ID mappings, motivating whole-dataset re-annotation.

  • Audit Protocol: The audit used a custom Python visualization script to overlay released YOLO boxes on images for folder-by-folder, frame-by-frame inspection.This enabled simultaneous evaluation of visible image content and corresponding annotations.
  • Audit Protocol: The two-pass process first recorded correction status and issue metadata, then systematically mapped defective samples to seven major annotation-error categories.Recorded metadata included target class, error type, approximate frame or sample range, visibility, and whether errors were isolated or temporal.
  • Annotation Issues: The audit found missing or incomplete annotations, false-positive propagation, mislocalized boxes, inconsistent partial-visibility criteria, non-representative artifacts, and quality degradation from blur.Examples included absent label files, boxes persisting after targets exited, markers or tags labeled as mines, and obscured target boundaries.
  • Annotation Issues: The IID subset encoded PFM-1 and PMA-2 as class IDs 0 and 1, respectively, while the OOD subset used the reversed mapping.This class-ID mismatch affects benchmark performance and is examined in Section IV-D.
  • Annotation Issues: Because many samples required correction, the authors manually re-annotated the whole dataset rather than applying only isolated local edits.The audit therefore supported a comprehensive dataset-level correction strategy.

IV. SULAND V2: A REFINED VERSION FOR BENCHMARKING · A. Re-Annotation Principles and Correction Procedure

SULAND v2 preserves SULAND v1’s images, task, and IID/OOD splits while manually correcting annotation inconsistencies for reliable benchmarking. Re-annotation applied explicit visibility, artifact, localization, class-consistency, and quality-control procedures.

  • A. Re-Annotation Principles and Correction Procedure: SULAND v2 preserves the original images, two-class task, and IID/OOD splits while standardizing Class 0 as PFM-1 and Class 1 as PMA-2.The dataset content was not altered; only annotation consistency was improved.
  • A. Re-Annotation Principles and Correction Procedure: Each sample was manually re-annotated in Label Studio, preserving original images and filenames while adding, removing, redrawing, or class-correcting annotations.Corrections were performed through sample-by-sample manual review rather than automated correction.
  • A. Re-Annotation Principles and Correction Procedure: Visible, class-identifiable targets were eligible for annotation, with adjacent video frames consulted to resolve presence, identity, and temporal continuity.Temporal context was used to resolve identity and visibility, not to annotate fully occluded or non-discernible targets.
  • A. Re-Annotation Principles and Correction Procedure: Partially visible targets were annotated consistently, whereas fully occluded, severely blurred, or non-discernible targets remained unannotated without sufficient evidence.This criterion covered occlusion by vegetation, terrain, debris, shadows, or image boundaries.
  • A. Re-Annotation Principles and Correction Procedure: Samples with stickers, flags, or markers were retained when the mine body remained discernible, with boxes restricted to the visible mine region.Annotations were removed only when the labeled region primarily covered an artifact and lacked sufficient target evidence.
  • A. Re-Annotation Principles and Correction Procedure: Bounding boxes were drawn tightly around visible target extents, including only visible regions for partially occluded targets to support IoU-based localization evaluation.Oversized, undersized, or displaced boxes can respectively include background, omit discriminative features, or introduce localization inconsistency.
  • A. Re-Annotation Principles and Correction Procedure: Quality control checked temporal, spatial, and class consistency by comparing every revised annotation with its SULAND v1 counterpart and categorizing changes as added, removed, tightened, or class-corrected.The four scenarios were defined at the bounding-box level.
  • A. Re-Annotation Principles and Correction Procedure: Three reviewers independently inspected correction crops arranged in contact sheets of 50 crops, accepting each correction by majority vote before final YOLO export.The contact sheets preserved enough local context while making exhaustive verification tractable.

B. Quantitative Comparison of SULAND v1 and SULAND v2 and Representative Annotation Corrections · C. Impact of Annotation Refinement on Detector Performance

SULAND v2 preserves SULAND v1’s images and splits while correcting annotation completeness, localization, validity, and class consistency. These refinements substantially change detector results, especially on OOD evaluation, while preserving an IID–OOD performance gap.

  • B. Quantitative Comparison of SULAND v1 and SULAND v2 and Representative Annotation Corrections: 14.7% net increase: annotated instances rise from 10,843 in SULAND v1 to 12,433 in SULAND v2, despite identical image files and splits.The change results entirely from annotation refinement, not added images or altered splits.
  • B. Quantitative Comparison of SULAND v1 and SULAND v2 and Representative Annotation Corrections: IID foreground images increase from 5,234 to 6,272 in training, while background-only images decrease from 17,522 to 16,484; OOD changes are modest.The IID changes indicate that many samples previously treated as background-only contained visible targets.
  • B. Quantitative Comparison of SULAND v1 and SULAND v2 and Representative Annotation Corrections: 40.4–44.9% of matched IID boxes fall below IoU 0.7, median IoU is approximately 0.72–0.73, and 68.0–77.9% of revised boxes become smaller.Median area ratios range from 0.85 to 0.91, indicating that tightening oversized or loosely localized boxes is the predominant geometric correction.
  • B. Quantitative Comparison of SULAND v1 and SULAND v2 and Representative Annotation Corrections: 3,162 OOD annotations require class correction, while OOD geometry is more consistent: median IoU 0.82, median area ratio 0.97, and 19.4% below IoU 0.7.The high file-modification rate primarily reflects systematic class-ID mismatch rather than extensive geometric revision.
  • B. Quantitative Comparison of SULAND v1 and SULAND v2 and Representative Annotation Corrections: 62.1% precision, 51.1% recall, and 56.1% F1 result when SULAND v1 annotations are evaluated as predictions against SULAND v2 at IoU ≥0.5.This class-aware comparison across combined IID splits shows that differences extend beyond isolated corrections.
  • B. Quantitative Comparison of SULAND v1 and SULAND v2 and Representative Annotation Corrections: Representative examples show corrections to annotation completeness, bounding-box localization, and class consistency for PFM-1 (Butterfly) and PMA-2 (Starfish).The examples distinguish original SULAND v1 boxes from refined SULAND v2 boxes and include added, removed, tightened, and class-corrected annotations.
  • C. Impact of Annotation Refinement on Detector Performance: 14.6–19.6 percentage-point gains on IID test mAP@50 and 10.3–16.4 on IID validation occur consistently across YOLOv8 model scales.The comparison evaluates YOLOv8 Nano, Small, Medium, Large, and XLarge separately on SULAND v1 and SULAND v2.
  • C. Impact of Annotation Refinement on Detector Performance: 17.8 to 43.6 percentage points: OOD improvements reach 48.6% mAP@50 for YOLOv8-M, yet a substantial IID–OOD gap remains after refinement.Refined annotations improve benchmark reliability without eliminating the underlying domain shift or making OOD scores solely a measure of generalization.

D. Cross-Version Training and Evaluation of SULAND v1 and SULAND v2 · V. BENCHMARKING OBJECT DETECTORS UNDER IID AND OOD CONDITIONS · A. Detector Families and Model Configurations

Cross-version evaluation shows that both refined training annotations and consistent evaluation ground truth materially affect reported performance, especially under IID conditions and corrected OOD scoring. The benchmark therefore compares 35 configurations from nine detector families on SULAND v2 under a common IID/OOD protocol.

  • D. Cross-Version Training and Evaluation of SULAND v1 and SULAND v2: YOLOv8 models were independently trained on SULAND v1 and SULAND v2, then evaluated against both annotation versions without retraining.This separates the effects of training annotations from evaluation ground truth under identical experimental settings.
  • D. Cross-Version Training and Evaluation of SULAND v1 and SULAND v2: The four combinations—v1→v1, v1→v2, v2→v1, and v2→v2—separate training-annotation effects from evaluation-annotation effects.Fixed evaluation versions isolate training effects, while fixed trained models isolate evaluation-ground-truth effects.
  • D. Cross-Version Training and Evaluation of SULAND v1 and SULAND v2: 51.6% to 81.4% and 60.4% to 87.5%: refined training annotations raised mean IID mAP@50 on SULAND v2 validation and testing, respectively.On OOD evaluation, the corresponding increase was smaller, from 38.4% to 42.9%.
  • D. Cross-Version Training and Evaluation of SULAND v1 and SULAND v2: 81.4% and 87.5% versus 42.1% and 43.0%: identical SULAND v2-trained detections scored much lower against SULAND v1 than SULAND v2 annotations.The discrepancy reflects differences in annotation completeness, localization, and consistency rather than poorer detector capability.
  • D. Cross-Version Training and Evaluation of SULAND v1 and SULAND v2: 10.1% and 10.5% increased to 35.0% and 38.2% after correcting SULAND v1’s inverted OOD class-ID mapping for the same predictions.The corrected values correspond to v1→v1* and v2→v1*, respectively.
  • D. Cross-Version Training and Evaluation of SULAND v1 and SULAND v2: Cross-evaluation supports SULAND v2 as the common annotation standard because training-label quality, evaluation-ground-truth consistency, and corrected class mapping materially alter benchmark results.The consistently stronger v2→v2 results motivate using SULAND v2 for subsequent detector comparisons.
  • A. Detector Families and Model Configurations: 35 model configurations from nine detector families broaden evaluation beyond YOLOv8 Nano and Small, covering one-stage, two-stage, transformer-based, and open-vocabulary paradigms.The comparison targets RGB surface-landmine detection, particularly under OOD conditions.
  • A. Detector Families and Model Configurations: The benchmark includes YOLOv8, YOLO11, YOLOv12, YOLO26, YOLO-Worldv2, Faster R-CNN, RT-DETR, D-FINE, and RF-DETR across varied scales and backbones.All configurations use SULAND v2 and a common protocol for comparing IID/OOD performance, class-wise behavior, model size, and inference speed.

B. Training and Evaluation Protocol … A. Overall Detector Performance

The benchmark uses a common fine-tuning and evaluation protocol with standardized detection, efficiency, and domain-shift metrics. Across configurations, YOLOv12-S leads IID mAP@50, while RF-DETR-L leads OOD performance, demonstrating changed rankings under domain shift.

  • B. Training and Evaluation Protocol: 35 configurations across nine detector families are fine-tuned from public pretrained weights on the IID training split and evaluated on IID test and OOD validation splits.Checkpoint selection uses IID validation mAP@50:95, and the original two-class task retains PFM-1 and PMA-2 targets.
  • B. Training and Evaluation Protocol: 100 epochs are used for most models and 50 for Faster R-CNN, with each detector retaining its best-epoch checkpoint after validation convergence.Every configuration reaches peak validation mAP@50:95 before training ends and then plateaus.
  • B. Training and Evaluation Protocol: Ultralytics detectors use 640×640 inputs with MuSGD at 0.01, Faster R-CNN uses SGD at 0.005, and D-FINE and RF-DETR use AdamW.D-FINE learning rates range from 8×10−4 for Nano to 2.5×10−4 for Large and XLarge, while RF-DETR uses 1 × 10−4 and 560×560 inputs.
  • C. Evaluation Metrics: mAP@50 measures precision at IoU 0.50, whereas mAP@50:95 averages AP from 0.50 to 0.95 in 0.05 increments for stricter localization assessment.Precision and recall are also reported overall and by class, with recall computed at the IID-validation F1-maximizing confidence threshold.
  • C. Evaluation Metrics: Trainable parameters and FPS quantify efficiency, while accuracy metrics are reported separately for IID and OOD subsets to assess domain-shift effects.The evaluation covers detection accuracy, localization quality, and target recovery alongside computational cost.
  • C. Evaluation Metrics: All detector predictions are converted to a common format and scored with one Ultralytics-based evaluator and shared matching rules for direct cross-family comparison.This procedure replaces native evaluators for Faster R-CNN, D-FINE, and RF-DETR.
  • A. Overall Detector Performance: Most detector families achieve strong IID performance, with YOLOv12-S attaining the highest mAP@50 of 0.908 and YOLO26-X the highest mAP@50:95 of 0.708.The benchmark compares detector families and configurations before analyzing generalization, speed-accuracy tradeoffs, and class-wise results.
  • A. Overall Detector Performance: RF-DETR-L leads OOD mAP@50, mAP@50:95, and recall, while most YOLO variants remain below approximately 0.53 OOD mAP@50 despite strong IID accuracy.Some configurations combine high OOD precision with comparatively lower target recovery, so detector comparison should jointly consider mAP, precision, and recall.

B. IID–OOD Generalization Gap … E. Qualitative Detection Results and Error Analysis

Across IID–OOD evaluation, domain shift reduces every detector’s performance, with robustness, speed, and class-wise behavior varying substantially by architecture. Qualitative examples show that these differences appear as missed detections, false positives, localization errors, and class-assignment errors in challenging OOD scenes.

  • B. IID–OOD Generalization Gap: Every evaluated detector performs below the IID–OOD equality line, although the magnitude of performance reduction varies considerably across detector families.The comparison covers both mAP@50 and mAP@50:95, with the latter also reflecting stricter localization requirements.
  • B. IID–OOD Generalization Gap: RF-DETR-L shows the strongest retention, decreasing from 0.880 to 0.799 mAP@50 while mAP@50:95 changes from 0.631 to 0.623.RF-DETR-L and Faster R-CNN remain closer to the IID–OOD equality line than most YOLO-family and RT-DETR configurations.
  • C. Accuracy–Speed Tradeoff: IID Pareto efficiency is dominated primarily by YOLO configurations, which combine high mAP@50:95 with inference rates of several hundred frames per second.This makes lightweight YOLO models attractive for high-throughput processing under familiar deployment conditions.
  • C. Accuracy–Speed Tradeoff: OOD tradeoffs favor RF-DETR-L and Faster R-CNN for accuracy, D-FINE-L as an intermediate compromise, and YOLO configurations for faster inference despite lower OOD accuracy.Model size is not a consistent predictor of detection accuracy or OOD robustness, so detector choice should reflect operating conditions.
  • D. Class-Wise Performance: Under IID conditions, most configurations achieve consistently high precision and recall for both PFM-1 (Butterfly) and PMA-2 (Starfish).Under OOD evaluation, precision is often better preserved than recall, with several YOLO configurations retaining comparatively high PMA-2 precision but much lower PMA-2 recall.
  • D. Class-Wise Performance: Faster R-CNN, D-FINE-L, and particularly RF-DETR-L retain higher recall than most YOLO and RT-DETR configurations, while PFM-1 and PMA-2 performance remains architecture-dependent.Aggregate mAP alone does not fully characterize OOD behavior because detectors with similar overall performance can differ in class-specific precision and target recovery.
  • E. Qualitative Detection Results and Error Analysis: Representative OOD examples show missed detections under clutter, vegetation, shadows, reduced contrast, viewpoint variation, or partial occlusion, alongside false positives, localization errors, and class-assignment errors.The qualitative comparison selected YOLO11-L, YOLO-Worldv2-X, Faster R-CNN-R50, and RF-DETR-L by detector category using the highest OOD mAP@50:95 configuration.
  • E. Qualitative Detection Results and Error Analysis: RF-DETR-L and Faster R-CNN-R50 recover more challenging targets in several illustrated OOD scenes, whereas YOLO11-L and YOLO-Worldv2-X show more misses or background sensitivity.These selected examples illustrate aggregate OOD differences but are not intended to establish that one detector outperforms another in every scene or quantify error frequency.

VII. DISCUSSION · A. Benchmark Reliability and Dataset Curation · B. Implications for Robustness Evaluation

Benchmark reliability and robustness evaluation are jointly necessary for RGB surface-mine detection. SULAND v2 strengthens evaluation by improving annotation consistency while emphasizing separate assessment across IID and changing deployment conditions.

  • VII. DISCUSSION: Annotation inconsistencies affect both detector supervision and evaluation references, so IID testing alone may not characterize behavior under changed conditions.SULAND v2 provides a more consistent basis for examining detector performance across both settings.
  • A. Benchmark Reliability and Dataset Curation: Annotation quality is central to benchmark validity because missing targets, invalid labels, localization differences, and class conventions influence training and evaluation.
  • A. Benchmark Reliability and Dataset Curation: Training-label errors alter supervision, whereas evaluation-label errors can penalize valid predictions or reward incorrect ones.Measured performance may therefore reflect compatibility with an annotation convention rather than detection capability alone.
  • A. Benchmark Reliability and Dataset Curation: Cross-version performance changes may reflect annotation completeness, localization, or supervision-reference disagreement rather than differences in model capability alone.Training and evaluation on the same version can reproduce version-specific labeling patterns, while cross-version evaluation exposes disagreement.
  • A. Benchmark Reliability and Dataset Curation: Benchmark releases should document class definitions, annotation criteria, correction procedures, dataset versions, and evaluation protocols.SULAND v2 retains the original imagery and split organization while applying unified annotation practices.
  • B. Implications for Robustness Evaluation: IID–OOD detector-ranking changes show that performance on a single test distribution is insufficient to characterize deployment robustness.IID accuracy reflects learned patterns from training data but does not establish preservation across changed deployment conditions.
  • B. Implications for Robustness Evaluation: Model scale is not a consistent solution to domain shift because larger configurations do not uniformly retain more accuracy than smaller same-family variants.Differences among detector families may involve feature representation, multiscale processing, proposal generation, or query-based detection, but these contributions were not isolated.
  • B. Implications for Robustness Evaluation: IID and OOD results should be reported separately, with aggregate mAP considered alongside localization-sensitive and class-specific precision and recall metrics.Future protocols should evaluate multiple OOD domains spanning geographic, seasonal, environmental, and acquisition changes.

C. Operational Relevance for Mine Action · D. Limitations and Future Directions

Operational deployment should treat RGB detection as decision support rather than autonomous clearance, balancing error consequences, speed, and robustness. The benchmark’s scope is limited by dataset composition, manual annotation uncertainty, entangled domain shifts, protocol-dependent efficiency, and the need for broader validation and methodological extensions.

  • C. Operational Relevance for Mine Action: Reduced OOD recall can leave visible hazards unflagged, while false alarms increase follow-up inspection effort.Operational relevance therefore depends on error consequences, not average accuracy alone.
  • C. Operational Relevance for Mine Action: Fast detectors may support preliminary screening, whereas more computationally demanding models may better preserve performance in new environments.A tiered workflow could combine rapid candidate generation with a detector selected for stronger OOD robustness.
  • C. Operational Relevance for Mine Action: Field use requires risk-based threshold selection, including recall-oriented operating points, confidence calibration, and false alarms per surveyed area.Suitable operating points may differ across survey stages, target types, and environmental conditions.
  • C. Operational Relevance for Mine Action: RGB detection can prioritize inspection and document visible surface conditions, but cannot autonomously declare an area safe.Fully buried, heavily occluded, or visually indistinguishable hazards require complementary sensing and established mine-action procedures.
  • D. Limitations and Future Directions: SULAND v2 covers two surface-laid target classes and primarily evaluates transfer between Italian and USA subsets, limiting representation of operational minefield diversity.Results may not transfer directly to other mine or UXO types, target conditions, seasons, camera systems, altitudes, or viewing geometries.
  • D. Limitations and Future Directions: Manual annotation refinement with multi-reviewer quality control may still leave omissions, localization inconsistencies, or class-assignment errors.SULAND v2 is a more consistent annotation release rather than an error-free reference.
  • D. Limitations and Future Directions: IID–OOD differences reflect entangled changes in terrain, vegetation, backgrounds, illumination, viewpoint, and target presentation, whose individual contributions remain unquantified.Detector results and feature-space analysis show visual differences and performance changes across conditions but do not isolate each factor.
  • D. Limitations and Future Directions: Standardized evaluation improves comparability but may not optimize every detector family, and inference speed should be interpreted comparatively rather than universally.Speed depends on hardware, software implementation, input resolution, and measurement procedure.

VIII. CONCLUSION … S2. AFFECTED FOLDERS BY ERROR CATEGORY

SULAND v2 establishes a standardized detector benchmark using manually refined annotations, showing that IID strength can diverge from robustness under geographic and environmental shift. The paper positions RGB detection as survey and decision support, with datasets, audit records, code, and supplementary analyses publicly available for continued evaluation.

  • VIII. CONCLUSION: SULAND v2 refines annotations and class-ID conventions while preserving the original imagery and split organization, enabling standardized IID and OOD benchmarking.The refinement addressed missing or invalid annotations, bounding-box localization, partial-visibility criteria, and class-ID conventions.
  • VIII. CONCLUSION: 35 configurations from nine detector families were evaluated under a common protocol, revealing strong IID performance but substantial OOD variation across architectures.YOLO-family models generally led IID accuracy and throughput, whereas RF-DETR-Large and Faster R-CNN retained more performance under geographic and environmental shift.
  • VIII. CONCLUSION: Reliable RGB surface-mine detection requires both high annotation quality and evaluation under distribution shift beyond the training environment.SULAND v2 provides a consistent basis for detector comparison and robustness analysis.
  • VIII. CONCLUSION: RGB detection is intended as survey-support and decision-support capability for visible surface targets, not as an independent clearance method.The conclusion frames operational use around visible targets and support functions.
  • VIII. CONCLUSION: Future work should expand geographic, target, seasonal, acquisition, and sensing coverage while studying domain generalization, adaptation, calibration, and uncertainty-aware detection.These extensions are proposed to broaden the benchmark and improve robustness analysis.
  • IX. DATASET AND CODE AVAILABILITY: The public repository provides SULAND v1 and v2 annotations, corrected OOD class IDs, sample-level audit records, and detector training, evaluation, and benchmarking code.The datasets are hosted on Hugging Face and the code on GitHub.
  • Supplementary Material: The supplementary material documents folder- and sample-level annotation issues, affected folders by error category, and training-convergence results for the evaluated detector configurations.It supports the dataset audit and assesses whether predefined training schedules were adequate.
  • S1. FOLDER- AND SAMPLE-LEVEL AUDIT RECORDS: Seven SULAND v1 error categories include incomplete annotations, false positives, localization errors, visibility inconsistencies, artifacts, image-quality degradation, and class-ID mismatch.Representative folder and sample summaries are provided, while all samples were manually reannotated for SULAND v2; localization and OOD class-ID issues affected larger dataset portions and were excluded from Table S1.

A. Missing or Incomplete Annotations … S3. TRAINING CONVERGENCE AND SCHEDULE ADEQUACY

SULAND v2 documents annotation problems across released folders, including missing, false, mislocalized, inconsistent, artifact-affected, degraded, and class-mismatched labels. Training trajectories show that the adopted schedules were adequate for unified detector comparison.

  • A. Missing or Incomplete Annotations: Missing or incomplete annotations affected ITA-v1, ITA-v4, ITA-v17, ITA-v21, ITA-v24, ITA-v25, ITA-v31, ITA-v32, and ITA-v34.
  • B. False Positive Annotations: False positive annotations affected ITA-v1, ITA-v14, and ITA-v37.
  • C. Mislocalized Bounding Boxes: All released IID and OOD folders were reviewed, with bounding boxes redrawn or tightened where required; ITA-v4-329 is a representative SULAND v1 case.
  • D. Inconsistent Partial-Visibility Criteria: Inconsistent partial-visibility criteria affected ITA-v16, ITA-v17, ITA-v18, ITA-v20, ITA-v22, ITA-v23, ITA-v24, ITA-v29, ITA-v31, and ITA-v35.
  • E. Non-Representative Artifacts: Non-representative artifacts affected ITA-v1, ITA-v4, ITA-v6, ITA-v9, ITA-v11, ITA-v12, ITA-v13, ITA-v14, and ITA-v32.
  • F. Image-Quality Degradation: Image-quality degradation affected ITA-v30, particularly samples ITA-v30-706--717.
  • G. Class-ID Mismatch: OOD folders US1--US10 used the reverse of the IID class convention, so class IDs were corrected across the OOD subset in SULAND v2.
  • S3. TRAINING CONVERGENCE AND SCHEDULE ADEQUACY: Validation performance reached a maximum or stable plateau before training ended, including for RT-DETR, D-FINE, and RF-DETR, indicating adequate schedules for unified comparison.Configurations used 100 epochs except larger Faster R-CNN variants, which used 50; the highest-validation-mAP@50:95 checkpoint was retained.
Loading 2607.28996v1…