Source-linked AI summary
Saliency-Depth Conditioning for Zero-Shot Segmentation of Communication-Tower Components in Cluttered UAV Imagery
Ali Lesani, Chul Min Yeum, Su-Min Kang
TL;DR
Communication-tower component segmentation in cluttered UAV imagery is hindered by limited annotations and visually similar background structures that confuse zero-shot localization. The paper introduces model-agnostic saliency–depth conditioning for Grounded-SAM and SAM 3, improving both baselines on TOW-300 while producing different precision–recall operating points.
Problem
Limited instance-level annotations make task-specific models difficult to develop, while cluttered UAV scenes cause zero-shot models to miss components or generate false positives.
Method
A training-free module combines appearance-based saliency and monocular relative depth to form a coarse tower prior, with geometric and depth-aware box refinement for the Grounded-SAM branch.
Results
The strategy consistently improves Grounded-SAM and SAM 3 on TOW-300; SD-SAM 3 has the strongest instance segmentation, while SD-Grounded-SAM provides higher precision and cleaner aggregate masks.
Takeaways & Limitations
Saliency–depth conditioning improves zero-shot segmentation across two architecturally distinct frameworks, with the integrations offering different operating points for instance performance and precision.
Takeaways & Limitations
Near top-down viewpoints can leave background content in the conditioned image, while conservative box refinement may miss valid components.
Abstract
from arXiv · showhide
Fine-grained segmentation of communication-tower components in UAV imagery is essential for automated inspection, yet task-specific models are hard to develop due to limited instance-level annotations. Zero-shot segmentation models offer a promising alternative, but in cluttered scenes, visually similar background structures interfere with component localization, causing missed instances and false positives. We propose a model-agnostic saliency-depth foreground-conditioning strategy combining appearance-based saliency with monocular relative depth to construct a coarse tower prior and suppress irrelevant content. We integrate this module with Grounded-SAM and SAM 3, yielding SD-Grounded-SAM and SD-SAM 3. SD-Grounded-SAM further applies geometric and depth-aware box refinement before mask generation, while SD-SAM 3 relies on SAM 3's internal setup. On TOW-300, a dataset of 340 communication-tower UAV images, our strategy improves both baselines: SD-SAM 3 achieves the strongest instance-segmentation performance, while SD-Grounded-SAM produces fewer false positives. Ablations confirm complementary gains from saliency, depth, and box refinement, improving robustness in cluttered scenes.
1 Introduction
Communication-tower component segmentation in UAV imagery is difficult because small, occluded targets resemble clutter, while limited annotations constrain supervised approaches. The paper addresses this with training-free saliency–depth conditioning integrated into Grounded-SAM and SAM 3.
- Challenge: Small and partially occluded tower components can resemble vegetation, buildings, cables, power lines, and metallic structures in crowded UAV scenes.Viewpoint, scale, elevation, and perspective changes further complicate localization.
- Challenge: Pixel-level annotation of small, overlapping components is costly, and fixed-dataset supervised models may generalize unreliably across tower and acquisition variations.
- Challenge: Zero-shot SAM-based segmentation still depends on accurate localization because SAM requires prompts, while open-vocabulary prompts can select whole towers or unrelated background objects.
- Approach: The proposed model-agnostic, training-free strategy combines appearance saliency with monocular relative depth to suppress irrelevant background and recover near-field tower regions.Saliency reduces the search space but may remove weak or occluded tower portions; depth is used to recover them.
- Approach: SD-Grounded-SAM refines Grounding DINO boxes using tower-relative geometry, containment, and relative depth, whereas SD-SAM 3 passes the conditioned image directly to SAM 3's internal pipeline.
- Results: On TOW-300's 340 high-resolution UAV images, conditioning consistently improves both zero-shot baselines; SD-SAM 3 leads instance segmentation, while SD-Grounded-SAM yields higher precision and cleaner aggregate masks.
2 Related Work
Related work spans UAV-based infrastructure inspection, supervised segmentation, and foundation models for zero-shot segmentation. The paper positions its approach as adapting general-purpose foundation models to infrastructure-specific structural characteristics.
- UAV-Based Visual Inspection: UAV-based visual inspection supports flexible and efficient image acquisition for infrastructure that is difficult, costly, or unsafe to access manually.
- Supervised Segmentation: Supervised segmentation remains dominant for dense visual understanding, with established semantic and instance-segmentation architectures used across domain-specific vision tasks.
- Foundation Models for Zero-Shot Segmentation: Foundation models such as CLIPSeg, LSeg, GroupViT, Grounded-SAM, and SAM 3 advance language-guided, open-vocabulary, and unified zero-shot segmentation.
- Foundation Models for Zero-Shot Segmentation: Despite broad generalization, foundation models are general-purpose systems rather than models tailored to infrastructure-inspection imagery.
3 Method
The method conditions UAV images with saliency and monocular depth before applying zero-shot segmentation. It supports two branches: refined-box prompting for Grounded-SAM and direct conditioned-image inference with SAM 3.
- Problem Setup: The task is to generate instance masks for individual tower-mounted components without task-specific model training.The framework uses general-purpose pretrained models with shared foreground conditioning.
- Saliency–Depth Foreground Conditioning: Saliency estimation and monocular relative depth are combined into a coarse tower prior and refined foreground mask.The fused mask is morphologically refined and filtered to suppress irrelevant background content.
- Saliency–Depth Foreground Conditioning: Morphological closing, opening, and largest-connected-component filtering produce a coherent foreground mask under the assumption that the tower is dominant.Closing connects nearby regions, opening removes isolated noise, and connected-component filtering removes remaining disconnected regions.
- Saliency–Depth Foreground Conditioning: The refined mask is applied element-wise to the RGB image, preserving dominant tower structure while suppressing surrounding background.The resulting conditioned image reduces visual ambiguity for downstream zero-shot segmentation.
- SD-Grounded-SAM: SD-Grounded-SAM refines Grounding DINO candidate boxes using tower geometry, pairwise containment, and depth before prompting SAM.Box generation uses the conditioned image, while mask prediction uses the original RGB image to preserve component detail.
- SD-SAM 3: SD-SAM 3 supplies the conditioned image and text prompt directly to SAM 3, leaving localization, instance identification, and mask generation internal to the model.No external box-generation or box-refinement stage is applied in this branch.
- Model-Agnostic Design: The conditioning module is model-agnostic because its saliency, depth, localization, and segmentation components can be replaced by compatible alternatives.The framework requires compatible intermediate outputs rather than the specific models used in the experiments.
4 Dataset
TOW-300 is a 340-image UAV dataset with instance-level annotations for communication-tower components, collected across varied inspection conditions. Its sparse structures, clutter, viewpoint variation, and visually similar background objects make zero-shot component localization and segmentation challenging.
- Dataset composition: TOW-300 contains 340 high-resolution images from three UAV flights covering two visually and structurally distinct communication towers.The images contain multiple mounted components, primarily antennas and radio units, and use either 5000×6300 or 9100×6300 pixel resolutions.
- Acquisition: Images were collected during real-world inspections using a DJI Mavic 3 Enterprise UAV across different times, illumination conditions, flight trajectories, viewpoints, and elevations.The acquisition included helical paths around the tower and vertical lawnmower patterns.
- Annotations: Each visible tower-mounted component is annotated as an individual instance with a manually drawn polygon mask and corresponding bounding box.The dataset uses one semantic class encompassing antennas and radio units while preserving separate instance annotations.
- Dataset challenges: Sparse tower lattices expose vegetation, buildings, rooftops, cables, and other background objects with shapes, textures, or colors resembling mounted components.These similarities increase ambiguity between foreground components and surrounding structures.
- Dataset challenges: Viewpoint, scale, and perspective variation causes components to appear small, overlap structural members, or become partially occluded.Elevated and near top-down observations can expose rooftop objects through the tower structure.
- Evaluation protocol: The validation/test split uses 40 images for inference-time hyperparameter selection and 300 images for final evaluation.No conventional training split is used because the proposed framework does not require model training.
5 Experiments
Saliency–depth conditioning consistently improves both Grounded-SAM and SAM 3, but their conditioned variants favor different operating points: SD-SAM 3 prioritizes instance discovery, while SD-Grounded-SAM emphasizes precision and cleaner aggregate foregrounds.
- Overall results: Both conditioned frameworks improve across instance- and image-level metrics, confirming benefits across architecturally distinct zero-shot segmentation systems.The downstream framework and box-refinement design determine the resulting precision–recall characteristics.
- Effect on Grounded-SAM: 0.8970 precision versus 0.4962 for Grounded-SAM demonstrates the largest reported Grounded-SAM gain from conditioning and box refinement.Mean matched IoU also increases from 0.8879 to 0.8989, indicating preserved and slightly improved mask quality for correctly localized instances.
- Effect on SAM 3: 0.5701 mAP and 0.7311 recall for SD-SAM 3 exceed SAM 3’s 0.4250 mAP and 0.6089 recall.Precision increases from 0.7662 to 0.7817, while mean matched IoU rises from 0.8787 to 0.9107.
- Comparison of conditioned frameworks: SD-SAM 3 achieves higher instance-level AP and recall, making it preferable when component discovery and coverage are prioritized.Its integrated localization mechanism is more effective at discovering small, partially occluded, or visually weak components after background interference is reduced.
- Comparison of conditioned frameworks: SD-Grounded-SAM achieves the highest precision and strongest image-level semantic segmentation, adopting a more conservative operating profile.Its explicit box-refinement stage removes uncertain, oversized, nested, and distant detections before mask generation.
- Image-level foreground quality: 118.4% semantic IoU improvement and 72.7% Dice improvement are reported for SD-Grounded-SAM relative to Grounded-SAM.Semantic IoU increases from 0.2876 to 0.6281 and Dice from 0.4467 to 0.7716, reducing irrelevant background area in merged masks.
6 Conclusion
The work introduces a model-agnostic saliency–depth conditioning strategy for training-free zero-shot segmentation of tower components, integrated into Grounded-SAM and SAM 3. On TOW-300, SD-SAM 3 provides the strongest instance segmentation, while SD-Grounded-SAM achieves the highest precision and semantic IoU.
- The proposed conditioning module combines appearance-based saliency with monocular relative depth to suppress irrelevant background and recover near-field tower structures.
- The strategy produces SD-Grounded-SAM and SD-SAM 3, with geometric and depth-aware box refinement additionally applied in the Grounded-SAM branch.
- Experiments on TOW-300 show consistent improvements for both frameworks across instance- and image-level segmentation metrics.
- SD-SAM 3 achieves the strongest overall instance segmentation performance, indicating improved component discovery through foreground conditioning.
- SD-Grounded-SAM achieves the highest precision and semantic IoU, reflecting the contribution of reliable spatial prompts from box refinement.