PaperlayerView exact revision on arXiv

Source-linked AI summary

DRA-SAM: Disagreement-Driven Referential Active SAM for Low-Altitude Drone Referring Segmentation

Jisheng Dang, Huaiyuan Peng, Wencan Zhang, Bimei Wang, Jingwen Zhao, Wenjun Ma, Hong Peng, Bin Hu, Qi Tian, Tat-Seng Chua

arXiv:2610.33057v1cs.CVcs.AI

TL;DR

Low-altitude drone referring segmentation must distinguish small, visually similar instances, while global predictions can leave competing masks or an insufficiently represented referent. DRA-SAM uses candidate beliefs and an unresolved state to choose a crop for local SAM3-I inference. It leads reported mIoU and referring-precision comparisons across two datasets, with matched-backbone gains on RIS-LAD.

  • Problem

    Low-altitude drone referring segmentation must distinguish small, visually similar instances under complex linguistic and spatial constraints, where multiple masks may plausibly match a query.

    Source: §I · p. 1

  • Method

    DRA-SAM uses an EvidenceScorer and training-free LOOK policy to select a crop from candidate-mask disagreement and unresolved-region evidence, then updates hypotheses with local SAM3-I predictions.

    Source: §I · p. 1

  • Results

    DRA-SAM leads reported test-set mIoU and referring-precision comparisons on both datasets; on RIS-LAD, matched-backbone mIoU rises from 49.55 to 52.34 and P@0.5 from 46.80 to 58.05.

    Source: §I · p. 2, §V · p. 9

  • Takeaways & Limitations

    Controlled policy comparisons and target-size analysis support disagreement-guided inference allocation, with particularly pronounced gains for small, crowded aerial referents.

    Source: §V · p. 9

Abstract

from arXiv · show

Referring segmentation in low-altitude drone imagery requires distinguishing small, visually similar instances under complex linguistic and spatial constraints. Global inference may yield multiple plausible masks or leave the intended referent insufficiently represented, motivating selective local re-inference. We propose DRA-SAM, a disagreement-driven framework that allocates an additional inference pass according to competing referential hypotheses and an unresolved state. A lightweight EvidenceScorer estimates beliefs over existing candidates and the unresolved state. A training-free Referential Disagreement LOOK policy balances belief-weighted candidate competition and unresolved-region coverage to select a crop for local re-inference. The same benchmark-adapted SAM3-I backbone processes the selected region, and the resulting evidence updates existing hypotheses and introduces new candidates. Among the compared methods, DRA-SAM achieves state-of-the-art performance on six of seven test metrics on both RIS-LAD and UAVid-RIS, including mIoU and P@0.5-P@0.9, and on all seven RIS-LAD validation metrics. On RIS-LAD, DRA-SAM improves the matched global SAM3-I baseline from 49.55 to 52.34 mIoU and from 46.80 to 58.05 P@0.5. On UAVid-RIS, the independently adapted DRA-SAM achieves 73.26 mIoU and 86.86 P@0.5. Controlled crop-policy comparisons demonstrate the effectiveness of disagreement-guided crop selection, while target-size analysis reveals particularly pronounced gains for aerial targets occupying less than 1% of an image. These results establish referential disagreement as an effective criterion for allocating local inference. Code is available at https://github.com/phy709/DRA-SAM.

I. INTRODUCTION

Low-altitude drone referring segmentation must distinguish small, visually similar instances under complex spatial and linguistic constraints. DRA-SAM allocates local re-inference using disagreement among candidate masks and an unresolved hypothesis, and reports leading results across two UAV benchmarks.

I. INTRODUCTION

  • Prior work addresses cross-modal interaction, instance-aware reasoning, word–pixel alignment, scale-aware representations, and uncertainty-guided refinement; this work instead studies where to allocate an additional inference pass.The cited work markers in the passage are,,,,,,,, and.Source: §I · p. 1
  • DRA-SAM treats ambiguity as competition among existing masks or discovery of a referent insufficiently represented in the initial proposals.Its unresolved hypothesis reserves belief for potentially missing targets, while candidate evidence identifies actionable disagreement.Source: §I · pp. 1–2
  • The framework uses a 382,920-parameter EvidenceScorer and a training-free Referential Disagreement LOOK policy to select crops from belief-weighted mask disagreement and unresolved-region coverage.The same benchmark-adapted SAM3-I checkpoint processes the crop, updating existing evidence and potentially adding candidates.Source: §I · pp. 1–2
  • DRA-SAM leads the reported test-set mIoU and referring-precision comparisons on both datasets; under a matched RIS-LAD backbone, it raises mIoU from 49.55 to 52.34 and P@0.5 from 46.80 to 58.05.The introduction also reports the highest values on all seven RIS-LAD validation metrics.Source: §I · pp. 1–2
  • For RIS-LAD targets covering less than 1% of an image, the reported P@0.5 gains are 11.70–15.70 points.The introduction also reports controlled crop-policy comparisons of disagreement-driven selection.Source: §I · p. 2

II. RELATED WORK

Related work advances referring segmentation through language–vision representations, aerial and remote-sensing methods, uncertainty modeling, and promptable segmentation. DRA-SAM complements these directions by selecting where to apply an additional prediction after global candidates are available.

A. Cross-Modal and Instance-Aware Referring Segmentation

  • Cross-modal interactions, instance-aware reasoning, alignment, and compositional or instruction-guided settings broaden referring segmentation; DRA-SAM instead allocates an additional prediction based on disagreement among concrete mask hypotheses.Examples include CrossVLT, instance-specific propagation, and multi-level language–mask alignment.Source: A. Cross-Modal and Instance-Aware… · p. 2

B. Remote-Sensing and Drone Referring Segmentation

  • Aerial referring work addresses small-object scale, crowded layouts, and diverse viewpoints, while DRA-SAM selects an additional inference location after the global candidate set is available.RIS-LAD targets low-altitude referring scenes, and UAVid-RIS provides a complementary aerial benchmark.Source: B. Remote-Sensing and Drone Referring… · p. 2

C. Uncertainty, Prediction Correction, and Adaptive Computation

  • Unlike uncertainty methods that guide fusion or local refinement, DRA-SAM measures disagreement between complete candidate masks, selects an explicit crop, and integrates local proposals.Its analytical action utility combines normalized hypothesis beliefs and spatial overlap; belief temperature is selected on internal validation.Source: C. Uncertainty, Prediction Correction,… · p. 2

D. Promptable and Foundation Segmentation

  • DRA-SAM adapts SAM3-I separately to each benchmark and freezes it for global and local inference, allowing the added stage to be studied without changing the segmentation backbone.The passage situates this approach alongside promptable and foundation segmentation models including SAM 3 [40].Source: D. Promptable and Foundation… · pp. 2–3

A. Problem Setup and Framework Overview

DRA-SAM represents global predictions as concrete mask hypotheses plus an unresolved possibility, then uses candidate and language-conditioned evidence to guide a crop decision. The selected local inference uses the same adapted SAM3-I backbone and updates the hypothesis set.

A. Problem Setup and Framework Overview

  • The framework represents hypotheses as specific predicted masks rather than object classes, with an unresolved hypothesis for referents inadequately covered by proposals.The 2,304 crop actions vary normalized center and square scale; local SAM3-I inference returns masks for evidence integration.Source: A. Problem Setup and Framework Overview · p. 3

B. Candidate Evidence and Initial Belief

  • EvidenceScorer assigns evidence to observed masks and a potentially missing referent using mask-pooled appearance and a pooled language representation.It has 382,920 trainable parameters; the union fallback uses mean proposal descriptors or global average pooling when proposals are absent.Source: B. Candidate Evidence and Initial Belief · p. 3
  • The candidate-belief vector represents relative support, while hU is a normalized language-compatible crop-selection prior rather than a foreground or final-mask prediction.The unresolved belief bU scales the prior's influence on crop choice, linking discovery to the current hypothesis state.Source: B. Candidate Evidence and Initial Belief · p. 3
  • LOOK uses visual–text affinity to shape candidate beliefs and the spatial prior, allowing different expressions to favor mask disagreement or unresolved regions.The policy guides the location and scale of one additional visual observation.Source: B. Candidate Evidence and Initial Belief · p. 3

C. Disagreement-Driven LOOK Selection

LOOK selects one crop by combining disagreement among plausible existing masks with coverage of the unresolved referent. Its joint location-and-scale choice balances magnified candidate inspection against broader contextual coverage.

C. Disagreement-Driven LOOK Selection

  • Competition rewards crops exposing consequential disagreement between plausible, spatially distinct candidates, allowing either candidate to be visible.This avoids requiring both candidates to fit in one crop, which can favor larger regions and reduce magnification.Source: C. Disagreement-Driven LOOK Selection · p. 4
  • LOOK combines belief-weighted disagreement among existing masks with unresolved-region coverage to score crop actions.Pairwise disagreement reflects mask differences, while the unresolved spatial prior directs crops toward language-compatible regions.Source: C. Disagreement-Driven LOOK Selection · pp. 3–4
  • The 2,304-action grid jointly selects crop location and scale, trading magnification of compact candidates against coverage of surrounding context.The four scales provide nominal magnifications of 3.33, 2.22, 1.67, and 1.33 when the referent lies fully within the crop.Source: C. Disagreement-Driven LOOK Selection · pp. 4–5
  • LOOK evaluates predicted masks, beliefs, and geometry without a crop-policy network, then forwards only the selected crop through SAM3-I.Action scoring is separated from the additional segmentation pass.Source: C. Disagreement-Driven LOOK Selection · p. 4

D. Local Re-Inference and Evidence Integration

The selected crop provides evidence for both existing hypotheses and new candidates. The update adjusts retained candidates, admits local masks, and reduces unresolved belief according to explored spatial mass.

D. Local Re-Inference and Evidence Integration

  • Local candidates are matched softly and many-to-many to existing hypotheses, without an IoU threshold or one-to-one assignment constraint.The resulting evidence updates existing hypotheses and supports new candidates using the local crop predictions.Source: D. Local Re-Inference and Evidence… · p. 5
  • Each local mask becomes a new hypothesis without a proposal gate, duplicate merging, or extra NMS, while retained logits receive matched-evidence updates and visibility-proportional miss penalties.The miss parameter ρmiss starts at 2.0.Source: D. Local Re-Inference and Evidence… · p. 5
  • The final belief distribution normalizes retained, new, and unresolved logits, with unresolved belief decreasing according to explored spatial mass.On RIS-LAD test, one crop adds 0–8 hypotheses (median 2), yielding 1–17 concrete candidates (median 4); no hard candidate cap is used.Source: D. Local Re-Inference and Evidence… · p. 5

E. EvidenceScorer Supervision and Training

EvidenceScorer is trained on cached features from a fixed benchmark-adapted SAM3-I checkpoint, with soft-target supervision for initial and updated beliefs. Ground truth is used for offline crop sampling and supervision, while test-time crop selection uses predicted evidence without ground truth.

E. EvidenceScorer Supervision and Training

  • Soft-target beliefs supervise both prior and post-update stages, using candidate IoU quality and unresolved quality defined as one minus the best candidate IoU.The training target temperature is τtrain = 0.10.Source: E. EvidenceScorer Supervision and… · p. 5
  • EvidenceScorer training uses cached features from a fixed benchmark-adapted SAM3-I checkpoint, with 30 epochs and final-epoch weights used for evaluation.The RIS-LAD cache contains 8,755 training and 957 internal-validation records; the official 1,396-sample validation split is reserved for reporting.Source: E. EvidenceScorer Supervision and… · p. 5
  • Offline local observations use up to 22 deduplicated crops per annotation, while test-time LOOK evaluates all 2,304 actions without ground-truth input.Training-time LOOK uses normalized box-IoU disagreement because the feature cache omits full candidate masks; inference uses predicted-mask IoU, and crop selection remains analytical.Source: E. EvidenceScorer Supervision and… · p. 5

F. Final Referent Prediction

Final prediction selects the highest-belief concrete mask after the local update, with a deterministic union fallback when unresolved belief exceeds every concrete belief.

F. Final Referent Prediction

  • After one local update, prediction selects the highest-belief concrete mask unless unresolved belief strictly exceeds every concrete belief, in which case it returns the deterministic union fallback.Confidence and risk gates are disabled.Source: F. Final Referent Prediction · p. 5

IV. EXPERIMENTS

Experiments use separate aerial referring benchmarks and distinguish scorer evaluation from benchmark-specific backbone adaptation. A fixed adapted SAM3-I checkpoint supports global inference, scorer training, and the additional local pass.

A. Datasets, Metrics, and Implementation Details

  • RIS-LAD provides 9,712 training annotations, 1,396 validation samples, and 2,763 test samples; UAVid-RIS provides 7,035 samples across train, validation, and test.RIS-LAD is used for detailed ablations, while UAVid-RIS supports independent full-framework evaluation.Source: A. Datasets, Metrics, and… · p. 6
  • Evaluation reports mIoU, oIoU, and P@θ, the percentage of samples whose prediction IoU reaches threshold θ.mIoU weights each referred instance equally, whereas oIoU weights by union area; a ground-truth-selected candidate oracle is evaluation-only.Source: A. Datasets, Metrics, and… · p. 6

1) Benchmark-Specific Backbone Adaptation and Fixed-Inference Protocol:

  • SAM3-I is adapted separately on each benchmark, then frozen for global inference, scorer training, and the local pass.For RIS-LAD, Stage3 referring adaptation optimizes adapter and query/concept-projection modules while visual and language backbones remain frozen.Source: 1) Benchmark-Specific Backbone… · p. 6
  • UAVid-RIS uses independently adapted SAM3-I and EvidenceScorer checkpoints, with neither component transferred directly from RIS-LAD.Both global and local passes share the resulting UAVid-RIS weights.Source: 1) Benchmark-Specific Backbone… · p. 6

B. Main Comparison on RIS-LAD

DRA-SAM leads most reported RIS-LAD and UAVid-RIS test comparisons, while matched-checkpoint ablations distinguish candidate re-ranking from the added local pass.

B. Main Comparison on RIS-LAD

  • DRA-SAM ranks first on all seven RIS-LAD validation metrics and six of seven test metrics; test mIoU is 52.34 versus DeCo-L’s 50.20.Its P@0.9 is 34.02 versus the strongest listed competing value of 21.46; test oIoU is 53.45, below DeCo-L’s 54.80.Source: B. Main Comparison on RIS-LAD · p. 6, Table I · p. 7

C. Main Comparison on UAVid-RIS

  • On UAVid-RIS, DRA-SAM ranks first in test mIoU and all five precision thresholds; mIoU is 73.26 and P@0.5 is 86.86.The checkpoints are independently adapted for this benchmark; AeroReformer reports 72.79 mIoU and 86.34 P@0.5.Source: C. Main Comparison on UAVid-RIS · p. 6, Table II · p. 7

D. Contributions of Candidate Re-ranking and Local Re-Inference

  • On RIS-LAD test, scorer-only re-ranking raises mIoU/oIoU/P@0.5 from 49.55/49.08/46.80 to 51.06/51.65/55.56; the full method reaches 52.34/53.45/58.05.The comparison holds the adapted SAM3-I checkpoint fixed and adds one disagreement-selected local prediction after re-ranking.Source: D. Contributions of Candidate… · pp. 6–7, Table III · p. 7
  • The RIS-LAD hyperparameters λ = 1.0 and τ = 1.2 are selected on the internal validation split and fixed before official testing.The corresponding sweeps report 52.4340 mIoU and 57.1633 P@0.5 for λ, and 52.5158 mIoU, 54.0356 oIoU, and 57.1633 P@0.5 for τ.Source: D. Contributions of Candidate… · p. 7

E. Controlled Comparison of LOOK Selection Policies

Controlled comparisons test crop selection under a shared local-inference protocol, then examine qualitative cases, utility components, and candidate-set changes after one LOOK pass.

E. Controlled Comparison of LOOK Selection Policies

  • Referential Disagreement leads six RIS-LAD test crop policies in mIoU, oIoU, and P@0.5, exceeding Generic Ambiguity by 0.33, 0.91, and 0.83 points.It exceeds Random by 1.25, 1.96, and 1.84 points; all policies use one local segmentation pass and evidence update, isolating crop selection.Source: Figure 3 · p. 7, E. Controlled Comparison of LOOK… · pp. 7–8

F. Qualitative Analysis of Active LOOK

  • Qualitative rows show LOOK crops distinguishing similar parked vehicles, preserving a relational cue in a crowded basketball scene, and recovering a roadway vehicle proposal.The cases illustrate discrimination among existing proposals and candidate-set expansion through local inference.Source: Figure 4 · p. 8, F. Qualitative Analysis of Active LOOK · p. 8

G. Contributions of Competition and Discovery Utilities

  • The combined competition-and-discovery utility reaches 52.34 mIoU and 58.05 P@0.5, exceeding competition-only by 0.39 and 0.97 points, respectively.Discovery-only has the highest oIoU at 53.78; the combined utility exceeds discovery-only by 0.16 mIoU and 0.18 P@0.5 points.Source: Table IV · p. 8, G. Contributions of Competition and… · p. 8

H. Oracle Analysis of Candidate Set Quality

I. Target-Size Analysis of Local Re-inference

DRA-SAM’s gains are especially pronounced for RIS-LAD targets below 1% of image area, while its metric trends align with instance-level localization of small referents. The conclusion reports leading comparisons across both datasets and stronger matched-backbone RIS-LAD results.

I. Target-Size Analysis of Local Re-inference

  • For RIS-LAD targets below 1% of image area, DRA-SAM gains 15.70, 13.19, and 11.70 points in P@0.5 across the three smallest size groups.It improves mIoU in all five size groups, including +4.59 points below 0.1% and +4.92 points in the 0.3–1% group; gains are more concentrated on compact targets, consistent with greater effective target resolution after crop reprocessing.Source: §I · p. 9
  • Because oIoU weights per-sample IoU by union area, compact-object gains typically count less than in mIoU, whereas mIoU and P@θ count each referred instance equally.The reported metric trends and size-stratified gains are consistent with improved instance-level localization of small referents.Source: §I · p. 9

V. CONCLUSION

  • DRA-SAM leads reported comparisons in mIoU and all five referring-precision thresholds across both datasets, and improves matched RIS-LAD mIoU from 49.55 to 52.34 and P@0.5 from 46.80 to 58.05.The conclusion attributes particular value to targeted inference allocation for small, crowded aerial referents.Source: §V · p. 9

Related papers

Loading 2610.33057v1…