PaperlayerView exact revision on arXiv

Source-linked AI summary

Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement

Hyun-Kurl Jang, Jihun Kim, Kuk-Jin Yoon

arXiv:2609.34528v1cs.CV

TL;DR

Specialized-domain OVSS adaptation relies on dense masks that can be costly and require expert annotation. The paper uses prompt-induced segmentation differences to elicit localized binary preferences and adapt models with RLPO and consistency regularization. It reports consistent MESS gains without dense masks and effectiveness under noisy preferences.

  • Problem

    Specialized-domain OVSS adaptation often relies on dense pixel masks that are costly to obtain and require domain-specific expertise.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    The framework mines localized binary preferences from prompt-induced disagreement and adapts OVSS with RLPO and consistency regularization.

    Source: §1 · p. 2

  • Results

    Across MESS, the method consistently improves OVSS performance without dense masks and remains effective under noisy preference feedback.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Prompt-induced variation can serve as a practical supervision signal for OVSS adaptation where dense annotation is costly.

    Source: §5 · p. 9

    The method assumes target concepts can be expressed through prompts and that queried candidates include at least one useful hypothesis; similarly inaccurate template predictions provide less informative preferences.

    Source: §6 · p. 9

Abstract

from arXiv · show

Open-vocabulary semantic segmentation (OVSS) enables pixel-level prediction over arbitrary text-specified vocabularies and has shown strong generalization on common benchmarks. However, OVSS performance often degrades in specialized domains such as medical imaging, remote sensing, and industrial inspection, where dense pixel-level masks for adaptation are costly to obtain and require domain-specific expertise. We propose a preference-guided adaptation framework that replaces dense mask supervision with binary preferences. We observe that different prompt templates produce systematically different segmentations for the same image, a phenomenon we call prompt disagreement, and we repurpose it as a built-in source of preference supervision. Building on this, we mine localized preference queries from regions of high cross-template uncertainty, and adapt the OVSS model with Region-Localized Preference Optimization (RLPO) together with consistency regularization that stabilizes updates outside the queried region. Across extensive experiments on the MESS benchmark, the proposed method achieves consistent gains across diverse OVSS backbones without any pixel-level annotation, and remains effective under noisy preferences. Our code is available at https://github.com/blue-531/pref-ovss.

1 Introduction

The introduction reframes prompt-template variation as preference supervision for adapting OVSS in specialized domains without dense masks. It presents localized optimization with consistency regularization and reports consistent MESS gains.

1 Introduction

  • Specialized-domain adaptation is difficult because domain-specific vocabulary, fine-grained categories, and ambiguous boundaries make dense mask annotation costly and expert-dependent.OVSS otherwise supports pixel-level prediction for text-specified classes, including categories unseen during training.Source: §1 · p. 1
  • The method improves OVSS performance across MESS domains and baselines without dense masks, remains effective with noisy preferences, and exceeds the per-domain template spread.The figure compares validation mIoU across prompt templates; the prose reports consistent gains across diverse OVSS baselines.Source: Figure 1 · p. 2, §1 · p. 2
  • Binary preferences between candidate segmentations provide relative supervision without requiring fully specified pixel-level ground-truth masks.Annotators choose which prediction better matches the intended segmentation.Source: §1 · p. 2
  • RLPO mines high-uncertainty regions and maximally disagreeing template pairs, applies preference learning within the selected region, and regularizes predictions outside it.The regularizer uses the preferred prediction as a pseudo-target for the rejected prediction outside the queried region.Source: §1 · pp. 2–3

2 Related Works

Related work describes OVSS approaches and the difficulty of adapting them to specialized domains without dense masks. It positions localized binary preferences as a gap in existing preference-based segmentation research.

2.1 Open-Vocabulary Semantic Segmentation

  • Two-stage, one-stage, and cost-aggregation methods provide distinct routes from vision-language features to per-pixel OVSS predictions.They respectively classify mask proposals, predict masks with a side adapter, or refine patch embeddings for dense prediction.Source: §2.1 · p. 3
  • Specialized-domain OVSS adaptation remains difficult because existing prompt-tuning, adapter, and personalized approaches typically rely on costly expert-produced dense masks.The paper proposes learning from binary comparisons between candidate segmentations instead.Source: §2.1 · p. 3

2.2 Preference Learning

  • Preference learning replaces fully specified targets with relative judgments, and DPO learns directly from pairwise preferences without an explicit reward model.The cited preference-learning work presents comparative feedback as useful when dense annotation is difficult to obtain.Source: §2.2 · p. 3
  • Prior preference-based segmentation studies focus on fixed-target settings, often with a single foreground structure or small closed label space, leaving OVSS adaptation largely unexplored.The paper addresses this gap with localized binary comparisons constructed from prompt-induced variation.Source: §2.2 · p. 3

3 Method

The method adapts OVSS from binary preferences over prompt-conditioned segmentations, combining localized query mining, RLPO, and consistency regularization.

3 Method

  • The framework adapts from binary preferences over target-image predictions through query mining, RLPO, and consistency regularization.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Preliminaries

  • Each arriving target image elicits one binary preference and one update; lightweight adapters train against a frozen initial reference.Source: §3.1 · pp. 4–5

3.2 Preference Query Mining

  • Queries ask for a binary judgment within a region selected by high cross-template entropy and the most disagreeing template pair there.Source: §3.2 · p. 5

3.3 Region-Localized Preference Optimization (RLPO)

  • RLPO compares winner and loser using class-balanced region scores, raising winner confidence and lowering loser confidence relative to the frozen reference.Source: §3.3 · pp. 5–6

3.4 Consistency Regularization

  • Consistency regularization aligns the loser prediction with the winner pseudo-label outside the queried region, only at sufficiently confident pixels.Source: §3.4 · p. 6

4 Experiments

Experiments evaluate preference-guided adaptation across MESS domains, OVSS backbones, and CLIP scales. Results show consistent gains, benefits from each adaptation component, and resilience to noisy preferences.

4.1 Experimental Setup

  • Experiments cover five MESS domain groups, using four OVSS backbone variants, lightweight adapters, and 14 prompt templates.The preference oracle selects the candidate with higher ground-truth IoU within the queried region; adaptation uses 64 images per dataset unless otherwise noted.Source: §4.1 · pp. 6–7

4.2 Main Results

  • Mean mIoU gains are +7.10 (SAN-B), +6.55 (CAT-Seg-B), +6.59 (SAN-L), and +10.62 (CAT-Seg-L), exceeding the dense-mask reference on selected domains.The largest domain gains occur in Medical Sciences, including +19.46 on SAN-B and +22.31 on CAT-Seg-L; improvements are robust across backbone scale and architecture.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 Ablation Studies

  • At matched K = 14, prompt disagreement achieves mean mIoU 45.88, above MC Dropout (39.09) and test-time augmentation (44.66).Template-induced candidates vary the segmentation hypothesis while preserving the target vocabulary, without an additional diversity mechanism or hyperparameter.Source: Table 2 · p. 8, §4.3 · p. 8
  • Removing LRLPO lowers mean mIoU from 45.88 to 40.51 (−5.37), while removing Lcons yields 43.45 (−2.43).RLPO is the primary adaptation driver; consistency regularization provides complementary stability outside the queried region.Source: Table 3 · p. 8, §4.3 · p. 8

4.2 Main Results

  • Qualitative comparisons show adapted predictions better align with ground truth across both CLIP scales, correcting bird-species confusions and aerial-scene hallucinations.The examples are SAN on CUB-200 and CAT-Seg on iSAID.Source: §4.2 · p. 7, Figure 3 · p. 8

4.3 Ablation Studies

  • Mean gains rise from +3.00, +4.24, and +7.05 at 4, 8, and 16 images to +10.62 at 64, then +10.53 at 128.Medical Sciences gains +11.51 at 4 images and saturates by 64, while Engineering peaks at +19.91 with 128 images.Source: §4.3 · pp. 8–9, Table 4 · p. 9

4.4 Robustness Analysis

  • At p = 0.05, gains remain essentially unchanged from clean preferences; at p = 0.20, the method still recovers roughly 8 mIoU on average.Medical Sciences and Engineering retain the largest gains under heavy noise.Source: Figure 4 · p. 9, §4.4 · p. 9

5 Conclusion

The paper turns prompt disagreement into localized binary-preference supervision for adapting OVSS to specialized domains. On MESS, it reports consistent gains without dense masks, with sample efficiency at small adaptation budgets and robustness to noisy feedback.

5 Conclusion

  • Prompt disagreement supplies localized comparison queries that train OVSS through Region-Localized Preference Optimization and consistency regularization using binary preferences over small regions.The method reports consistent improvements across MESS without dense masks, and remains sample-efficient at small adaptation budgets and robust under noisy feedback.Source: §5 · p. 9

6 Limitations

The framework assumes prompts can express the target vocabulary and that their candidate segmentations include a useful hypothesis in the queried region. The appendix specializes DPO to template-conditional, region-localized segmentation preferences.

6 Limitations

  • The framework assumes natural-language prompts express the target vocabulary and that prompt-induced candidates include a useful hypothesis in the queried region.When templates are similarly inaccurate, choosing the less-wrong candidate offers limited guidance; this may affect highly domain-specific concepts poorly captured by natural-image-style templates.Source: §6 · p. 9

A Derivation of the Region-Localized Preference Optimization Loss

  • RLPO specializes standard DPO to OVSS through cross-template Bradley–Terry comparisons, class-balanced regional scoring, and an online-DPO interpretation.The derivation treats each image-template pair as a context and each template-induced segmentation as its response.Source: A Derivation of the Region-Localized… · p. 17, §A.2 · pp. 17–18

A.1 DPO Recap

  • The DPO recap substitutes the reward log-ratio into a Bradley–Terry preference, where shared-context normalizers cancel to yield the DPO loss.The appendix identifies this as the standard derivation later specialized to OVSS.Source: §A.1 · p. 17

A.2 Adaptation to Template-Conditional Segmentation

  • Cross-template winner and loser predictions come from sibling image-template contexts; the residual normalizer offset is independent of trainable parameters and contributes no gradient.The preference score is restricted to the queried region, and class-balanced averaging prevents dominant classes from suppressing smaller present classes.Source: §A.2 · pp. 17–18
  • RLPO is online or iterative DPO because template-induced hard predictions are fixed within each optimization step but evolve with model parameters across steps.The argmax response corresponds to the zero-temperature limit of softmax sampling.Source: §A.2 · p. 18

B Template Analysis

The template analysis finds that sentence and scale variations offer complementary domain-specific strengths, while their combination provides the strongest mean mIoU and more diverse candidate segmentations.

B Template Analysis

  • The analysis compares a full ViLD pool with sentence-only and scale-only diagnostic sets, separating sentence variation from scale variation.The scale-only set fixes the sentence structure and varies four scale modifiers; the sentence-only set contains five templates without scale modifiers.Source: B Template Analysis · p. 18
  • Sentence-only and scale-only subsets perform similarly overall, but their relative strengths vary across domains, with scale-only stronger on some agriculture and engineering datasets and sentence-only stronger on several medical datasets.The passages characterize scale modifiers alone as not consistently improving segmentation quality.Source: B Template Analysis · p. 18
  • 45.88 mean mIoU from the full 14-template set exceeds the sentence-only subset at 43.79 and scale-only subset at 43.65.The full ensemble combines sentence and scale variations and introduces more diverse, semantically aligned candidates for preference-query mining.Source: B Template Analysis · pp. 18–19, Table 5 · p. 19

C Implementation Details

The implementation uses lightweight vision and text adapters while freezing other backbone parameters. Human judgments closely match the preference oracle overall, with disagreement concentrated in hard, near-tie comparisons.

C Implementation Details

  • The adapter uses rank-4 vision LoRA on Q and V projections in the last four CLIP image-encoder blocks and a rank-4 residual text adapter; other backbone parameters stay frozen.Training uses AdamW and a streaming batch size of one image, with shared settings for temperature, consistency weight, confidence threshold, and entropy quantile.Source: C Implementation Details · p. 19

D Validation of the Preference Oracle

D.1 Human–Oracle Agreement

  • 0.920 of human judgments match the oracle overall, with agreement of 0.967 on easy, 0.953 on medium, and 0.840 on hard pairs.The reported residual disagreement is concentrated in near-tie queries.Source: §D.1 · p. 20, Table 7 · p. 20

D.2 Noise Concentrated on Near-Tie Queries

The near-tie stress test flips preferences only for queries with IoU margin below a threshold, using random replacement or adversarial inversion. Performance degrades gracefully, including when all queries below 0.15 are inverted.

D.2 Noise Concentrated on Near-Tie Queries

  • The stress test targets queries with IoU margin below threshold τ, replacing oracle choices randomly or actively inverting them in the adversarial model.The inversion condition is described as an upper bound on realistic annotator error.Source: §D.2 · p. 20
  • 42.43 mean remains above the 35.26 zero-shot baseline when every near-tie query with margin below 0.15 is actively inverted.The authors describe this as the most severe tested setting.Source: §D.2 · p. 21
  • Humans disagree with the oracle on only 16% of hard pairs, placing the realistic error regime comfortably within the range the method tolerates.Source: §D.2 · p. 21

E Comparison with Alternative Adaptation Strategies

On CAT-Seg-L with the same 64 target-domain images, the method reaches 45.88 mIoU with one binary judgment per image, exceeding point-supervised and annotation-free alternatives. Dense-mask references score higher.

E Comparison with Alternative Adaptation Strategies

  • The comparison uses CAT-Seg-L and the same 64 target-domain images across methods, spanning annotation-free inference through point, preference, and dense-mask supervision.The alternatives include prompt ensembling, point supervision, dense-mask prompt selection, matched-budget dense-mask adaptation, and fully supervised prompt tuning.Source: E Comparison with Alternative… · p. 21
  • 45.88 mIoU with one binary judgment per image exceeds the point baseline’s 42.41, while matched-budget dense masks reach 46.67 and full supervision reaches 53.17.The authors do not claim to match the multi-epoch fully supervised reference.Source: E Comparison with Alternative… · p. 21
  • Prompt ensembling scores 34.74 versus the 35.26 zero-shot baseline, while dense-mask prompt selection reaches 38.20 and the point baseline reaches 42.41.Prompt selection requires per-dataset dense masks; the point baseline requires clicks on every object instance.Source: E Comparison with Alternative… · p. 21

F Detailed Results on MESS Benchmark

The appendix expands MESS results and ablations, reports hyperparameter sensitivity, and details the method’s small trainable footprint and adaptation costs. The reported mean mIoU is robust to the tested hyperparameter variations.

F Detailed Results on MESS Benchmark

  • Per-dataset results expand the group-level MESS comparisons across SAN-B, CAT-Seg-B, SAN-L, and CAT-Seg-L, while per-dataset ablation breakdowns cover loss, candidate generation, and training-set size.Table 14 highlights the better of zero-shot and adapted results within each backbone; the supervised reference is excluded from that comparison.Source: F Detailed Results on MESS Benchmark · pp. 21–22

G Hyperparameter Sensitivity

  • Across four hyperparameters, mean mIoU stays within approximately ±1 point of default and every variation remains above the 35.26 zero-shot baseline.The default settings achieve the best overall mean and are competitive across domain groups.Source: G Hyperparameter Sensitivity · p. 22, Table 10 · p. 22

H.1 Trainable Parameters

  • The trainable footprint is 71,681 parameters on CAT-Seg-L and SAN-L, roughly 0.017% of the full model.It comprises a 65,536-parameter vision LoRA and a 6,145-parameter text adapter attached to shared CLIP components.Source: §H.1 · pp. 22–23

H.2 Adaptation Cost

  • Adaptation adds +1,466 MiB and +1,543 ms per step on CAT-Seg-L, versus +5,474 MiB and +365 ms on SAN-L.CAT-Seg-L’s IoU-based candidate selection requires a full sliding-window forward per template; SAN-L scores all 14 candidates with native single-pass forwards.Source: §H.2 · p. 23

I Additional Qualitative Results

Additional visualizations show adapted predictions across MESS domains and illustrate how cross-template uncertainty localizes preference queries. The query examples identify high-entropy areas where templates disagree most strongly.

I Additional Qualitative Results

Loading 2609.34528v1…