Source-linked AI summary
Mitigating Saliency Collapse: Robust Saliency-Aware Long-Text Image-Text Alignment
Qiuyu Kong, Zanxi Ruan, Marco Cristani, Yiming Wang
TL;DR
Long-text alignment models can over-rely on visually prominent text, leaving informative context underused and retrieval vulnerable when salient content is reduced. ROSA decomposes image-text pairs into salient and contextual views and trains their alignment with auxiliary objectives. The paper reports improved long-text retrieval, robustness to salient-text reduction, and plug-and-play gains from the objectives.
Problem
Method
Results
ROSA improves long-text retrieval and robustness under reduced salient information, while its auxiliary objectives consistently boost existing fine-tuning methods.
Source: §1 · p. 3, §6 · p. 10Takeaways & Limitations
The paper concludes that joint learning from salient and contextual evidence can mitigate saliency collapse in long-text alignment.
Source: §6 · p. 10ROSA relies on external saliency and LLM decomposers whose generalization beyond evaluated settings remains open, and experiments mainly cover CLIP-based methods.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Long-text image-text alignment is essential for fine-grained scene understanding, yet existing methods mainly focus on extending context length or improving local alignment. We identify an overlooked failure mode, saliency collapse, where models over-rely on text that features visual prominence while underutilizing non-salient context. Consequently, retrieval performance degrades sharply when salient content is reduced, despite that the non-salient context remains informative. To address this, we propose RoSA, a saliency-aware finetuning framework that decomposes image-text pairs into complementary salient and contextual views, enabling joint learning of prominent and contextual semantics. We propose a modality-independent decomposition strategy to combine object-level visual saliency estimation with LLM-based text decomposition to bridge vision and language. We employ auxiliary objectives to explicitly align these views, ensuring robust representation with both salient and contextual features. RoSA greatly improves long-text retrieval and demonstrates strong robustness to saliency collapse across benchmarks. Moreover, our auxiliary objectives are plug-and-play, bringing consistent gains to existing tuning methods. Code and checkpoints are available at: https://github.com/intelligolabs/RoSA
1 Introduction
Long-text alignment supports fine-grained scene understanding, but existing approaches have largely focused on context length, local alignment, and structural priors. The section identifies saliency collapse and introduces ROSA, which jointly learns from salient and contextual views through modality-independent decomposition and auxiliary alignment objectives.
1 Introduction
- Long-text alignment is important for fine-grained scene understanding, while prior approaches address the 77-token CLIP text-encoder limit through longer context, local alignment, or structural priors.The cited approaches include context extension, local spatial alignment, and auxiliary structural priors.Source: §1 · p. 1
- Existing long-text alignment methods over-rely on salient text and degrade faster when it is reduced than when matched amounts of random text are removed.Saliency collapse arises because information-dense salient evidence offers a retrieval shortcut that can leave contextual evidence underused.Source: §1 · pp. 1–2
- ROSA decomposes image-text pairs into complementary salient and contextual views and applies auxiliary alignment objectives alongside the primary image-text objective.Its objectives include saliency-focused and context-preserving contrastive alignment, plus visual anchoring; decomposition occurs during fine-tuning and adds no inference cost.Source: §1 · p. 2
- ROSA uses an off-the-shelf object saliency estimator for images and an LLM to decompose captions in a modality-independent strategy.The paper motivates object-level saliency as a bridge between visual prominence and language semantics.Source: §1 · p. 2
- ROSA improves long-text retrieval across benchmarks, is more robust under salient-text reduction, and its auxiliary objectives consistently boost existing models.The contribution summary reports best performance on standard long-text retrieval benchmarks and superior robustness as salient text is progressively reduced.Source: §1 · p. 3
2 Related Works
Prior work addresses long-text alignment through context extension, fine-grained or structural alignment, and visual saliency guidance, but typically does not jointly decompose visual and textual inputs into salient and contextual parts. ROSA applies saliency as a multimodal training prior to align both kinds of content.
2 Related Works
- Long-text alignment methods extend encoders, add global-local interactions, or strengthen structural cues, while ROSA separates salient semantics from contextual descriptions for joint learning.The section names Long-CLIP, TULIP, GOAL, and StructXLIP.Source: §2 · p. 3
- ROSA differs from prior saliency-guided learning by splitting long captions into salient and contextual text segments alongside corresponding visual views.Earlier methods generally used saliency to guide visual attention, grounding, or reasoning while retaining global textual supervision.Source: §2 · p. 3
- ROSA decomposes image and caption inputs for salient and contextual alignment, whereas DAPT targets short-text prompt tuning with category-level prompts.The section says DAPT is difficult to extend to long-text alignment, which involves class-agnostic descriptions with densely entangled content.Source: §2 · pp. 3–4
3 Preliminary Study
The preliminary study tests whether long-text retrieval models rely disproportionately on salient descriptions and whether non-salient context remains informative. Across controlled query reductions and two datasets, it finds greater performance loss from salient deletion and substantial information in context that current multimodal models underuse.
3 Preliminary Study
- The study defines salient text as descriptions referring to visually salient regions and examines existing models when that information is incomplete or absent.It uses LLM-partitioned salient and contextual text, then varies the percentage of salient tokens retained in the query.Source: §3 · p. 4
- Salient text alone retains more than 80% of full-caption R@1 at α=100, while progressively reducing salient text consistently lowers R@1 across models and both datasets.The study evaluates LongCLIP-FT, GOAL, and StructXLIP on DOCCI and INSECT, with text-to-image R@1 averaged over three runs.Source: §3 · p. 4
- Salient deletion produces a larger average R@1 drop than matched-length random deletion and remains more damaging under controls for entity count, lexical density, caption position, and text-only discriminability.Figure 2 compares salient and random-span reduction on DOCCI and INSECT; the random control removes the same number of tokens.Source: §3 · pp. 4–5, Figure 2 · p. 5
- Non-salient queries achieve 71.86% R@1 on DOCCI and 41.90% on INSECT for caption retrieval, while StructXLIP gets 30.96% and 4.08% for text-to-image retrieval, respectively.The comparison suggests context is informative but is not effectively represented by current multimodal alignment models.Source: §3 · p. 5
- The analyses characterize saliency collapse as disproportionate retrieval degradation when salient cues are reduced despite the presence of rich contextual cues.The section concludes that models should encode salient signals and complementary contextual evidence.Source: §3 · p. 5
4 Method
ROSA independently decomposes each image-caption pair into salient and contextual views, then fine-tunes the encoders with complementary alignment objectives alongside the original contrastive loss. The auxiliary views are used only during fine-tuning, so inference retains the standard pipeline without extra cost.
4 Method
- ROSA combines the original image-text contrastive loss with three saliency-aware objectives during fine-tuning, while inference uses the standard pipeline with no extra cost.The objectives align salient views, align contextual views, and anchor the salient image representation to the original image.Source: §4 · pp. 5, 7, Figure 3 · p. 6
- The original image-text alignment objective is a symmetric InfoNCE loss with temperature parameter τ.The encoders map image and caption inputs into a shared embedding space for contrastive alignment.Source: §4 · pp. 5–6
- A saliency detector and mask produce visual views, while an LLM independently partitions each caption into salient and contextual text without rewriting it.Invalid, unsupported, or incomplete LLM outputs are automatically detected and regenerated.Source: §4 · p. 6
- The saliency-focused contrastive loss aligns salient visual regions with salient text, and the context-preserving loss aligns contextual images with contextual text.The contextual objective encourages learning non-salient but informative cues for cases when salient information is incomplete.Source: §4 · pp. 6–7
- The visual anchoring loss keeps the salient-view representation close to the original image representation to prevent severe representation drift.This addresses the risk that saliency-guided fine-tuning moves the original image representation away from the pretrained embedding space.Source: §4 · p. 7
5 Experiments
ROSA is evaluated on four long-text retrieval benchmarks against CLIP-based fine-tuning methods, with tests of full-caption retrieval, reduced-saliency robustness, and auxiliary design choices. The results report gains in standard and reduced-query settings, plug-and-play improvements, and ablation evidence for the decomposition and alignment choices.
5 Experiments
- Evaluation covers four fine-grained retrieval benchmarks, comparing representative CLIP-based fine-tuning methods under the corresponding training and test splits.The setup uses CLIP ViT-B/16, KPS to extend the text encoder, object-level visual saliency, and Qwen2.5 text decomposition; results average three seeds.Source: §5 · p. 7
5.1 Comparison with existing CLIP-based methods
- ROSA achieves the best results on most metrics across four benchmarks and retrieval directions, with Sketchy gains of +1.64% T→I R@1 and +3.28% I→T R@1 over the best competitor.Cross-modal maps show cleaner concentration on caption-related visual entities than StructXLIP.Source: Table 2 · p. 8, §5.1 · pp. 8–9, Figure 4 · p. 9
- +7.08% on INSECT T→I R@10 for DoRA and +8.32% on DOCCI T→I R@1 for SigLIP2 are the largest reported gains from adding ROSA’s objectives.The objectives yield consistent improvements across the evaluated CLIP-based fine-tuning methods.Source: Table 3 · p. 8, §5.1 · p. 8
- 5.13% on DOCCI Salient50%+Context and 11.04% on DOCCI Context-only are ROSA’s reported R@1 gains over the best competitor.ROSA achieves the best results in most reduced-query settings across DOCCI, DCI, and INSECT; its advantage grows as salient cues weaken or disappear.Source: §5.1 · p. 8, Table 4 · p. 9
5.2 Ablation studies
- +3.54% INSECT T→I R@10 and +3.37% I→T R@5 are reported when visual anchoring is added to Lsal + Lctx.Each loss improves retrieval; Lsal + Lctx improves most metrics over either alone, and Lctx helps when salient text is reduced or absent.Source: Table 5 · p. 9, §5.2 · p. 9
- Salient-only visual anchoring achieves the best results across both datasets and retrieval directions among the tested anchoring variants.The authors suggest that anchoring contextual views to the full image may weaken their role as complementary evidence, while anchoring both views can introduce competing constraints.Source: Table 6 · p. 9, §5.2 · pp. 9–10
- FASA yields the best performance with consistent gains among the compared saliency-detection alternatives.SAM3’s slight INSECT T→I R@1 gain does not persist at larger recall levels or in I→T retrieval.Source: Table 7 · p. 10, §5.2 · p. 10
- The modality-independent decomposition strategy most often scores best overall, with Qwen2.5-VL the second-best strategy.The authors suggest that early cross-modal coupling may create a learning shortcut; their strategy avoids that coupling while producing generally semantically aligned content.Source: Table 8 · p. 10, §5.2 · p. 10
6 Conclusion
The conclusion characterizes saliency collapse as over-reliance on visually prominent descriptions and underuse of informative non-salient context. It presents ROSA’s complementary-view objectives as a way to address this vulnerability, while noting open generalization and scope limits.
6 Conclusion
- Current long-text alignment models over-rely on visually prominent descriptions and underuse informative non-salient context.The paper characterizes this vulnerability as saliency collapse.Source: §6 · p. 10
- ROSA decomposes image-text pairs into complementary views and uses auxiliary objectives to learn salient and contextual evidence jointly.The paper reports consistent long-text retrieval gains in a plug-and-play manner and superior robustness under reduced salient information.Source: §6 · p. 10
- Generalization of external saliency and LLM decomposers beyond evaluated settings remains open; experiments focus mainly on CLIP-based methods.The paper also notes that saliency reduction may not cover the full diversity of human attentional mechanisms and that offline text decomposition can challenge scalability.Source: §6 · p. 10
Supplementary Material
The supplementary material provides dataset details, decomposition validation, and additional analyses supporting the main paper.
Supplementary Material
- Section A provides detailed dataset descriptions and statistics.Source: Supplementary Material · p. 15
- Section B documents salient/contextual view generation and validation, including textual and visual decomposition, human validation, stability, and cross-modal correspondence.Source: Supplementary Material · p. 15
- Sections C–F cover controlled saliency-collapse analyses, additional robustness and exploratory analyses, and qualitative comparisons with the best competitor.Section C includes single-factor controlled deletion and entity-controlled reconstruction.Source: Supplementary Material · p. 15
A Dataset details
The experiments use four fine-grained retrieval benchmarks and create fixed, complementary salient and contextual views of each image–text pair. Human judgments and stability tests assess the textual decomposition, while a swapping control tests whether the intended cross-view correspondence matters.
A Dataset details
- The four benchmarks cover general scenes, fashion, and insects, with captions ranging from 16 words on average in Sketchy to over 1,000 in DCI.DOCCI has 15K images and 136-word captions; the INSECT experiment uses a 6K fine-grained biology retrieval split with 80.78-word descriptions.Source: A Dataset details · p. 15
- Semantic entities are noun-phrase-level visual concepts extracted with spaCy after removing overly generic or non-specific terms.The paper reports that all four datasets contain rich semantic entities.Source: A Dataset details · p. 15
B.1 Salient/contextual pair generation
- Qwen2.5 separates caption content into salient and complementary contextual text, while FASA masks salient image regions and retains their complement as the contextual image.The decompositions are generated offline and fixed across experiments; outputs are checked for format, unsupported content, and incomplete caption reconstruction before training.Source: §B.1 · pp. 16–18
B.2 Human Validation
- Annotators judged 84.00% of salient and 84.67% of contextual descriptions visually observable, with ground-truth identification rates of 94.00% and 90.67%, respectively.The evaluation covered 150 samples from DOCCI, DCI, and Sketchy.Source: §B.2 · p. 18, Table 9 · p. 18
- 77.58% token F1 and 69.04% IoU measure LLM–human saliency-span agreement, versus 47.88% and 36.96% for equal-length random spans.The LLM split covers 92.74% of human-salient tokens across 5,100 DOCCI test samples.Source: §B.2 · p. 18
B.4.1 Salient–contextual swapping control
- Correct salient–contextual pairing wins all eight reported metrics, including +2.31/+2.81 DOCCI T→I/I→T R@1 over swapped pairing.The comparison keeps decomposed views, training settings, and computation fixed.Source: Table 11 · p. 19, §B.4.1 · p. 19
B.3 Prompt and LLM Stability
- Replacing Qwen2.5 with Llama-3.1 leaves retrieval performance comparable, while output semantic similarity ranges from 0.808–0.912 across prompts and 0.841–0.909 across LLMs.Source: §B.3 · pp. 18–19, Table 10 · p. 19
B.4.2 Robustness to cross-modal mismatch
The authors test whether independently generated cross-modal views introduce harmful mismatches and whether textual factors explain the saliency-deletion asymmetry. Filtering low-agreement auxiliary pairs does not improve overall results, and matching individual text factors does not remove the observed deletion gap.
B.4.2 Robustness to cross-modal mismatch
- The mismatch analysis tests whether independently generated salient/contextual image–text pairs could create noisy auxiliary supervision during fine-tuning.A frozen CLIP model scores the pairs before the lowest-scoring proportions are filtered.Source: §B.4.2 · p. 19
- The unfiltered model achieves the best overall performance on DOCCI and INSECT; filtering low-agreement pairs often degrades performance.Filtering removes the lowest-scoring 5%, 10%, or 20% pairs from auxiliary losses while retaining original image–text pairs.Source: §B.4.2 · p. 19, Table 12 · p. 20
C.1 Single-factor controlled deletion
- Positive average contextual-deletion minus salient-deletion T→I R@1 gaps remain in all six model–dataset settings under each of four separate matching controls.The controls separately match entity count, TF–IDF density, caption position, or text-only discriminability; none jointly matches all four factors.Source: §C.1 · p. 20, C Controlled Analyses of Saliency… · p. 20
C.2 Entity-controlled reconstruction
Entity-controlled reconstruction fixes query structure while progressively replacing salient entities with matched contextual entities. Retrieval declines as replacements increase, and ROSA has the highest mean R@1 at every tested ratio with a smaller endpoint drop than the baselines.
C.2 Entity-controlled reconstruction
- ROSA’s T→I R@1 drops 7.49% (21.02% to 13.53%), versus StructXLIP’s 10.72% (20.14% to 9.42%).ROSA has the highest mean R@1 at every replacement ratio.Source: §C.2 · p. 20, Table 14 · p. 21
- T→I R@1 consistently decreases as salient entities are replaced by matched contextual entities, while ROSA shows a smaller endpoint drop than the baselines.The reconstruction uses four entities and replacement ratios from 0% to 100%.Source: §C.2 · p. 20, Table 14 · p. 21
- Across 39/39 baseline endpoint comparisons, ROSA narrows the saliency-reliance gap under a reconstruction that fixes template, entity count, and entity positions while closely matching token length and IDF.Source: §C.2 · pp. 20–21
D Robustness to alternative query construction
The authors test whether robustness persists when test queries use alternative decompositions, including human-built queries, without retraining the checkpoints. ROSA retains its advantage across most comparisons and all Context-only comparisons.
D Robustness to alternative query construction
- The test regenerates queries with a rewritten Qwen2.5 prompt, Llama-3.1, or human construction while keeping checkpoints frozen and avoiding retraining.Human annotators partitioned the complete 5,100-sample DOCCI test set.Source: D Robustness to alternative query… · p. 21
- Across the original construction and three alternatives, ROSA wins 39/42 comparisons and all 14/14 Context-only comparisons.Its advantage remains largest when salient evidence is absent, including with human queries constructed without an LLM.Source: D Robustness to alternative query… · p. 21
- On DOCCI Context-only T→I R@1, ROSA’s gains are 12.18%, 9.30%, and 8.84% for rewritten-prompt, Llama-3.1, and human queries, respectively.On INSECT, the corresponding gains are 0.10% and 1.98% for rewritten-prompt and Llama-3.1 queries.Source: D Robustness to alternative query… · p. 21
E.1 Cross-domain Evaluation on PathGen
ROSA performs strongly on the condensed PathGen benchmark, and additional grounding experiments show gains across localization, local-description retrieval, and fine-grained discrimination. PathGen is constructed to reduce redundant captions while retaining diverse pathological themes.
E.1 Cross-domain Evaluation on PathGen
- PathGen condensation targets redundancy in a dataset whose patches and captions often repeat within the same whole-slide image.The authors motivate a compact, semantically diverse benchmark as practically meaningful and computationally desirable when million-scale expert annotations are impractical.Source: §E.1 · pp. 21–22
- PathGen pairs are condensed by clustering captions within each WSI and retaining the sample nearest each cluster centroid.WSIs are split 80%/20% before condensation, keeping each WSI entirely within one split; the resulting dataset contains 15,749 training and 4,037 testing pairs.Source: §E.1 · p. 22
- ROSA consistently achieves the strongest results among CLIP-based fine-tuning baselines on challenging PathGen retrieval.The passage reports relatively low absolute retrieval performance across methods; Table 16 reports R@K for both retrieval directions.Source: §E.1 · pp. 22–23, Table 16 · p. 22
- ROSA’s saliency-guided module improves both CLIP-based fine-tuning and PEFT adaptation methods on PathGen.Table 17 reports Recall@K for Text→Image and Image→Text, with parentheses showing absolute change over each baseline.Source: Table 17 · p. 23, §E.1 · p. 24
E.2 Fine-grained grounding
- +0.91% oIoU and +1.08% mIoU over StructXLIP on zero-shot RefCOCOg localization.Evaluation uses DCI-trained encoders without RefCOCOg or segmentation-specific training.Source: §E.2 · p. 24, Table 18 · p. 24
- ROSA consistently outperforms GOAL and StructXLIP on DOCCI-FG local-description retrieval.The benchmark uses 5,000 images and 35,533 fine-grained local descriptions; Table 19 reports T→I Recall@K.Source: §E.2 · p. 24, Table 19 · p. 24
- Adding ROSA to StructXLIP improves local fine-grained discrimination accuracy by 3.77%, 2.91%, and 3.61% in standard, Pick5, and CLIP-hard-negative settings.The evaluation uses 1,274 mask-aligned DCI subcrops without full images.Source: §E.2 · p. 24
F Qualitative results
Across four datasets, ROSA produces more focused and semantically consistent attention responses than StructXLIP. The qualitative comparisons also note that both methods are less accurate on underrepresented insect images.
F Qualitative results
- Across Sketchy, INSECT, DOCCI, and DCI, ROSA produces more focused, semantically consistent attention responses than StructXLIP.Figure 9 visualizes attention for the blue bold text in long captions.Source: F Qualitative results · p. 24, Figure 9 · p. 25
- On Sketchy, ROSA covers target garments more completely; on DOCCI, it localizes the described nose sticker more accurately.On underrepresented INSECT images, both methods are generally less accurate, though ROSA still aligns better with text.Source: F Qualitative results · pp. 24, 26
NeurIPS Paper Checklist
- The checklist reports that code and models will be made publicly available upon acceptance, and lists GPU resources and approximate training time.Experiments used RTX 5090 GPUs with 32GB memory and A100 GPUs with 40GB memory; runs took approximately 1–3 hours depending on dataset.Source: NeurIPS Paper Checklist · pp. 29–30
- The paper reports no error bars or statistical significance tests, while stating that methods use fixed benchmark splits and identical retrieval metrics.The checklist says improvements are consistently observed across multiple datasets, backbones, and retrieval directions.Source: NeurIPS Paper Checklist · p. 29