PaperlayerView exact revision on arXiv

Source-linked AI summary

Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrieval

Haoyue Liu, Ye Chen, Zhichao Wang, Xiaoying Tang

arXiv:2608.18521v1cs.AIcs.LG

TL;DR

Dense-caption retrieval suffers premature InfoNCE saturation because near-duplicate captions leave the hardest negatives unresolved. HN-CLIP uses detached text–text similarities to impose adaptive margins on those negatives. Across four benchmarks, it improves R@1 over strongest competitors, accelerates training, and remains effective with less data and across fine-tuning frameworks.

  • Problem

    InfoNCE rapidly loses optimization signal under strong initialization while highly similar negatives remain unresolved in dense-caption retrieval.

  • Method

    HN-CLIP adds a detached, diagonal-masked caption-similarity matrix to negative logits, assigning larger adaptive margins to more similar captions without changing the batch.

  • Results

    +2.4–+4.3 R@1 over the strongest competitors is achieved across four dense-caption benchmarks, while all six tested fine-tuning frameworks improve on in-domain benchmarks.

  • Takeaways & Limitations

    The objective reaches the strongest full-data baseline with only one fifth of the training data and adds no inference-time computation.

  • Takeaways & Limitations

    The approach relies on text–text geometry as a hardness signal, although the authors state it is recomputed from the current encoder and intentionally allowed to drift.

Abstract

from arXiv · show

Dense-caption retrieval has recently been improved by introducing segmentation, edge maps, LLM-filtered captions, and cross-modal modules into contrastive fine-tuning. However, these methods largely inherit the same InfoNCE objective, whose optimization can prematurely saturate under a strong pre-trained initialization: on dense captions, the loss falls below 10^{-3} on 80% of batches within the first epoch, while its gradient reaches exact zero in fp32 in 47% of measurements. We find that this behavior is closely related to the large number of near-duplicate captions in dense-caption benchmarks, where a few highly similar negatives remain unresolved after the easy majority has already been separated. As a remedy, we introduce HN-CLIP, which uses the text encoder's own text-text geometry to construct per-negative adaptive similarity margins. Specifically, a detached caption-similarity matrix is added to the negative logits, assigning larger margins to more similar captions without mining, synthesizing, or resampling negatives. The resulting objective requires only one caption-similarity matrix and a masked logit addition during training, with no auxiliary data, additional parameters, offline preprocessing, or inference-time overhead. Extensive experiments on four dense-caption retrieval benchmarks show that HN-CLIP improves over the strongest competitors by +2.4--+4.3 R@1 while training 2.4x faster than GOAL and 5.4x faster than StructXLIP. Moreover, the proposed objective improves all six tested fine-tuning frameworks on the in-domain benchmarks and reaches the strongest full-data baseline with only 20% of the training data.

1 INTRODUCTION

Dense-caption retrieval is difficult because near-duplicate descriptions leave hard negatives unresolved after InfoNCE quickly separates easier ones. HN-CLIP addresses this saturation by assigning larger adaptive margins to more similar captions, improving retrieval without adding inference machinery.

  • Dense-caption benchmarks require distinguishing long, detail-rich descriptions of visually similar scenes, motivating methods built on Long-CLIP.
  • 80% of measured batches fall below 10^-3 InfoNCE loss within the first epoch, while gradients are exactly zero in fp32 in 47% of measurements.This saturation leaves highly similar negatives unresolved despite rapid separation of easier negatives.
  • 0.92–0.94 cosine similarity is reached by each caption’s hardest negative, making a small set of near-duplicates decisive for R@1.
  • HN-CLIP converts detached caption–caption similarities into per-negative adaptive margins, increasing training pressure for more similar captions without mining or changing the batch.
  • +2.4–+4.3 R@1 over the strongest competitors is achieved across four dense-caption benchmarks, with no auxiliary inputs, preprocessing, added parameters, or inference-time computation.

2 METHOD

HN-CLIP fine-tunes a dual encoder by combining a text-similarity–boosted global objective with token-level late interaction. The detached caption geometry keeps optimization focused on hard negatives while preserving training and inference simplicity.

  • 2 METHOD: HN-CLIP fine-tunes a pre-trained dual encoder without auxiliary data views, preprocessing pipelines, or architectural modules.
  • 2.2 TEXT-SIMILARITY–BOOSTED HARD NEGATIVES: The boosted objective converts the batch caption-similarity matrix into per-negative adaptive margins and keeps loss and gradients active on hard negatives.
  • 2.2 TEXT-SIMILARITY–BOOSTED HARD NEGATIVES: The method computes detached caption similarities, masks the diagonal, and adds them to negative logits using γ, with γ defaulting to 0.5.
  • 2.2 TEXT-SIMILARITY–BOOSTED HARD NEGATIVES: Text–text geometry is preferred because it is available before optimization, symmetric across retrieval directions, and aligned with benchmark difficulty; recomputation permits deliberate margin drift.
  • 2.3 TOKEN-LEVEL ALIGNMENT BRANCH: The full objective combines the hard-negative loss with a token-level branch that aligns words and image patches, while inference remains byte-identical.
  • 2.4 GRADIENT-DYNAMICS ANALYSIS: 26% of batches remain active in every epoch, and the boosted gradient exceeds the standard gradient by 102–106× in per-epoch median where the latter is nonzero.On DOCCI, retrieval error continues falling from 20.5% to 15.1% for T→I and from 22.9% to 17.1% for I→T after standard InfoNCE flattens.

3 EXPERIMENTS

Experiments evaluate HN-CLIP against machinery-based baselines, framework integrations, domain transfer, data scaling, and objective ablations. Across these settings, HN-CLIP improves retrieval, converges faster, transfers across domains, and retains gains with less data.

  • 3.2 A1: HN-CLIP achieves competitive performance without any machinery: HN-CLIP achieves the best result in 23 of 24 Table 1 columns, ties the remaining column, and leads R@1 in all eight retrieval directions.Its R@1 gains over the strongest competitor are +3.52/+3.63 on DOCCI, +3.40/+4.00 on DCI, +3.69/+3.13 on Long-DCI, and +4.30/+2.40 on Urban-1K.
  • 3.2 A1: HN-CLIP achieves competitive performance without any machinery: HN-CLIP’s first epoch matches or exceeds most baselines’ final average R@1 on all four benchmarks.On DOCCI, its first epoch reaches 84.3 average R@1, exceeding FineLIP, GOAL, and StructXLIP’s final accuracy; the second epoch reaches 86.1 and surpasses every baseline.
  • 3.3 A2: The boost consistently improves existing frameworks in-domain: Every tested fine-tuning framework improves with the plug-in objective on both in-domain benchmarks.The tested frameworks are Long-CLIP, FineLIP, GOAL, StructXLIP, LoRA, and DoRA; gains increase with caption length and reach +3.7–+4.8 R@1 for LoRA/DoRA on Long-DCI.
  • 3.4 A3: The boost transfers across domains and data scales: HN-CLIP transfers strongly across domains, scoring 85.00/83.14 R@1 when trained on DCI and transferred to DOCCI.It exceeds every baseline’s in-domain DOCCI result and leads every column in the reverse and harder DOCCI→Long-DCI transfer settings.
  • 3.4 A3: The boost transfers across domains and data scales: At 20% of the training data, HN-CLIP beats the strongest full-data baseline, with the average-R@1 margin widening to +7.3 as data shrinks.It leads at every tested data fraction, and the 20% rows exceed GOAL’s 100% rows on both benchmarks.
  • 3.5 A4: Components and sensitivity: Ablations identify adaptive negative weighting, rather than token-level granularity, as the main source of improvement.LHN alone produces the bulk of the gains, while the token branch contributes only after LHN prevents saturation; every in-domain γ from 0.25 to 1 beats γ=0.

4 RELATED WORK

Prior dense-caption methods enrich the aligned inputs or representations while retaining InfoNCE unchanged. HN-CLIP instead changes negative weighting using text–text geometry and is designed to compose with those frameworks.

  • Long-text vision-language alignment: Long-CLIP provides the long-context backbone, while FineLIP, GOAL, SmartCLIP, DreamLIP, StructXLIP, FILIP, and PEFT enrich different alignment components.These methods respectively add cross-modal modules, regional matching, token reweighting, caption decomposition, edge-map and lexicon alignment, token-level alignment, or parameter-efficient adapters.
  • Long-text vision-language alignment: HN-CLIP differs by repairing how negatives are weighted within the inherited InfoNCE objective rather than adding machinery to what is aligned.Its objective-level modification composes with diverse fine-tuning frameworks.
  • Hard negatives in contrastive learning: Hard-negative research spans importance reweighting, synthetic mixing, debiasing, mining, and false-negative handling.HN-CLIP belongs to this broader line of work while using caption geometry to assign per-negative margins without mining or resampling.

5 CONCLUSION

The paper concludes that dense-caption retrieval benefits from sustaining optimization on near-duplicate negatives rather than adding more architectural machinery. HN-CLIP achieves strong benchmark, efficiency, and framework-wide results within that scope.

  • 5 CONCLUSION: InfoNCE largely saturates within the first epoch because near-duplicate captions remain hard after easy negatives are separated.HN-CLIP addresses this by using the same text–text geometry that exposes the problem.
  • 5 CONCLUSION: HN-CLIP sets a new state of the art on four benchmarks and matches or exceeds most baselines’ final accuracy after one epoch.The modification requires only one fifth of the training data to surpass the strongest full-data baseline.
  • 5 CONCLUSION: HN-CLIP improves all six tested fine-tuning frameworks on the in-domain benchmarks.The conclusion frames this as evidence that the objective-level modification generalizes across existing framework choices.

A.2 ONE BATCH, FOUR BENCHMARKS

A representative batch from each benchmark shows uniformly high caption–caption similarity, making nearly every in-batch negative a near-duplicate to some degree. HN-CLIP uses this geometry as its central training signal.

  • A.2 ONE BATCH, FOUR BENCHMARKS: 0.82–0.84 mean off-diagonal similarity makes every negative in the four displayed batches a near-duplicate to some degree.The detached caption-similarity matrix supplies each negative with a proportionate margin.
  • A.2 ONE BATCH, FOUR BENCHMARKS: Near-duplicate examples include different sports cars, dirt bikes, and white horses whose captions reach 0.93–0.97 cosine similarity.The captions describe different images while recombining shared elements of natural scenes.
  • A.2 ONE BATCH, FOUR BENCHMARKS: 0.84–0.88 mean pairwise similarity and 0.92–0.94 hardest-companion similarity characterize dense-caption geometry across four benchmarks.The hardest-companion statistic identifies the especially similar negatives most relevant to retrieval.
  • A.2 ONE BATCH, FOUR BENCHMARKS: 86–99% of in-batch negatives have caption similarity at least 0.8, so default γ=0.5 gives virtually every negative a non-trivial adaptive margin.Benchmark differences mainly concern the prevalence of extreme negatives with ¯G ≥0.9.
  • A.2 ONE BATCH, FOUR BENCHMARKS: Figure A5 identifies the detached, diagonal-masked, scaled caption-similarity matrix as HN-CLIP’s entire mechanism.The reference implementation follows standard Long-CLIP fine-tuning for data loading, optimization, scheduling, and evaluation.

B.2 GRADIENT CONCENTRATION

HN-CLIP concentrates optimization on similar negatives by adding detached caption-similarity margins to logits. Recomputing these margins during training is important because it lets them decay as embeddings separate, whereas frozen margins can over-constrain transfer.

  • B.2 GRADIENT CONCENTRATION: e5 ≈148 gradient-ratio amplification follows from a caption-similarity gap of Δ¯G=0.1 with s≈100 and γ=0.5.The relative repulsion is re-weighted multiplicatively by exp(sγ Δ¯G).
  • B.2 GRADIENT CONCENTRATION: Detached adaptive margins make the positive beat each negative by γ¯Gij before that negative’s loss vanishes.The positive’s direct force is affected only through the softmax normalizer because the diagonal margin is zero.
  • B.2 GRADIENT CONCENTRATION: 3.80/6.80 R@1 separates the default from the frozen variant on the transfer benchmark, where the never-decaying margin performs worst.The authors connect sustained margins with sharper training-distribution boundaries but poorer transfer.
  • B.2 GRADIENT CONCENTRATION: s and γ affect gradient concentration through their product sγ, whereas increasing batch size adds more boosted near-duplicate pairs.These effects operate on different axes because nearly every in-batch negative is similar on the studied benchmarks.

C EXTENDED GRADIENT ANALYSIS

Across training-set fractions and datasets, standard InfoNCE collapses rapidly while the boosted objective preserves optimization signal. The boosted gradients remain strongly compatible with the standard objective’s direction.

  • C EXTENDED GRADIENT ANALYSIS: 5/20/50/100% training-data scales show standard loss reaching the numerical floor within the first epoch, while boosted loss keeps producing signal.The same pattern appears across DOCCI and DCI.
  • C EXTENDED GRADIENT ANALYSIS: 102–106 median gradient-norm ratios favor the boosted objective wherever the standard gradient is nonzero.The ratio peaks between 10^5 and 10^8 across runs.
  • C EXTENDED GRADIENT ANALYSIS: 0.58 median gradient cosine, positive in 98% of measurements, indicates sustained directional alignment with the standard objective.The analysis uses a 10^8-dimensional parameter space, where random directions are orthogonal in expectation.

D.1 TRAINING SETUP AND BACKBONE CHOICE

Experiments use a shared Long-CLIP-L backbone, common effective batch and training budget, and a shared evaluation loop across methods. The setup is designed for long captions and framework-level plug-in comparisons.

  • D.1 TRAINING SETUP AND BACKBONE CHOICE: Long-CLIP-L uses a ViT-L/14 backbone with a positionally stretched 248-token text encoder, while evaluation captions average 107–134 words.The caption lengths exceed standard CLIP’s 77-token window.
  • D.1 TRAINING SETUP AND BACKBONE CHOICE: All methods train for 10 epochs with micro-batch 16, 8× gradient accumulation, and effective batch 128.The shared budget supports the training-efficiency comparison described in Table A3.
  • D.1 TRAINING SETUP AND BACKBONE CHOICE: Figure A8 varies training-set fraction by rows and dataset by columns while displaying losses, gradient norms, ratios, and gradient cosine per cell.The figure extends gradient analysis across data scales rather than defining the main training protocol.
  • D.1 TRAINING SETUP AND BACKBONE CHOICE: Replacing each framework’s global contrastive term with LHN leaves its framework-specific modules and auxiliary losses unchanged.The plug-in study tests Long-CLIP, FineLIP, GOAL, and StructXLIP under this replacement protocol.
  • D.1 TRAINING SETUP AND BACKBONE CHOICE: Recall@K is reported in both retrieval directions using one shared evaluation loop with full-split encoding, L2 normalization, and cosine scoring.The lead is unchanged under last-epoch evaluation, and the Long-DCI R@1 ranking persists under a second seed.

E.1 PLUG-AND-PLAY, ALL FOUR BENCHMARKS

The plug-and-play study shows that adding LHN improves in-domain retrieval across frameworks and datasets, while transfer performance is mixed. The full objective remains strongest overall, with transfer configurations narrowing toward plain fine-tuning.

  • In-domain results: All eighteen in-domain framework–dataset combinations improve in average R@1, with gains increasing with caption length.R@1 improves in 35 of 36 per-direction cases, reaching +23.87 for FineLIP on Long-DCI.
  • Transfer results: Transfer performance on VG→Urban-1K is mixed: FineLIP gains +10.7/+15.7 R@1, GOAL changes by −0.6/+0.7, and other frameworks lose 0.2–6.1 R@1.
  • Ablation: The full objective is strongest overall, whereas transfer configurations compress into a narrow band with plain fine-tuning mildly ahead.

E.3 CROSS-DOMAIN EVALUATION

Cross-domain and reduced-data evaluations show that HN-CLIP generally preserves its advantage beyond the training distribution and remains competitive with substantially less data. Direction-specific results confirm no hidden retrieval asymmetry, while deeper-rank margins are more compressed on easier benchmarks.

  • Cross-domain transfer: HN-CLIP leads every reported column in DCI↔DOCCI and DOCCI→Long-DCI transfer evaluations.It exceeds every baseline’s in-domain DOCCI result when trained on DCI and leads the reverse transfer and harder DOCCI→Long-DCI transfer.
  • Seed replication: All R@1 results reproduce within ±0.4 on average under a second seed, while the largest R@1 deviation is 1.28.Across six metrics, the maximum seed-to-seed difference is 1.44 for HN-CLIP.
  • Sample efficiency: HN-CLIP’s 20% training-data results exceed GOAL’s 100% results on both evaluated benchmarks.The average-R@1 margin over GOAL widens as the data fraction shrinks.
  • Per-direction convergence: HN-CLIP leads in both retrieval directions on all four benchmarks from the first epoch.The split-direction view also exposes a FineLIP Image→Text collapse on DCI at epoch 4.
  • Deeper ranks: At R@5/R@10, margins compress on DOCCI and Urban-1K but remain wide on DCI and Long-DCI.On Long-DCI, HN-CLIP is best in three of four R@5/R@10 columns and ties StructXLIP in the remaining Image→Text R@5 column.

F ADDITIONAL QUALITATIVE RESULTS

Qualitative DOCCI retrievals illustrate HN-CLIP’s behavior in galleries crowded with near-duplicate images. The method retrieves images matching fine-grained caption details more reliably than the strongest baseline in the reported examples.

  • Qualitative retrievals: HN-CLIP resolves discriminative details such as lettering, flower species, and signage in crowded DOCCI galleries.
  • Qualitative retrievals: The strongest baseline retrieves appearance-level look-alikes, placing the ground truth at rank 18–52 in the reported examples.
  • Additional evaluations: The appendix includes full-resolution loss, γ-sweep, cross-domain, seed-replication, and sample-efficiency tables for the reported evaluations.
  • Qualitative retrievals: Figure A10 presents Text→Image retrievals on DOCCI, with green indicating ground truth.
Loading 2608.18521v1…