PaperlayerView exact revision on arXiv

Source-linked AI summary

ERCLIP: Small-Data Adaptation of CLIP for Free-Length Image-Text Retrieval

Hoang-Bao Le, Allie Tran, Binh T. Nguyen, Liting Zhou, Cathal Gurrin

arXiv:2610.29939v1cs.IR

TL;DR

CLIP truncates detailed image descriptions at its 77-token text window, while long-caption retrieval methods often rely on large corpora. ERCLIP adapts CLIP from sentence-level caption training and frozen-CLIP preservation. With 20k pairs and one epoch, it improves long-caption retrieval while keeping CLIP’s inference cost.

  • Problem

    CLIP’s 77-token text encoder truncates detailed descriptions, and long-context methods rely on large long-caption corpora while rarely reporting effects on CLIP’s existing tasks.

    Source: §1 · p. 1

  • Method

    ERCLIP splits long captions into sentences for CLIP encoding and fusion during training, and uses frozen-CLIP preservation while retaining the standard 77-token inference interface.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 4

  • Results

    Long-caption R@1 gains 6.5–11.2 points; preservation recovers 30–42% of ImageNet loss and adds 1.3–2.4 COCO I2T points.

    Source: §4.2 · pp. 6–7, §5 · p. 8

  • Takeaways & Limitations

    ERCLIP fits settings with thousands of long captions and one GPU when a standard CLIP checkpoint and usable short-caption retrieval and zero-shot classification are required.

    Source: §5 · p. 8

    At test time, long captions are still truncated to CLIP’s 77-token context; sentence-wise test encoding is left to future work.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

CLIP's 77-token text encoder truncates detailed image descriptions. We present ERCLIP, a small-data adaptation of CLIP for long-caption image--text retrieval. During training, a long caption is split into sentences that the CLIP text encoder encodes independently; the sentence embeddings are fused by mean pooling or a gated residual Transformer scorer and receive their own contrastive loss. At inference ERCLIP is an unmodified CLIP dual encoder: test captions are still read through the 77-token window, and only training sees the full caption. We find that this fine-tuning erodes CLIP's zero-shot classification, most when the fusion output is not a combination of CLIP embeddings, and add a frozen-CLIP preservation loss on the image and short-caption embeddings. With 20k pairs and one epoch, ERCLIP improves long-caption R@1 over CLIP by 6.5--11.2 points, closing 62--77% of the gap to LongCLIP with 2% of its long captions. Preservation recovers 1.5--2.1 ImageNet zero-shot points (30--42% of the loss) and adds up to 2.4 points of COCO image-to-text R@1, at no inference cost.

1 Introduction

CLIP’s 77-token text window truncates detailed descriptions, while long-context retrieval methods typically use large long-caption corpora. ERCLIP adapts CLIP with 20k pairs and one training epoch, retaining its inference architecture while addressing zero-shot classification costs.

1 Introduction

  • Long-context retrieval methods use large long-caption corpora, and their effects on CLIP’s existing tasks are rarely reported.Classification fine-tuning costs are documented, but not for long-caption retrieval.Source: §1 · p. 1
  • ERCLIP splits long captions into encoder-sized sentences, fuses their embeddings, and aligns the result with images using 20k pairs and one epoch.The sentence pathway is used only during training, leaving CLIP’s inference architecture and cost unchanged.Source: §1 · pp. 1–2
  • The adaptation erodes zero-shot classification, with the cost depending on fusion and growing with training; gated residual fusion and frozen-CLIP preservation are used to limit it.The paper jointly evaluates long-caption retrieval, short-caption retrieval, and zero-shot classification at a fixed budget.Source: §1 · p. 2

2 Related Work

Prior long-caption retrieval methods use expanded context or structured text processing and are trained on large corpora. ERCLIP instead examines sentence-level fusion and preservation under a scarce-data budget.

Long-caption retrieval

  • LongCLIP extends positional embeddings to 248 tokens, while SmartCLIP and TULIP use structured selection or relative positional encodings, and HiMo-CLIP models semantic hierarchy and monotonicity.LongCLIP uses 1M ShareGPT4V captions; the cited methods are trained on large corpora.Source: §2 · p. 2
  • LongCLIP’s ablations found that aligning images with both long and short captions and anchoring short-text features to frozen CLIP were inferior to its own strategy.Source: §2 · p. 2

Preserving pretrained behaviour

  • ERCLIP studies related objectives with sentence-level fusion through a 77-token encoder, image and short-caption anchoring, and 20k pairs for one epoch.Its preservation setting targets long-caption retrieval, unlike prior cited preservation work focused on classification.Source: §2 · p. 2

3 Method

ERCLIP fine-tunes CLIP on image, short-caption, and long-caption triplets, using sentence-level fusion to represent long captions during training. Its objective combines short- and long-caption contrastive losses with frozen-CLIP preservation, while inference retains CLIP’s 77-token interface.

3 Method

  • ERCLIP fine-tunes CLIP’s image and text encoders on image, short-caption, and long-caption triplets.The training graph shares text-encoder weights and discards the sentence pathway, fusion, teacher, and losses at inference.Source: §3 · p. 2, Figure 1 · p. 3

3.1 Sentence-Level Encoding and Fusion

  • Splitting long captions into ordered sentences lets the shared text encoder process each sentence within its token window before fusing their embeddings.Mean pooling weights sentences equally without adding parameters; the Transformer scorer contextualizes sentence embeddings with positional embeddings and weights the original embeddings.Source: §3.1 · p. 3
  • Gated residual fusion starts at mean pooling and moves toward attention pooling only as its scalar gate grows.The scorer reweights original CLIP sentence embeddings; the LSTM ablation instead emits a new vector.Source: §3.1 · p. 3

3.2 Training Objective

  • Training uses symmetric InfoNCE for short- and fused long-caption embeddings, with both contrastive terms sharing the image embedding.The short-caption and long-caption terms are not a triplet loss.Source: §3.2 · p. 4
  • A frozen CLIP copy anchors image and short-caption embeddings, while the long-caption embedding remains free; at inference, captions are still truncated to 77 tokens.The preservation term needs no labels or extra data, and the model retains its backbone’s parameters and latency.Source: §3.2 · p. 4

4 Experiments

Across a fixed one-epoch, 20k-pair evaluation, ERCLIP improves long-caption retrieval while preservation limits—but does not eliminate—the loss in zero-shot classification. The experiments compare retrieval, classification, training length, and ablations across CLIP backbones and training-set sizes.

4 Experiments

  • The experiments use 20k COCO–ShareGPT4V pairs, report four long-caption benchmarks plus COCO retrieval and zero-shot classification, and fix all table results at epoch 1.Each configuration is a single run; differences below about one point between variants are treated as ties.Source: §4 · pp. 4–5

4.1 Long-Caption Retrieval

  • One epoch on 20k pairs raises CLIP’s four-benchmark average by 6.5 (B/32), 7.6 (B/16), and 11.2 points (L/14), closing 62% (B/16) and 77% (L/14) of the CLIP-to-LongCLIP gap.The large-corpus methods are reference results, not same-budget baselines.Source: Table 1 · p. 5, §4.1 · p. 5

4.2 Effect on Basic Tasks

  • Mean and Transformer fusion cost 3.2–6.1 ImageNet points while COCO retrieval improves; the LSTM loses 12.4–14.3 ImageNet points and has the worst long-caption retrieval.Preservation recovers 1.5–2.1 ImageNet points (30–42% of the loss) on every backbone and fusion type, but does not remove the erosion.Source: Table 2 · p. 6, §4.2 · pp. 6–7, Figure 2 · p. 7

4.4 Ablations

  • Raising λp from 0 to 2 raises ImageNet from 58.2 to 60.8, while long-caption and COCO R@1 rise by up to 1.3 points and plateau for λp ≥0.5.As training-set size increases from 1k to 10k pairs, long-caption R@1 rises from 56.3 to 61.9; it stays within 0.4 through 50k, while COCO R@1 rises from 40.8 to 49.3 and ImageNet falls from 63.5 to 59.7. These ablations use a one-epoch schedule that fully decays the learning rate and are not directly comparable to Table 2.Source: Table 3 · p. 7, §4.4 · pp. 7–8

4.3 Training Length

  • From epoch 1 to 5, long-caption R@1 falls by 0.4–1.2 points for every variant while COCO R@1 rises by 0.5–1.2 points on B backbones and is flat on L/14.Preservation limits the accompanying CIFAR-10 decline but not the loss of long-caption alignment; longer-budget schedules and other training configurations were not tested.Source: §4.3 · p. 7, Figure 3 · p. 8

5 Discussion and Conclusion

With 20k pairs and one epoch, ERCLIP improves long-caption retrieval without changing CLIP’s inference, while preservation recovers part of the ImageNet loss. It is suited to scarce-data, single-GPU settings; the paper notes several evaluation and scope limitations.

5 Discussion and Conclusion

  • 6.5–11.2 points of long-caption R@1 gain over CLIP, with preservation recovering 30–42% of the ImageNet loss.Inference cost is unchanged.Source: §5 · p. 8
  • ERCLIP fits settings with thousands of long captions and one GPU, such as a specialised domain, when a standard CLIP checkpoint is required.With a million long captions and several GPUs, LongCLIP or SmartCLIP is the better choice; they read full captions and retrieve better.Source: §5 · p. 8
  • Test captions remain truncated to 77 tokens, and sentence-wise inference is not evaluated.The paper also reports single-seed results, no baseline retrained on the same 20k pairs, general-domain data only, and an inferred rather than measured link between drift and zero-shot loss.Source: §5 · p. 8

Related papers

Loading 2610.29939v1…