PaperlayerView exact revision on arXiv

Source-linked AI summary

No Tokens Wasted: Leveraging Long Context in Biomedical Vision-Language Models

Min Woo Sun, Alejandro Lozano, Javier Gamazo Tejero, Vishwesh Nath, Xiao Xiao Sun, James Burgess, Yuhui Zhang, Kun Yuan, Robert Tibshirani, Sean Huver, Serena Yeung-Levy

arXiv:2510.03978v1cs.CVcs.CL

TL;DR

Biomedical VLMs use short text windows, while long biomedical captions and article context can exceed what those windows capture. This paper studies longer-context pretraining, introduces enriched captions and long-text benchmarks, and reports better retrieval and classification with longer context. The findings support long-context modeling as a promising direction for biomedical VLMs.

  • Problem

    Biomedical VLMs typically use short text windows, and existing captions may omit visual details found in article context, leaving long-caption pretraining underexplored.

    Source: §1 · pp. 1–2

  • Method

    The work introduces 1M context-enriched BIOMEDICA-LongCAP pairs and trains BMC-LongCLIP with text context windows up to 512 tokens.

    Source: §1 · p. 2

  • Results

    Longer context improves retrieval and classification; BMC-LongCLIP achieves up to 30 point absolute gains in Recall@1 and 2 point average improvements in classification accuracy.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The results establish long-context modeling as a promising direction for advancing biomedical VLMs.

    Source: §5 · p. 5

    Long-text benchmarks remain scarce across biomedical domains, limiting fuller evaluation of long-context models.

    Source: §5 · p. 5

Abstract

from arXiv · show

Embedding vision-language models (VLMs) are typically pretrained with short text windows (<77 tokens), which forces the truncation of long-format captions. Yet, the distribution of biomedical captions from large-scale open source literature reveals that a huge portion of captions far exceed 77 tokens. To this end, we investigate the impact of pretraining on long-format biomedical captions by extending the context length of text encoders in VLMs. We find that longer context (thus, enabling additional supervision provided in long-format captions) correlates with better retrieval and classification performance. Given this finding, we introduce BIOMEDICA-LongCAP, a dataset of 1M image-caption pairs enriched with context-aware descriptions from full-text articles, providing longer and additional textual supervision. Using BIOMEDICA-LongCAP, we train BMC-LongCLIP, a long-context biomedical VLM with a text encoder supporting windows of up to 512 tokens. Our model extends context capacity by 6.6x, reducing token waste from 55% to just 2.2%. On long-caption retrieval benchmarks, BMC-LongCLIP achieves up to +30% absolute gains in Recall@1 and +2% average improvements in classification, while also converging faster than short-context. Our results demonstrate that long-context modeling is a promising direction for advancing biomedical VLMs.

1. Introduction

Biomedical VLMs typically use short text windows, limiting how much caption information they can process, while biomedical captions also omit visual details embedded in article context. This work investigates longer-context pretraining, introduces enriched captions and long-text benchmarks, and reports improved retrieval and classification with longer context.

1. Introduction

  • Typical 77-token windows truncate long biomedical descriptions, discarding valuable information and leaving more than 434 million BIOMEDICA tokens unused.The paper notes that biomedical caption semantics and complexity can exceed this short context.Source: §1 · pp. 1–2
  • Existing biomedical image captions can omit critical visual details embedded in inline manuscript references, leaving the effect of highly descriptive caption pretraining largely unexplored.These omitted details may include figure analysis in the corresponding scientific manuscript.Source: §1 · p. 2
  • BIOMEDICA-LongCAP contributes 1M biomedical image–caption pairs enriched with contextual information from source text through LLM-based augmentation.The work also introduces two biomedical benchmarks for long-text multimodal retrieval.Source: §1 · p. 2
  • BMC-LongCLIP is pretrained at 77, 154, and 512 tokens to study how context length affects convergence and downstream zero-shot performance.The paper investigates pretraining biomedical multimodal embedding models with long captions.Source: §1 · p. 2
  • Longer context improves retrieval and classification, with the largest retrieval gains on PMC; BMC-LongCLIP reports up to 30 point absolute gains in Recall@1 and 2 point average improvements in classification accuracy.The model also converges faster than short-context baselines.Source: §1 · p. 3

2. Methods

The methods construct context-enriched biomedical captions and pair a long-context text encoder with a pretrained vision encoder. Evaluation uses long-text retrieval benchmarks, biomedical zero-shot classification datasets, and comparisons with prior biomedical VLMs.

2. Methods

  • BIOMEDICA-LongCAP contains 1M image–caption pairs enriched with article context, with a VLM-based pipeline retaining only caption features visually supported by each image.Context includes inline mentions, abstract text, and acronym expansions; average caption lengths are 127 tokens for BIOMEDICA-6M and 323 for BIOMEDICA-LongCAP.Source: §2 · p. 3
  • BMC-LongCLIP pairs a DFN-2B-pretrained ViT-L/14 CLIP vision encoder with Bio-Clinical ModernBERT, pretrained on 53.5B biomedical tokens with an 8,192-token context window.The vision and text encoders have 304M and 150M parameters, respectively.Source: §2 · p. 3

2.1. Benchmarks

  • The CXR benchmark pairs 1,000 chest X-ray images with full radiology reports, while the PMC benchmark uses long captions from 1,000 recent PMC-OA articles.The PMC captions concatenate inline references with figure captions.Source: §2.1 · p. 3
  • Zero-shot classification is evaluated across 39 benchmarks spanning biology, radiology, dermatology, and pathology.Source: §2.1 · p. 4

2.2. Baselines

  • The comparison includes PMC-CLIP, BiomedCLIP, MedSigLIP, and BMC-CLIP as biomedical VLM baselines.Source: §2.2 · p. 4

3. Experiments

The experiments vary text context length under otherwise identical training settings, compare two global batch sizes, and add a model trained with the enriched LongCAP dataset.

3.1. Context-length ablation

  • Models are trained with 77-, 154-, and 512-token context windows while batch size, learning rate, optimizer, and epochs are held fixed.Source: §3.1 · p. 4

3.2. Batch size and BIOMEDICA-LongCAP

  • The study investigates the effect of scaling training batch size while holding other settings fixed; the cited passage establishes the study design but does not state particular batch-size values.Source: §3.2 · p. 4
  • BMC-LongCLIP+ is trained on BIOMEDICA-6M and the 1M-pair BIOMEDICA-LongCAP dataset with a 16K batch and 512-token context window.LongCAP captions are enriched using full-text context.Source: §3.2 · p. 4

4. Results

Longer text context improves retrieval and training efficiency, with BMC-LongCLIP outperforming prior biomedical VLMs on long-text benchmarks and improving zero-shot classification. Results vary by domain and are strongest when long context is paired with enriched captions.

4.1. Context-length ablation

  • Longer context consistently improves retrieval across recall levels and accelerates convergence, with the largest retrieval gains on PMC and steadier, more modest gains on CXR.The authors report that the benefit is especially evident at stricter retrieval thresholds on the text-heavy PMC benchmark.Source: §4.1 · p. 4

4.2. Benchmarking against baselines

  • On CXR, BMC-LongCLIP variants reach 10–14% Recall@10 versus <6% for baselines; on PMC, they achieve 89–95% R@10 and 69–81% R@1.On PMC, the R@10 range is on par with or slightly better than BiomedCLIP (91–94%) and above MedSigLIP (46–60%); at R@1, BMC-LongCLIP surpasses BiomedCLIP (69–73%) and MedSigLIP (20–31%).Source: §4.2 · pp. 4–5
  • BMC-LongCLIP (8K) achieves 50.2% average accuracy across six biomedical domains, versus 41.9% for BiomedCLIP and 36.6% for MedSigLIP.Table 2 reports average accuracy per domain and includes ablations in Panel A and prior-model comparisons in Panel B.Source: §4.2 · p. 5, Table 2 · p. 5

4.3. Effect of batch size and long-caption training

  • Doubling batch size underperforms in microscopy and dermatology, while BMC-LongCLIP+ (16K) with BIOMEDICA-LongCAP recovers the drop and matches or exceeds the 8K model.Microscopy remains comparatively weak and warrants further investigation.Source: §4.3 · p. 5

5. Conclusion

The conclusion identifies scarce long-text benchmarks across biomedical domains as a boundary on evaluation. BIOMEDICA-LongCAP enriches captions with article context and refines them to retain image-supported details, while expanding acronyms for readability.

5. Conclusion

  • Scarcity of long-text benchmarks across biomedical domains limits fuller evaluation of long-context models.The authors identify expansion of these resources as essential for a fuller evaluation.Source: §5 · p. 5

Appendix A. BIOMEDICA-LongCAP details

  • BIOMEDICA-LongCAP augments original figure captions with context from the article, including inline mentions, abstract text, and acronym information.The augmented caption is generated using only the collected information.Source: Appendix A · p. 7
  • A feasibility assessment checks whether each caption feature is discernible from the image alone, labeling features FEASIBLE or NOT FEASIBLE with a rationale.Information not visually present is excluded unless explicitly overlaid as feasibility text.Source: Appendix A · p. 7
  • Caption refinement preserves FEASIBLE features and removes or rewords NOT FEASIBLE features so the description reflects visually supported information.The refinement is based on the feasibility assessment.Source: Appendix A · p. 7
  • The pipeline explicitly expands acronyms using a curated list derived from the full-text article to make captions readable and unambiguous.Source: Appendix A · p. 7

Appendix B. CXR Benchmark

The CXR retrieval benchmark pairs chest radiographs with full-text reports and treats each pair as a one-to-one ground-truth match. Retrieval uses normalized image and text embeddings, while the evaluated reports average 168.3 tokens.

  • Each chest radiograph is paired with its full-text report, and each image–report pair is treated as a one-to-one ground-truth match.Source: Appendix B · p. 7
  • Text and image encoders produce L2-normalized embeddings; retrieval ranks candidates by cosine similarity and reports Recall@1, @5, @10, and @100 in both directions.Text-to-image and image-to-text retrieval are defined analogously.Source: Appendix B · p. 7
  • The 1,000 evaluation reports average 168.3 tokens, with a median of 158 and lengths ranging from 49 to 427 tokens.Source: Appendix B · p. 7

Appendix C. PMC Benchmark

The PMC benchmark samples distinct image–caption pairs from recent full-text articles, while the zero-shot classification benchmark reformulates dataset tasks as multiple-choice image–text similarity evaluations.

Appendix C. PMC Benchmark

  • The benchmark uses recently published 2025 biomedical articles from PubMed Central, downloading media bundles with .nxml full-text and associated images, and samples exactly one unique image–caption pair per article without reusing articles.Source: Appendix C · p. 7
  • The 1,000 CXR reports average 510 tokens, with a median of 460 and a range of 251 to 1,022 tokens.Source: Appendix C · p. 7

Appendix D. Zero-shot classification benchmark

  • In the zero-shot classification benchmark, labels become short text descriptions and each image is paired with shuffled multiple-choice answers, including distractors.Evaluation computes similarity between image and answer embeddings.Source: Appendix D · p. 7

Appendix E. Compute details

Appendix E identifies a table of compute resources used for training.

  • Table 3 lists the compute resources used for training.Source: Table 3 · p. 8

Appendix F. Training parameters

Appendix F identifies a table of hyperparameters used for pretraining.

  • Table 4 lists the hyperparameters used for pretraining.Source: Table 4 · p. 8

Appendix G. Training loss curves by training context length

Figure 3 compares training loss curves across context lengths and illustrates faster convergence with longer text windows.

  • Longer text windows accelerate convergence, as illustrated by training loss curves across context lengths.Source: Figure 3 · p. 8
Loading 2510.03978v1…