Source-linked AI summary
From Generalist to Specialist: A Context-Fusion Framework for Endoscopic Polyp Reporting with a Frozen VLM
Ruijie Yang, Yan Zhu, Peiyao Fu, Siyuan Li, Te Luo, Zhihua Wang, Quanlin Li, Pinghong Zhou, Xian Yang, Shuo Wang
TL;DR
Reliable endoscopic polyp reporting must combine lesion sizing, Paris classification, and morphological description, but existing approaches do not introduce specialist knowledge into VLMs while preserving their pretrained capabilities. This paper fuses retrieved image-report evidence and learned specialist guidance around a frozen VLM, achieving the strongest overall balance among evaluated methods with only 0.006% as many trainable parameters as the frozen model.
Problem
Reliable automated reporting must integrate lesion size, Paris type, and morphology, while existing specialization does not preserve pretrained VLM capabilities and a unified generative interface.
Method
The framework fuses query-specific retrieved image-report pairs with learned specialist tokens to guide a frozen VLM in producing unified structured polyp reports.
Results
The framework achieved the strongest overall balance among evaluated methods, using trainable parameters equal to only 0.006% of the frozen VLM.
Takeaways & Limitations
Within the evaluated backbone and dataset, context fusion supports lightweight specialist adaptation of a frozen VLM for endoscopic polyp reporting.
Takeaways & Limitations
The findings do not establish cross-backbone generality because all adaptation experiments used a single frozen VLM.
Abstract
from arXiv · showhide
Reliable endoscopic polyp reporting requires integrating quantitative lesion sizing, standardized Paris classification, and clinically meaningful morphological description within a single record. General-purpose vision-language models (VLMs) offer a unified interface for image understanding and report generation. Existing specialization strategies, however, typically rely on task-specific models or model-weight adaptation, leaving unresolved how to introduce reliable specialist knowledge while preserving both this unified interface and the VLM's pretrained capabilities. We introduce a context-fusion framework that specializes a frozen general-purpose VLM through both implicit instruction context and explicit transduction context without modifying its pretrained weights. Specifically, a self-supervised polyp encoder retrieves related image-report pairs as explicit, query-specific evidence, while learned continuous specialist tokens provide implicit instruction context shared across cases. Experiments were conducted on 2,056 expert-annotated public endoscopic images. We compared the framework with general-purpose VLMs, task-specific predictors, and weight-adaptation methods to assess specialist performance, unified reporting, and adaptation efficiency. Across numerical, categorical, and report-generation metrics, the proposed framework substantially improved direct frozen-VLM inference and achieved the strongest overall performance among the evaluated methods. It added trainable parameters equal to only 0.006% of the frozen VLM's parameter count. When the top-1 retrieved case carried the correct target category, our framework corrected 70.5% of the errors made by a weight-adaptation baseline. These findings support the context-fusion framework as a lightweight and effective strategy for specialist adaptation of a frozen VLM.
1. Introduction
Reliable automated polyp reporting must combine explicit size and morphology documentation with standardized Paris classification, yet existing modular and end-to-end systems have complementary limitations. The proposed context-fusion framework specializes a frozen general-purpose VLM through retrieved clinical evidence and learned specialist tokens, achieving strong overall performance without changing pretrained weights.
- Introduction: Reliable reporting supports lesion management, surveillance, communication, and quality review while requiring polyp size, morphology, and standardized Paris classification.
- Introduction: Directly prompted general-purpose VLMs lack reliable specialist knowledge and may hallucinate, while weight adaptation can alter pretrained reasoning capacity and response diversity.
- Introduction: The framework combines query-specific retrieved image–report pairs with persistent learned specialist tokens while leaving all pretrained VLM weights unchanged.Explicit transduction context supplies clinical evidence, and implicit instruction context provides reporting guidance across cases.
- Introduction: 2,056 public endoscopic images with expert annotations form the benchmark for comparisons against prompted VLMs, weight-adaptation methods, and modular task-specific predictors.
- Introduction: The proposed context-fusion framework achieves the best overall performance among evaluated strategies while maintaining high efficiency and adding only 0.5 M trainable parameters.
2. Related work
Prior endoscopic reporting methods either assemble reports from task-specific predictions or generate text directly from images, while medical VLM adaptation typically depends on large domain-specific multimodal data or partial model training. Context-engineering methods provide instruction optimization, demonstration selection, or external retrieval, but reliable specialist adaptation on smaller endoscopic datasets remains insufficiently studied.
- Endoscopic reporting: Automated endoscopic reporting broadly uses task-specific prediction with rule-based assembly or direct image-to-text generation.Task-specific systems constrain reports to predefined recognizers and fields, whereas image-to-text systems generate BASIC-based polyp descriptions.
- Endoscopic reporting: BLEU, ROUGE-L, and METEOR were used to evaluate report-generation quality as rigorous assessment became a distinct methodological challenge.
- Medical VLM adaptation: Medical VLMs typically acquire broad capability through large, heterogeneous image–text corpora for continued pretraining and multimodal instruction tuning.For narrower reporting domains, adaptation is usually restricted to part of the model.
- Context engineering: Context engineering spans instruction optimization, demonstration selection, and external knowledge retrieval, including query-specific selection of input–output examples and visual demonstrations.These approaches retrieve textual knowledge or paired clinical evidence, but reliable specialist adaptation from smaller endoscopic reporting datasets remains insufficiently studied.
3. Method
The method specializes a frozen VLM for unified endoscopic polyp reporting by fusing shared learned specialist context with query-specific retrieved evidence, without changing pretrained weights. It generates diameter, Paris-type, and morphology fields in one autoregressive response.
- Problem formulation: The framework formulates specialist reporting as context-conditioned generation while keeping the pretrained VLM parameters fixed.Each query produces diameter, Paris-type, and free-text morphology fields in a single autoregressive response.
- Context fusion: Two complementary contexts specialize the frozen VLM: retrieved image–report pairs provide query-specific evidence, while continuous specialist tokens provide shared conditioning.The composed context generates all three report fields in one response and separates adaptation from VLM weights.
- Explicit transduction context: EndoFinder creates fixed, L2-normalized image representations used to build a case bank for similarity-based retrieval.Retrieved cases preserve each image, its report, and similarity score; paired reports demonstrate mappings to diameter, Paris type, and clinical description.
- Implicit instruction context: Prompt tuning learns a shared sequence of continuous specialist tokens in the frozen VLM’s embedding space.The tokens are optimized from all training reports for numerical, categorical, and descriptive fields, while containing no query-specific facts.
- Context fusion: Context fusion orders query-image tokens, specialist tokens, reporting rules, retrieved evidence, and output schema so the frozen VLM conditions each generated report token on their joint multimodal input.Only the specialist-token parameters are optimized; the VLM, EndoFinder encoder, and case bank remain fixed.
4. Experiments
Across numerical, categorical, and report-generation tasks, context fusion produced the strongest overall specialist performance while preserving a unified reporting interface and requiring far fewer trainable parameters than weight adaptation. Ablations showed complementary benefits from implicit instruction and explicit transduction contexts, with retrieval quality and task-field context affecting performance.
- Overall performance: The unified framework outperformed task-specific predictors, with 0.200 versus 0.306 diameter MAE and 0.698 versus 0.657 Paris accuracy against supervised ViT models.Unlike the single-task predictors, it generated diameter, Paris type, and morphology in one response.
- Overall performance: 0.200 versus 0.203 diameter MAE, 0.698 versus 0.664 Paris accuracy, and 0.858 versus 0.843 ROUGE-L gave Ours a stronger overall balance than Finetune.LoRA reached 0.226 MAE, 0.650 Paris accuracy, and 0.826 ROUGE-L.
- Adaptation efficiency: 0.524 M trainable parameters were optimized by Ours, compared with 9.056 M for LoRA and 40.119 M for Finetune.LoRA and Finetune therefore required 17.3 and 76.6 times as many trainable parameters, respectively.
- Adaptation efficiency: 70.5% of Finetune’s errors were corrected by Ours when the top-1 retrieved report contained the target Paris type.This comprised 43 corrected cases among 61 eligible errors; Ours changed 14 Finetune-correct predictions to errors among 201 cases, or 7.0%.
- Context ablation: Combining implicit and explicit contexts reduced diameter MAE to 0.200, increased Paris precision, recall, and macro-F1 to 0.791, 0.716, and 0.744, and raised BLEU-4, ROUGE-L, and CIDEr to 0.730, 0.858, and 0.491.The individual contexts also improved all three tasks, with explicit transduction context generally providing the stronger single-context report-generation result.
- Retrieval analysis: EndoFinder’s Paris hit@K increased from 0.637 at K=1 to 0.932 at K=7, while macro-F1 increased from 0.699 to 0.758 and CIDEr from 0.428 to 0.489 between K=1 and K=3.Oracle retrieval at K=3 achieved 0.901 Paris macro-F1 and 0.130 diameter MAE, compared with 0.758 and 0.204 for EndoFinder, indicating remaining evidence-ranking headroom.
5. Discussion
The discussion frames specialist adaptation as context organization rather than VLM weight modification, with retrieved clinical precedents and learned specialist tokens providing explicit and implicit context. It also identifies task-composition effects, hard-case benefits, and limitations that motivate broader validation.
- Context-fusion interpretation: Context organization replaces VLM weight modification: explicit transduction context supplies query-specific clinical precedents, while implicit instruction context provides a learned compact condition.Retrieved cases remain inspectable as explicit evidence for each prediction.
- Task composition: Jointly requesting diameter and Paris type improved morphology generation, although their text was excluded from report scoring.The discussion interprets quantitative and categorical predictions as intermediate clinical constraints during inference.
- Hard-case analysis: 70.5% of Finetune errors were corrected when the top-ranked retrieved report contained the target Paris type, while 7.0% of corresponding Finetune-correct predictions became errors.This imbalance indicates that relevant precedents can aid difficult categorical decisions without broadly destabilizing already-correct predictions.
- Limitations: The results remain limited by uncertain monocular diameter estimation, a 2,056-image structured subset, under-represented rare Paris types, and evaluation on a single frozen VLM.The current schema also omits report fields beyond diameter, Paris type, and morphology.
- Future work: Future work should test larger calibrated cohorts, broader Paris-type coverage, external clinical validation, other frozen VLM backbones, and field-aware or conflict-aware retrieval.These directions are intended to assess generality and improve evidence selection across report fields.
6. Conclusion
The paper presents context fusion as a lightweight strategy for adapting a frozen VLM to endoscopic polyp reporting. The framework combines query-specific image–report pairs with learned specialist guidance and achieves the strongest overall balance among evaluated methods with minimal trainable parameters.
- Conclusion: The framework adapts a frozen VLM by combining query-specific image–report pairs with learned specialist guidance.This context-fusion design targets specialist endoscopic polyp reporting without modifying the frozen VLM.
- Conclusion: It jointly estimates lesion diameter, classifies Paris type, and generates morphology descriptions.
- Conclusion: 0.006% of the frozen VLM’s parameters are trainable, while the framework achieves the strongest overall balance among evaluated methods.Within the evaluated backbone and dataset, the results support context fusion as a lightweight approach to specialist VLM adaptation.
Data availability
The study’s annotations are publicly available through the EndoVL dataset on Figshare, using its 2,056-image polyp-reporting subset.
- Data availability: The annotations are available through the EndoVL dataset on Figshare at DOI 10.6084/m9.figshare.32320998.v1.The study uses the 2,056-image polyp-reporting subset defined in Section 4.1.