PaperlayerView exact revision on arXiv

Source-linked AI summary

EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgery

Guankun Wang, Long Bai, Junyi Wang, Kun Yuan, Zhen Li, Tianxu Jiang, Xiting He, Jinlin Wu, Zhen Chen, Zhen Lei, Hongbin Liu, Jiazheng Wang, Fan Zhang, Nicolas Padoy, Nassir Navab, Hongliang Ren

arXiv:2501.11347v2cs.CV

TL;DR

Surgical training needs flexible guidance, but existing approaches do not fully address diverse trainee queries or the visual-domain gap in surgical scenes. EndoChat combines a multiparadigm surgical dataset with enhanced visual representation and contrast-based reasoning, and the authors report strong performance across surgical dialogue and understanding tasks alongside positive expert feedback.

  • Problem

    Surgical training needs flexible support for diverse trainee queries, while surgical MLLMs face visual-domain gaps that can impair scene understanding and reasoning.

    Source: §1 · pp. 2, 4, §3.2.3 · p. 18

  • Method

    EndoChat is trained on Surg-396K and combines multiscale visual-feature extraction with visual contrast between original and distorted inputs.

    Source: §1 · p. 4, §3.2.3 · p. 18, §5 · p. 28

  • Results

    EndoChat achieves state-of-the-art performance across surgical scene-understanding subtasks and outperforms general-purpose and medical MLLMs across dialogue paradigms, with most surveyed surgeons giving positive feedback.

    Source: §1 · pp. 5–6, §5 · p. 28

  • Takeaways & Limitations

    Expert feedback and the reported evaluations support EndoChat’s practical applicability as an assistant for surgical training and education.

    Source: §5 · p. 28, §6 · p. 30

    The relatively small number of unique surgical cases may hinder generalization to different surgical techniques.

    Source: §5 · p. 29

Abstract

from arXiv · show

Recently, Multimodal Large Language Models (MLLMs) have demonstrated their immense potential in computer-aided diagnosis and decision-making. In the context of robotic-assisted surgery, MLLMs can serve as effective tools for surgical training and guidance. However, there is still a lack of MLLMs specialized for surgical scene understanding in clinical applications. In this work, we introduce EndoChat to address various dialogue paradigms and subtasks in surgical scene understanding that surgeons encounter. To train our EndoChat, we construct the Surg-396K dataset through a novel pipeline that systematically extracts surgical information and generates structured annotations based on collected large-scale endoscopic surgery datasets. Furthermore, we introduce a multi-scale visual token interaction mechanism and a visual contrast-based reasoning mechanism to enhance the model's representation learning and reasoning capabilities. Our model achieves state-of-the-art performance across five dialogue paradigms and eight surgical scene understanding tasks. Additionally, we conduct evaluations with professional surgeons, most of whom provide positive feedback on collaborating with EndoChat. Overall, these results demonstrate that our EndoChat has great potential to significantly advance training and automation in robotic-assisted surgery.

1. Introduction

Surgical trainees need timely guidance, yet surgeons’ workloads limit consistent real-time support and existing systems struggle with diverse, open-ended interaction. EndoChat addresses this with a surgical instruction dataset and visual mechanisms, and the authors report strong task performance and positive expert feedback.

1. Introduction

  • Surgeons’ clinical, teaching, and research workloads make consistent real-time trainee guidance difficult, motivating flexible support during training.Existing structured VQA systems also struggle with open-ended questions and complex multi-turn dialogue, while surgical MLLMs face diverse-query and visual-domain-gap challenges.Source: §1 · pp. 2–4
  • EndoChat outperforms general-purpose and medical MLLMs across dialogue paradigms and surgical scene-understanding subtasks, with experienced endoscopists providing positive feedback.The paper also reports state-of-the-art performance across attribute-related subtasks and positive evaluations of its training and education potential.Source: §1 · pp. 5–6
  • EndoChat is designed to support diverse conversational paradigms in endoscopic surgery, and its evaluations include comparisons with commercial and open-source MLLMs.The authors report gains in surgical understanding accuracy and dialogue capability, state-of-the-art attribute-subtask performance, and positive evaluations by experienced surgeons.Source: §1 · pp. 5–6
  • Surg-396K contains 396K image-instruction pairs, built by systematically extracting surgical information and generating structured annotations through a multi-conversation pipeline.The dataset defines five conversation paradigms and seven attribute-specific surgical subtasks.Source: §1 · pp. 4–5
  • The Mixed Visual Token Engine extracts and fuses visual information at multiple scales, while visual contrast compares outputs from original and distorted inputs to mitigate hallucinations.The contrast method applies adaptive plausibility constraints to refine token selection and improve consistency between visual inputs and language outputs.Source: §1 · pp. 5–6

2. Related work

Related work traces multimodal language models from general architectures and medical assistants to surgical dialogue systems. The review identifies predefined interaction formats and insufficient adaptation to surgical visual data as persistent limitations.

2.1. Multimodal Large Language Models

  • General MLLMs combine a visual encoder, text tokenizer, alignment module, and LLM; medical applications span diagnostic and reporting tasks, but surgical use remains challenged by scarce high-quality annotations and domain shift.The review also notes large-scale vision-language paired data as important for MLLM training.Source: §2.1 · pp. 6–7

2.2. Surgical Vision-Language Models

  • Earlier surgical VQA systems represent surgical entities textually, with later methods adding localization or template-generated dialogue, but structured data and encoder-decoder designs limit queries to predefined content.This restricts complex interactions and clinical applicability.Source: §2.2 · pp. 7–8
  • Surgical MLLMs extend instruction tuning, add localization or memory mechanisms, unify tasks, or build conversational datasets, yet most still rely on generic captioning or predefined tasks.The review identifies real-world dialogue paradigms as an overlooked requirement for surgeon interactions.Source: §2.2 · p. 8

3.1. Surgical Multimodal Instruction Dataset: Surg-396K

Surg-396K is an endoscopic-surgery instruction dataset built to support diverse dialogues and broad surgical scene understanding. Its construction combines structured attribute extraction, conversation generation, and data cleaning; the authors report greater scale and annotation diversity than existing benchmarks.

3.1. Surgical Multimodal Instruction Dataset: Surg-396K

  • Surg-396K contains 41K images and 396K instruction-following annotations, organized into five conversation paradigms and seven attribute-related surgical subtasks.The paradigms target varied natural-language dialogue scenarios, while the subtasks aim for comprehensive surgical scene-understanding coverage.Source: §3.1 · p. 9

3.1.1. Preliminary for Constituent Datasets

  • Surg-396K integrates EndoVis-VQLA, CoPESD, and Cholec80-VQA; the first two supply GPT-4V-expanded data, while Cholec80-VQA contributes directly to two conversation types.Cholec80-VQA is used directly because its annotations lack grounding information and contain less content per image.Source: §3.1.1 · pp. 9, 11
  • The dataset pipeline proceeds through attribute analysis, information extraction, instruction-tuning data generation, diverse conversation generation, and data cleaning.The pipeline overview also presents distribution statistics for Surg-396K.Source: Figure 2 · p. 10

3.1.2. Attribute Retrieval

  • Attribute retrieval organizes surgical information into Observation, Operation, and Analysis, from instrument and target attributes to motion, spatial relations, and scene descriptions.The framework also uses attribute-specific QA templates to generate instruction-tuning data.Source: §3.1.2 · p. 11

3.1.3. Diverse Conversation Generation

  • Five conversation types vary response scope and grounding, from concise Single Phrase answers to contextual Visual QA, region-focused answers, and bounding-box Grounding QA.Detailed Description is generated by GPT-4V to cover surgical-image attributes; Visual QA expands short answers into sentences, while Grounding QA responds only with bounding boxes.Source: §3.1.3 · pp. 12–13

3.1.4. Surgical Sub-task Formulation

  • Seven attribute-related subtasks assess surgical understanding from basic visual observations to higher-level analysis, drawing mostly on Single Phrase QA and Grounding QA and using Detailed Description for description tasks.Grounding QA supplies spatial annotations, while Single Phrase QA captures concise attributes such as instrument types, counts, and basic spatial relations.Source: §3.1.4 · pp. 13–14

3.1.5. Data Cleaning

  • Data cleaning samples one-fifth of the generated data for manual checks of completeness, relevance, and semantic clarity, then applies documented fixes to non-sampled content.Reviewers assess whether text includes essential operations and tools, matches its conversation-type label, and is semantically clear and accurate.Source: §3.1.5 · p. 14

3.1.6. Comparison with Existing Surgical Scene Understanding Datasets

  • 41.4K images and 396K annotations give Surg-396K greater scale and annotation diversity than existing benchmarks, with multiple procedures and multimodal formats.The comparison includes earlier and more recent surgical scene-understanding datasets; the authors characterize Surg-396K as a more comprehensive and versatile benchmark.Source: §3.1.6 · pp. 14–15, Table 2 · p. 15

3.2. Visual Enhanced MLLM: EndoChat

EndoChat combines a mixed visual encoder and token engine with visual-contrast reasoning to support visual grounding conversations in endoscopic surgery.

3.2. Visual Enhanced MLLM: EndoChat

  • EndoChat processes visual grounding conversations by encoding image tokens, enhancing them with the Mixed Visual Token Engine, aligning them with language tokens, and passing them to a language model.The model also uses visual contrast to mitigate object hallucinations.Source: §3.2 · p. 15

3.2.1. Preliminary for EndoChat

  • SPHINX combines multiple vision encoders and visual embeddings, using multi-scale sub-images and multi-task training for visual instruction-following tasks.EndoChat builds upon SPHINX and its LLaMA-2 base.Source: §3.2.1 · pp. 15–16

3.2.2. Mixed Visual Token Engine

  • MVTE generates global visual tokens from mixed-encoder features, using contextual attention and concatenation to retain multi-scale visual information.EndoChat processes five images each at 224×224 and 512×512 resolutions, encoded by DINOv2 and OpenCLIP.Source: §3.2.2 · p. 16

3.2.3. Hallucination Mitigation through Visual Contrast

  • Visual contrast compares output distributions from original and Gaussian-noise-distorted images to counter language priors and object hallucinations.An adaptive constraint refines the contrastive distribution to avoid punishing valid outputs and to preserve plausible content.Source: §3.2.3 · pp. 18–19

3.2.4. Training Strategy

  • EndoChat uses LLaMA2-13B with task-specific prompts and LoRA fine-tuning, optimizing low-rank matrices rather than all LLM parameters.The paper states that this reduces training time and computational overhead while preserving broader generic-object and spatial-landmark knowledge.Source: §3.2.4 · p. 19

4. Results

EndoChat outperforms comparison models across reported surgical scene-understanding tasks, and endoscopists gave positive assessments in a 150-case evaluation.

4.1. Implementation Details

  • The model was trained for one epoch on Surg-396K using four NVIDIA A800 GPUs, with training completed in approximately 20 hours.The comparison set included general MLLMs and specialized models.Source: §4.1 · pp. 19–20

4.2. EndoChat tailors interactions to varying complexity levels of surgical conversation.

  • 55.51% accuracy on EndoVis-17 Single Phrase QA exceeded the 0% accuracy of other models; EndoChat also outperformed all MLLMs on Detailed Description.For EndoVis-17, EndoChat also achieved F-score 29.78, AP@50 90.25, and mIoU 86.62.Source: §4.2 · pp. 20–21, Table 3 · p. 21, Table 6 · p. 23

4.3. EndoChat enhances interactions through multiple modalities

  • EndoChat achieved the highest mIoU on EndoVis-18-VQLA (86.89) and EndoVis-17-VQLA (86.62), and outperformed other models in Region Based QA.Grounding QA uses bounding-box responses, while Region Based QA takes a user-provided box as input.Source: §4.3 · pp. 21–22, Table 4 · p. 22, Table 6 · p. 23

4.4. EndoChat is a surgeon-like interaction tool

  • On Cholec80-VQA, EndoChat achieved the highest scores across all Visual QA evaluation metrics, surpassing zero-shot MLLMs and specialized models.The reported results also show advantages on other Surg-396K parts, including BLEU-4 and METEOR.Source: §4.4 · pp. 23–24, Table 5 · p. 23

4.5. EndoChat advances comprehensive understanding of surgical scenerios

  • EndoChat led instrument counting at 85.14% accuracy, versus BiomedGPT at 78.84% and LLaVA-Med at 49.88%, and reached 39.88% object-localization accuracy.It also achieved the highest accuracy and F-scores for motion recognition and direction prediction and surpassed other models in instrument-category and target-tissue recognition.Source: §4.5 · pp. 24–25, Table 7 · p. 25

4.6. Ablation study for the effectiveness of core modules in EndoChat

  • MVTE raised Single Phrase accuracy from 66.74% to 71.47% and F-score from 33.75% to 43.74%; hallucination mitigation improved Visual QA and Region Based QA metrics.For Region Based QA, CIDEr rose from 5.4069 to 5.5735 and ROUGE-L from 80.83 to 81.21.Source: Table 8 · p. 25, §4.6 · p. 26

4.7. Expert evaluation of EndoChat by endoscopists

  • In 150 cases, endoscopists rated 74.7% of descriptions correct, found 76.6% of analyses useful, and agreed training could be aided in 72%.69.3% of cases reflected willingness to use EndoChat in surgical training.Source: §4.7 · pp. 26–27, Figure 4 · p. 28

5. Discussion

The discussion reports strong performance across dialogue paradigms and surgical scene-understanding tasks, alongside positive surgeon feedback. It explains the design priorities and identifies limits involving case diversity, compute, and clinical-data ethics.

5. Discussion

  • EndoChat is presented as a surgical chatbot and copilot built on multiparadigm data and enhanced visual representation and inference strategies.The stated aim is surgical education and training through multimodal understanding and dialogue.Source: §5 · p. 28
  • Experienced endoscopists provided positive feedback in most evaluation cases, while EndoChat performed strongly across dialogue paradigms and surgical scene-understanding tasks.The authors also report ablations of visual feature learning and visual contrast-based reasoning.Source: §5 · p. 28
  • The design prioritizes downstream task accuracy through single-phrase QA and surgeon-oriented dialogue paradigms for varied questions and contexts.The authors say short answers can help link visual information with text annotations, while paradigms constrain divergent MLLM dialogue and focus responses on surgeons’ questions.Source: §5 · p. 29
  • A relatively small number of unique cases may hinder generalization to different surgical techniques; substantial compute also complicates edge deployment, and clinical-data privacy and ethics require study.The authors identify expanding procedures and cases, resource-constrained deployment, and privacy and ethical use as future concerns.Source: §5 · p. 29

6. Conclusion

The conclusion presents Surg-396K and EndoChat as a dataset-and-model contribution for multiparadigm surgical scene understanding. The authors report visual-feature and hallucination-mitigation mechanisms, positive expert feedback, and plans for further validation and integration.

6. Conclusion

  • The paper introduces Surg-396K, a comprehensive surgical multimodal dataset that includes 396K image-instruction pairs.Source: §6 · p. 29
  • Based on Surg-396K, EndoChat combines the Mixed Visual Token Engine with visual contrast-based hallucination mitigation for surgical scene understanding and dialogue.The authors say these mechanisms capture visual features and reduce inaccuracies in generated responses.Source: §6 · p. 30
  • The authors plan to open-source model weights, training code, and data, conduct broader clinical validation, and integrate EndoChat into surgical training or endoscopic surgery systems.They envision monitor- and voice-based assistance for surgeons or trainees.Source: §6 · p. 30

9. Author contributions statement

The author-contributions statement assigns work spanning conceptualization, methods, validation, investigation, dataset generation, evaluation, software, analysis, supervision, and writing.

9. Author contributions statement

  • Guankun Wang is credited with conceptualization, methodology, validation, investigation, and original-draft writing; Long Bai is credited with conceptualization, methodology, investigation, and original-draft writing.Source: §9 · pp. 30–31
  • Junyi Wang is credited with validation, investigation, dataset generation, and review and editing; Zhen Li with endoscopist evaluation and review and editing.Source: §9 · p. 31
  • Other listed contributions include software, computing-resource support, formal analysis, data curation, conceptualization, supervision, and writing.These roles are assigned among the remaining named authors in the statement.Source: §9 · p. 31

Related papers

Loading 2501.11347v2…