PaperlayerView exact revision on arXiv

Source-linked AI summary

MeddeX: A Retrieval-Augmented LLaMA-3 System with Gemini Vision for Evidence-Backed Medical Report Analysis

Rajdeep Pandhare, Rohit Kundaikar, Yakub Kojar, Shreyansh Tiwari

arXiv:2610.20632v1cs.CLcs.CV

TL;DR

Complex medical reports create time and interpretive burdens for clinicians, while existing LLM and RAG systems have gaps in long-document handling, evidence linking, domain adaptation, and multimodal inputs. MeddeX combines a fine-tuned LLaMA-3 model with RAG and a hybrid extraction pipeline. The fine-tuned model consistently outperforms its base model across five metrics, with a reported BLEU gain of +71.4%.

  • Problem

    Clinical report review is burdensome, and existing LLM and RAG solutions have limitations in long-context handling, evidence linking, medical terminology adaptation, and multimodal inputs.

    Source: §I · p. 1

  • Method

    MeddeX combines QLoRA fine-tuned LLaMA-3 generation with RAG and a hybrid pipeline that extracts, chunks, indexes, retrieves, and uses report context to generate answers.

    Source: §I · p. 1, B. Processing Pipeline · pp. 2–3

  • Results

    The fine-tuned model consistently outperformed the base model across all five metrics, with BLEU improving by +71.4%.

    Source: B. Quantitative Results · p. 4, D. Analysis · p. 5

  • Takeaways & Limitations

    The system's report-scoped grounding and multimodal handling target reduced cross-report contamination and hallucination risk while supporting scanned, UTF-16-encoded, and radiological inputs.

    Source: E. Discussion · p. 5

    The evaluation dataset is relatively small, and the study lacks a large-scale multi-site clinical trial.

    Source: E. Discussion · p. 6

Abstract

from arXiv · show

Modern healthcare generates increasingly complex medical reports that challenge clinical professionals in terms of time and interpretive burden. Manual review processes are slow, error-prone, and difficult to scale. This paper presents MeddeX, a Medical Report Analyzer that combines a QLoRA fine-tuned LLaMA-3 (8B) model with Retrieval-Augmented Gen eration (RAG) and a multimodal Gemini Vision layer to deliver accurate, concise, and evidence-backed summaries of medical documents. The system handles text-native PDFs, scanned image based PDFs, and direct radiological images through an intelligent hybrid extraction pipeline that includes a novel garbage-text detection mechanism for corrupted and UTF-16-encoded doc uments. Beyond single-report Q&A, MeddeX introduces report scoped retrieval for precise contextual grounding, an automated longitudinal report comparison engine, and a self-healing backfill pipeline that lazily processes legacy documents on demand. A FastAPI backend, Supabase vector store, and Next.js fron tend complete the end-to-end architecture. Evaluation against the base LLaMA3-Med42-8B model demonstrates consistent improvements across ROUGE-1 (+9.2%), ROUGE-2 (+10.0%), ROUGE-L (+11.8%), BLEU (+71.4%), and BERTScore (+0.8%), validating the benefit of domain-specific fine-tuning. MeddeX represents a practical and robust step toward AI-assisted clinical decision support.

I. INTRODUCTION

Manual review of complex medical reports imposes substantial cognitive load, while existing LLM and RAG systems have limitations in long-document handling, evidence linking, medical terminology, and multimodal inputs. MeddeX combines domain-adapted generation with retrieval and multimodal document processing to address these challenges.

  • MeddeX uses a QLoRA fine-tuned LLaMA-3 (8B), based on LLaMA3-Med42-8B, for domain-adapted generation.Source: §I · p. 1
  • Report-scoped dense retrieval grounds generated outputs in evidence from the specific source document.Source: §I · p. 1
  • Gemini Vision supports OCR for scanned PDFs, UTF-16-encoded documents, and direct image inputs, while garbage-text detection reroutes corrupted or hex-encoded extractions.Source: §I · p. 1
  • A longitudinal comparison engine analyzes changes across multiple reports over time.Source: §I · p. 1
  • A lazy backfill pipeline processes legacy documents uploaded before the chunking system was introduced.Source: §I · p. 1
  • The system pairs a FastAPI backend and Supabase vector and document storage with a Next.js frontend.Source: §I · p. 1

II. LITERATURE REVIEW

Prior work spans medical language models, RAG, domain adaptation, and multimodal document analysis, but the literature identifies gaps in grounded and adaptable clinical summarization. The reviewed approaches provide partial solutions rather than a unified account of these challenges.

A. Medical NLP and Summarization

  • BioBERT, ClinicalBERT, and PubMedBERT showed that domain pretraining significantly improves downstream clinical NLP tasks, including named entity recognition, relation extraction, and question answering; BART and Longformer [4] were applied to medical documents.Longformer’s sliding-window attention provides a partial solution to the long-context problem. These models generate summaries without grounding claims in verifiable source passages, limiting clinician trust.Source: A. Medical NLP and Summarization · p. 2

B. Retrieval-Augmented Generation

C. LLM Fine-Tuning for Medical Domains

D. Multimodal Medical Document Analysis

The literature review identifies persistent challenges in processing clinical documents, including multimodal inputs and document integrity. It also describes unmet needs in long-document handling, evidence attribution, domain adaptation, longitudinal analysis, and end-to-end usability.

D. Multimodal Medical Document Analysis

E. Research Gaps

  • Most LLMs degrade beyond their context window, making chunked retrieval necessary, while summaries without source attribution are insufficient for clinical trust.Source: E. Research Gaps · p. 2
  • Generic LLMs may misinterpret subtle clinical terminology and lab-value contexts, motivating domain adaptation.Source: E. Research Gaps · p. 2
  • UTF-16 artifacts or corrupted PDF pages can silently yield garbled text and undermine downstream analysis.Source: E. Research Gaps · p. 2
  • Systems that treat reports in isolation leave a gap in automated comparisons across visits for tracking health trends.Source: E. Research Gaps · p. 2
  • Clinical adoption requires seamless, low-latency interfaces integrated with secure storage.Source: E. Research Gaps · p. 2

III. SYSTEM ARCHITECTURE

MeddeX is organized as a four-layer system with three primary API endpoints and shared document-processing modules. Its pipeline extracts and chunks uploaded files, indexes their content, retrieves report-relevant context, and generates and stores responses.

A. Overview

  • The architecture has Frontend, Backend API, Model, and Storage layers, with /upload_report, /consult, and /compare_reports supported by shared processing modules.The shared modules include extraction, chunking, embedding, and generation.Source: A. Overview · p. 2, Figure 1 · p. 2

B. Processing Pipeline

  • The pipeline receives a PDF or image, detects potentially corrupted page text, routes text-native pages through PyPDF2 and scanned, garbled, or image-based pages through Gemini Vision OCR, then splits extracted text into overlapping 500-character chunks with 50-character overlap.Source: B. Processing Pipeline · p. 2
  • Chunks are embedded with all-MiniLM-L6-v2 and stored in Supabase; queries retrieve top-k chunks scoped to the active report, then pass retrieved context and recent chat history to the fine-tuned LLaMA-3 generator.When no report is selected, retrieval uses a global fallback; responses are returned to the frontend and stored with a report_id.Source: B. Processing Pipeline · p. 3

C. Retrieval-Augmented Generation Formulation

MeddeX uses dense retrieval to condition LLaMA-3 generation on relevant report chunks, with outputs intended to remain traceable to source evidence. Its extraction pipeline detects corrupted PDF text and routes pages through text extraction or Gemini Vision OCR.

C. Retrieval-Augmented Generation Formulation

  • Dense embeddings score chunk relevance, and the top-k retrieved passages are concatenated as context for the fine-tuned LLaMA-3 generator.The generator receives retrieved passages along with recent chat history.Source: C. Retrieval-Augmented Generation… · p. 3
  • Conditioning generation on retrieved chunks is intended to make factual claims in summaries traceable to source evidence.The passage identifies traceability as a requirement for clinical trustworthiness.Source: C. Retrieval-Augmented Generation… · p. 3

A. Garbage-Text Detection

  • UTF-16-encoded or malformed PDFs can yield hex-like text that would otherwise be embedded and retrieved as valid clinical content.The _is_garbage_text() function guards against these corrupted extractions.Source: A. Garbage-Text Detection · p. 3
  • Pages are flagged when over 30% of characters are contiguous runs of 20+ uppercase hex digits, or over 20% are non-printable or non-ASCII.Flagged pages are rerouted to Gemini Vision OCR.Source: A. Garbage-Text Detection · p. 3

B. Smart Multimodal Extraction

  • MeddeX routes pages to direct PyPDF2 extraction, Gemini fallback after garbage detection, or full-image Gemini OCR for scanned pages.The routing logic selects among these three extraction paths.Source: B. Smart Multimodal Extraction · p. 3

C. Gemini Vision OCR Prompt

  • The Gemini Vision OCR prompt instructs the model to extract structured clinical information comprehensively.The passage does not specify the prompt's individual fields.Source: C. Gemini Vision OCR Prompt · p. 3

D. Report-Scoped Retrieval

MeddeX scopes retrieval to a selected report to prevent context bleed, compares two reports longitudinally, and lazily processes legacy documents when accessed.

D. Report-Scoped Retrieval

  • An optional report_id scopes retrieval to a specific document, preventing context bleed across a user's uploaded reports.This addresses correctness when patients have several uploaded documents.Source: D. Report-Scoped Retrieval · p. 3

E. Longitudinal Report Comparison

  • The /compare_reports endpoint compares two reports across parameter changes, clinically significant findings, stable values, and an overall health trend.It retrieves each report's full text and generates a structured comparison summary.Source: E. Longitudinal Report Comparison · p. 4

F. Lazy Backfill Pipeline

  • On first access, the backfill helper extracts and chunks legacy files, creates embeddings under a new UUID, and uses the database on subsequent calls.Cached chunks that fail the garbage-text check are purged and re-extracted.Source: F. Lazy Backfill Pipeline · p. 4

G. Embedding and Vector Storage

MeddeX stores report chunks as dense vectors for scoped or global retrieval, then generates answers and comparisons through its API. The generation module uses a QLoRA fine-tuned LLaMA3-Med42-8B checkpoint.

G. Embedding and Vector Storage

  • all-MiniLM-L6-v2 encodes chunks as 384-dimensional vectors stored in Supabase with chunk text and report metadata.Scoped and global RPC retrieval functions use cosine similarity via pgvector.Source: G. Embedding and Vector Storage · p. 4

H. Fine-Tuned LLM Generation

  • The generation module uses a QLoRA fine-tuned LLaMA3-Med42-8B checkpoint with nucleus sampling at top-p 0.9 and temperature 0.3.The maximum is 300 new tokens for Q&A and 600 for report comparisons.Source: H. Fine-Tuned LLM Generation · p. 4

I. API Endpoints

  • The API provides upload, consultation, and report-comparison endpoints for storing reports, answering questions, and generating longitudinal summaries.Consultation accepts an optional report_id; comparison accepts two report IDs and retrieves report text via the backfill pipeline.Source: §I · p. 4

V. EVALUATION AND RESULTS

The evaluation compares base and QLoRA fine-tuned LLaMA3-Med42-8B on de-identified medical reports and curated clinical Q&A pairs. The fine-tuned model outperforms the base across all five reported metrics; QLoRA also reduces fine-tuning VRAM requirements.

A. Experimental Setup

  • Evaluation compares base and QLoRA fine-tuned LLaMA3-Med42-8B using de-identified medical reports and curated clinical Q&A pairs.Metrics include ROUGE-1/2/L, BLEU, and BERTScore.Source: A. Experimental Setup · p. 4

B. Quantitative Results

C. Computational Resource Comparison

  • QLoRA reduces fine-tuning VRAM requirements by approximately 4× versus full BF16 finetuning, enabling training on an RTX 3090 rather than an enterprise A100.Inference VRAM is similarly reduced; MeddeX uses temperature 0.3 versus 0.7 for the base model to favor factual consistency over generative diversity.Source: C. Computational Resource Comparison · p. 5

D. Analysis

Fine-tuning improves MeddeX’s evaluation metrics, with the largest gain in BLEU. The discussion connects its design to resource efficiency, report-specific grounding, multimodal document handling, and clinical utility, while noting evaluation and context-length limits.

D. Analysis

  • +71.4% BLEU was the largest gain; ROUGE improved 9–12%, while BERTScore rose +0.8%.The discussion attributes the BLEU gain to more precise phrase-level matches and describes the BERTScore change as refinement from a strong base score of 0.8765.Source: Figure 3 · p. 5, D. Analysis · p. 5

E. Discussion

  • QLoRA reduces fine-tuning VRAM by ∼4× and inference VRAM by half versus the full-precision base model.The discussion describes this as resource-efficient adaptation without full model retraining.Source: E. Discussion · p. 5
  • Report-scoped RAG conditions generation on per-document context, reducing hallucination risk and cross-report contamination, while Gemini Vision and garbage-text detection handle scanned, UTF-16, and radiological inputs.These capabilities address document diversity that text-only systems may mishandle.Source: E. Discussion · p. 5
  • Longitudinal comparison tracks health-parameter changes across visits, and lazy backfill processes legacy reports without missing preprocessing history.Source: E. Discussion · p. 6
  • The evaluation uses a relatively small dataset and lacks a large-scale multi-site trial; rare-terminology hallucinations and 1,500-character report truncation remain limitations.The truncation may omit details from very long reports.Source: E. Discussion · p. 6

VI. CONCLUSION AND FUTURE WORK

MeddeX combines fine-tuned language modeling, retrieval, vision, and document-processing features, with reported gains over the base model across five metrics. Planned work spans validation, longer-report processing, integration, efficiency, explainability, and broader multimodal support.

A. Conclusion

  • MeddeX combines QLoRA-tuned LLaMA-3, RAG, Gemini Vision, garbage-text detection, report-scoped retrieval, longitudinal comparison, and lazy backfill for medical-report analysis.Source: A. Conclusion · p. 6
  • Evaluation shows consistent gains over base LLaMA3-Med42-8B across five metrics, including a +71.4% BLEU improvement.Source: A. Conclusion · p. 6

B. Future Work

  • Planned work includes multi-center validation, hierarchical summarization for long comparisons, EHR integration, and model distillation for resource-constrained inference.Source: B. Future Work · p. 6
  • Other planned extensions are calibrated confidence and source attribution, plus end-to-end radiology report generation from raw imaging.Source: B. Future Work · p. 6

Related papers

Loading 2610.20632v1…