Source-linked AI summary
sebis at ArchEHR-QA 2026: How Much Can You Do Locally? Evaluating Grounded EHR QA on a Single Notebook
Ibrahim Ebrar Yurt, Fabian Karl, Tejaswi Choppa, Florian Matthes
TL;DR
Cloud-based EHR QA is difficult to deploy because clinical data are sensitive and large models require substantial infrastructure. This paper evaluates all four ArchEHR-QA 2026 subtasks with locally executable models on commodity hardware, finding competitive leaderboard performance and comparable results from smaller models when configured appropriately.
Problem
Privacy constraints and computational requirements leave local, privacy-preserving EHR QA underexplored despite its clinical importance.
Method
The study evaluates classifiers, embedding methods, and small or quantized language models across all four ArchEHR-QA subtasks, entirely locally without external APIs or cloud infrastructure.
Results
The submissions achieve competitive results across multiple subtasks, perform above average in Subtasks 1 and 4, and remain comparable to other systems in Subtask 3.
Takeaways & Limitations
Fully local, privacy-preserving EHR QA appears technically feasible with current models and accessible hardware.
Takeaways & Limitations
The setup relies on high-end commodity hardware and evaluation is limited to the ArchEHR-QA dataset, restricting immediate generalizability.
Abstract
from arXiv · showhide
Clinical question answering over electronic health records (EHRs) can help clinicians and patients access relevant medical information more efficiently. However, many recent approaches rely on large cloud-based models, which are difficult to deploy in clinical environments due to privacy constraints and computational requirements. In this work, we investigate how far grounded EHR question answering can be pushed when restricted to a single notebook. We participate in all four subtasks of the ArchEHR-QA 2026 shared task and evaluate several approaches designed to run on commodity hardware. All experiments are conducted locally without external APIs or cloud infrastructure. Our results show that such systems can achieve competitive performance on the shared task leaderboards. In particular, our submissions perform above average in two subtasks, and we observe that smaller models can approach the performance of much larger systems when properly configured. These findings suggest that privacy-preserving EHR QA systems running fully locally are feasible with current models and commodity hardware. The source code is available at https://github.com/ibrahimey/ArchEHR-QA-2026.
1. Introduction
EHR QA can improve access to clinical information, but privacy constraints and hardware requirements make cloud-dependent systems difficult to deploy. This work evaluates fully local approaches on commodity hardware and reports competitive performance across the shared task.
- Privacy regulations and sensitive health information often prevent healthcare institutions from sending EHR data to external cloud services.
- Local EHR QA is needed because many clinical environments lack infrastructure for hosting large models and rely on standard workstations or notebooks.
- The study evaluates fine-tuned classifiers, embedding methods, and small or quantized language models across all four ArchEHR-QA subtasks.
- A complete clinical QA pipeline can run entirely on standard commodity hardware.
- Out-of-the-box dense embedding models beat fine-tuned cross-encoders for evidence extraction and alignment.
- Synthetic data and quantized or small language models are presented as local strategies for stabilizing classifier training and supporting generation.
2. Related Work
Earlier EHR QA emphasized span extraction, while newer datasets and LLM studies support multi-note and generative settings. However, computational costs and privacy concerns still impede deployment in clinical environments.
- Early EHR QA systems commonly framed the task as extracting answer spans from a single clinical note.
- EHRNoteQA introduced patient-specific questions spanning ten topics and requiring information from multiple discharge summaries.
- Recent work evaluated 27 LLMs in open-ended and multiple-choice EHR QA settings.
- Current LLM-based clinical QA approaches remain difficult to deploy because of computational costs and privacy concerns.
- Local clinical NLP can improve privacy, latency, and compliance but may require greater computational and operational effort.
3. Methods
The shared task decomposes grounded EHR QA into question interpretation, evidence identification, answer generation, and evidence alignment. The authors combine prompting, retrieval, embeddings, generative models, and supervised classifiers for these subtasks.
- The task requires identifying evidence supporting an answer and generating a concise response grounded in a clinical document.
- Question interpretation converts a verbose patient narrative into a clinician-interpreted question restricted to 15 words.
- Evidence identification extracts a minimal sufficient subset of document sentences needed to formulate the answer.
- Answer generation produces a patient-friendly response of at most 75 words grounded in the clinical document.
- Evidence alignment links each generated answer sentence to supporting clinical-document sentences through a many-to-many mapping.
- Question interpretation and answer generation use few-shot or zero-shot prompting with Qwen and gpt-oss models, including a two-step revision setup.
- Evidence identification combines retrieval and supervised classification, including synthetic data, multi-head BERT-style training, and thresholded similarity scores.
- Evidence alignment compares embedding, generative, and supervised approaches, including binary pairwise classification between answer and evidence sentences.
4. Experimental Apparatus
Experiments use the ArchEHR-QA 2026 dataset and evaluate parameter-efficient classifiers, embedding models, and autoregressive models locally on Apple Silicon hardware. Performance is assessed with generation and extraction or alignment metrics using official scripts.
- The ArchEHR-QA 2026 dataset models patient questions answered by clinicians with explicit evidence grounding.
- Each case combines a patient question, clinical note excerpt, expert annotations, evidence links, and clinical specialty information.
- The dataset contains 20 development cases, 100 test cases, and 47 test-2026 cases, with Subtask 4 evaluated on both test sets.
- The evaluated architectures include encoder-only classifiers, dense embedding and cross-encoder models, and autoregressive language models.
- Generative experiments use default decoding settings with temperature 0.7 and top-p 0.9.
- Classifier hyperparameters are selected through grid search and 5-fold cross-validation on the development set.
- Generative tasks are evaluated with BLEU, ROUGE, SARI, AlignScore, MEDCON, and BERTScore, while extraction and alignment use micro Precision, micro Recall, and micro F1.
- All experiments run locally on Apple Silicon, primarily using an M4 Pro MacBook with 48GB unified memory and a Mac Studio for gpt-oss-120B.
5. Results
The evaluation reports results for evidence identification and answer generation, with selected approaches achieving distinct overall test scores across the subtasks.
- 51.61 and 44.43 were the overall test scores for the selected Qwen3-Embedding-8B and Bio_ClinicalBERT-synth-HYDRA approaches in evidence identification.
- Table 6 reports development and test results for answer generation, while the best-scoring setup used a Qwen3.5-35B list-wise prompt.
- 59.45, 74.84, and 8.33 were the reported overall test scores for answer generation approaches.
6. Discussion
The discussion finds that smaller configured models can remain competitive, while retrieval and domain specialization affect trade-offs across subtasks. It also identifies distribution shift, low-resource data, and hardware requirements as important boundaries.
- Smaller architectures can obtain comparable results to larger models in Subtasks 1 and 3, indicating model scale alone is not the main performance driver.
- Prompt repetition slightly improves three of four Subtask 1 test metrics but does not considerably outperform the single-query approach.
- High recall is rewarded in Subtask 2: classifying every candidate as relevant achieves perfect recall and competitive F1, while precision-oriented methods often reduce recall.
- Synthetic data and HYDRA produce more balanced precision–recall trade-offs in some cases, suggesting additional training signals can help when real data are limited.
- The biomedical model consistently outperforms the general-purpose model in Subtask 2, reflecting domain-specific terminology and contextual cues.
- Subtask 4 shows a development–test discrepancy that may reflect distribution shift and overfitting to the development distribution.
- The local systems establish a baseline but still require high-end commodity hardware, leaving execution on phones or clinical tablets for future work.
- Low-resource data likely contributes to fine-tuned cross-encoders' generalization challenges and motivates proxy datasets and improved synthetic-data generation.
7. Conclusion
Using commodity hardware, the approaches achieve competitive results across multiple subtasks, with above-average scores in Subtasks 1 and 4. The findings indicate that smaller, resource-efficient models can support fully local EHR QA.
- Results: Above-average scores were achieved in Subtasks 1 and 4, while Subtask 3 remained comparable to other systems and Subtask 2 lagged.Subtask 4 showed substantial room for improvement.
- Results: Smaller models can achieve performance comparable to significantly larger models with appropriate configurations and training strategies.
- Implications: Resource-efficient approaches therefore have potential for clinical-domain tasks.
- Conclusion: Fully local EHR QA is technically feasible with current models and accessible hardware, although it may lag behind cloud-based systems in some aspects.The reported performance gap appears small enough for many applications requiring privacy and on-device processing.
Limitations
The study’s local setup still depends on high-end commodity hardware and evaluation on a single dataset, limiting immediate generalizability.
- Hardware scope: The local setup relies on high-end commodity hardware that may be unavailable in some under-resourced clinical environments.The example system is an Apple MacBook with an M4 Pro chip and 48 GB memory.
- Evaluation scope: Evaluation is limited to the ArchEHR-QA dataset, restricting immediate generalizability across clinical specialties and multilingual populations.
Ethical Considerations
The work emphasizes local execution as a way to avoid transmitting protected health information to external cloud APIs. It also notes that generative models can produce errors that require careful clinical review.
- Privacy: Local execution eliminates the need to transmit sensitive protected health information to third-party cloud APIs.
- Safety: Generative models evaluated in the study are prone to hallucinations and false conclusions.
- Safety: In clinical contexts, unreviewed model errors could negatively impact patient care.
8. Bibliographical References
The paper’s operational materials provide prompt templates for question interpretation, answer generation, and evidence alignment. These templates constrain outputs to concise, clinically grounded responses and direct evidence links.
- Prompt templates: Question interpretation rewrites a verbose question into one concise clinician-perspective question.The specified output uses third person, one sentence, and a maximum of 15 words.
- Prompt templates: Answer generation produces a professional response using only information explicitly stated in the clinical note excerpt.The response is limited to 75 words and should avoid speculation or outside medical knowledge.
- Prompt templates: Evidence alignment links each answer sentence to directly supporting clinical note sentences.The instructions prohibit over-citation, under-citation, and inference beyond the note text.
- Output format: The evidence-alignment output uses case identifiers, answer identifiers, and arrays of evidence identifiers in strict JSON.
B. Synthetic Data Generation Details
The study augments a limited development set by generating synthetic clinical QA cases locally, then applies quality controls to enforce realistic structure and relevance-label distributions.
- Generation Pipeline: The pipeline uses an initial LLM generation phase followed by targeted LLM-based repair for formatting and distribution errors.Both stages are guided by manually defined quality thresholds intended to mirror the real dataset.
- Quality Constraints: Synthetic cases are constrained to 10–20 sentences, with each sentence containing 10–500 characters.These constraints regulate case structure and sentence length.
- Prompt and Output Format: The generation prompt requests a new patient question, a concise clinician question, a long multi-section clinical note, and sentence-level relevance labels.The output is specified as JSON containing the questions, sentence objects, and relevance-label objects.
- Relevance Labeling: Labels distinguish sentences that directly answer the question, provide helpful context, or do not help answer it.The prompt defines these categories as essential, supplementary, and not-relevant, respectively.