Source-linked AI summary
UIC-AIHealth4All at ArchEHR-QA 2026: Answer-First Evidence Grounding for Clinical Question Answering
Mohammad Arvan, Hossein Haeri, Natalie Parde, Rebecca T. Feinstein
TL;DR
Grounded EHR question answering requires jointly selecting evidence and generating answers that remain verifiable against clinical notes, a relationship not established by independently scoped QA methods. UIC-AIHealth4All uses answer-first candidate generation for evidence and answers, self-consistency voting for alignment, and linguistic analysis of clinician-model differences. The system ranked third in evidence identification, ninth in answer generation, and fifth in alignment, while model outputs remained harder to read despite matched length.
Problem
Grounded EHR question answering must jointly select supporting evidence and generate verifiable answers, but the benefit of generating candidate answers before evidence classification was not established.
Method
The system generates cited candidate answers before classifying evidence, uses correction calls for constrained answer generation, applies five-sample vote-threshold alignment, and analyzes 45 linguistic features.
Results
The pipeline ranked third on evidence identification (Strict Micro F1 62.90), ninth on answer generation (Overall 31.90), and fifth on alignment (F1 79.81), while readability remained 3.2 Flesch-Kincaid grade levels harder.
Takeaways & Limitations
Answer-first ordering offers a practical advantage for evidence identification, while readability and stylistic register require explicit optimization beyond factual accuracy.
Takeaways & Limitations
The study used one ArchEHR-QA 2026 dataset, and it conducted no human or clinical evaluation of appropriateness, tone, or clinical accuracy.
Abstract
from arXiv · showhide
We describe the UIC-AIHealth4All system for ArchEHR-QA 2026, a shared task on grounded question answering from electronic health records. We participated in Subtasks 2 (evidence identification), 3 (answer generation), and 4 (answer-evidence alignment). For Subtasks 2 and 3, we propose an answer-first pipeline in which the model generates candidate answers citing specific note sentences before classifying the full evidence set, exploiting the asymmetry between judging relevance in the abstract versus relative to a generated answer. For Subtask 4, we apply self-consistency voting over five independent model calls, retaining links by vote threshold. Our pipeline ranked third on evidence identification (Strict Micro F1 62.90), ninth on answer generation (Overall 31.90), and fifth on answer-evidence alignment (F1 79.81). A post-hoc linguistic analysis of 45 stylistic features reveals that model outputs remain 3.2 Flesch-Kincaid grade levels harder to read than clinician-authored references despite matching their word and sentence counts, suggesting readability warrants explicit optimization in clinical NLP systems. Code and prompts are available at https://github.com/mo-arvan/archehr-qa-2026-uic-aihealth4all.
1. Introduction
UIC-AIHealth4All addresses grounded question answering from EHRs by testing whether answer-first processing improves evidence identification and answer generation. Across three subtasks, it combines answer-first modeling with alignment voting and finds strong task performance alongside a persistent readability gap.
- Motivation: Grounded clinical QA requires anchoring every answer claim to specific source passages so medical decisions can be verified.This creates a joint constraint: evidence selection must support the answer, and the answer must not exceed its evidence.
- Research gap: Whether generating a candidate answer before classifying evidence improves evidence-identification accuracy had not been established.Prior work commonly treated retrieval, reading comprehension, and information extraction as independent tasks.
- Scope: The team participated in Subtasks 2, 3, and 4, covering evidence identification, answer generation, and answer-evidence alignment.The shared task evaluates evidence, answers, and alignments against expert clinical judgments.
- Approach: The answer-first pipeline generates candidate answers citing note sentences, classifies the full evidence set relative to those answers, and then produces the submission answer.For alignment, the system uses independent model calls and vote-threshold aggregation to balance precision and recall.
- Results: 62.90 Strict Micro F1 ranked the pipeline third on evidence identification, while answer generation ranked ninth overall at 31.90 and alignment ranked fifth at 79.81 F1.Iterative constraint enforcement also achieved 100% output compliance within five correction attempts.
- Readability: 3.2 Flesch-Kincaid grade levels separated model outputs from clinician references despite matched word and sentence counts.The analysis found a residual register gap after length matching.
2. Method
The system decomposes grounded EHR question answering into answer-first evidence classification, constrained answer generation, and self-consistency alignment. It evaluates both task performance and stylistic proximity to clinician-authored answers using matched statistical analyses and a category-level distance measure.
- Task setup: Subtasks 2, 3, and 4 respectively select supporting note sentences, generate answers of at most 75 words, and align reference-answer sentences to note sentences.Cases contain patient questions, clinician reformulations, clinical notes, and clinician-authored references with sentence-level annotations.
- Answer-first pipeline: The answer-first pipeline generates multiple cited candidate answers before classifying each note sentence as Essential, Supplementary, or Not Relevant.A sentence is Essential only when removing it would leave the generated answers unsupported.
- Answer generation: Final answers use Essential and optionally Supplementary sentences, undergo validation, and trigger correction calls for word-limit, citation, or format violations.The correction loop allows up to five attempts per case.
- Ensembling: The Subtask 2 ensemble combines seven configurations by majority vote, predicting Essential when at least four runs agree.Configurations vary model versions and candidate-answer counts to increase prediction diversity.
- Alignment: Self-consistency alignment samples five independent calls and retains each answer-to-note link appearing in at least three samples.The threshold was selected using a development-set precision-recall trade-off.
- Evaluation: The study evaluates task metrics, compliance measures, and stylistic differences across 45 linguistic features spanning six categories.The categories include lexical, syntactic, readability, stylistic, sentiment and tone, and clinical communication features.
- Statistical analysis: Paired Wilcoxon tests, Cohen’s d, and standardized linguistic distance quantify systematic and practically large model-clinician differences.The distance averages normalized absolute model-clinician differences using clinician variability as the scale.
- Stylistic analysis: A radar chart compares category-level profiles for the submitted two-step pipeline and an exploratory three-step rewrite variant.Clinician scores form a unit circle; values above or below 1.0 indicate model excess or deficit relative to clinicians.
3. Results
The system performed competitively across evidence identification, answer generation, and answer-evidence alignment, while compliance correction eliminated structural failures. Test-set drops and content-safety refusals expose important operational boundaries.
- Evidence identification: 62.90 Strict Micro F1 ranked the answer-first ensemble third on the evidence-identification test set.The ensemble exceeded the best individual test submission by 3.2 F1 points.
- Evidence identification: Development recall of 81.82% fell to 66.96% on test, a 14.9-percentage-point decrease suggesting distribution shift.The test set contained 47 cases.
- Answer generation: 31.90 Overall ranked ninth for answer generation on the test set, compared with 32.21 Overall on development.SARI was highest on both sets, while BLEU was lowest.
- Alignment tuning: Lowering the vote threshold from 4 to 3 recovered 4.2 recall points at a 2.4-point precision cost and gained 1.34 development F1 points.Five samples represented a fivefold increase over a single call.
- Constraint compliance: 100% output compliance was achieved on both development and test within five correction attempts.First-try compliance was lower on test, but the correction loop closed the gap fully.
- Failure modes: Word-count violations comprised 94.9% of test errors, while citation, schema, and pipeline failures were absent.Development-set tuning therefore did not generalize fully to length adherence on longer test notes.
- Content safety: Content filtering affected 11 unique cases and persistent model refusal affected 8, all refusal cases involving sensitive clinical content.Retries typically resolved infrastructure filtering, whereas correction prompts could not resolve model refusals.
3.5. Linguistic Analysis
The linguistic analysis found significant stylistic divergence between model outputs and clinician references across readability, lexical, and structural features, while word and sentence counts were comparable. A three-step rewrite reduced linguistic distance and improved stylistic alignment.
- Model versus Clinician Divergence: 8 of 45 linguistic features significantly diverged between model outputs and clinician references after Benjamini-Hochberg correction.The analysis used Wilcoxon signed-rank tests to identify systematic differences.
- Model versus Clinician Divergence: The Flesch-Kincaid grade was 3.2 points higher for model outputs, indicating lower readability than clinician references.Model outputs also had a 21-point lower Flesch Reading Ease score and a 4.1-point higher Coleman-Liau grade.
- Model versus Clinician Divergence: Word count and sentence count did not differ significantly between model outputs and clinician references.Word count was 73.5 versus 72.7, and sentence count was 4.75 versus 4.45.
- Two-Step versus Three-Step Linguistic Distance: The three-step rewrite reduced linguistic distance from 0.519 to 0.381 and improved Subtask 3 Overall by approximately 2 development-set points.The rewrite also produced gains across all six linguistic feature categories.
- Two-Step versus Three-Step Linguistic Distance: The Three-Step configuration improved readability and syntactic alignment relative to the Two-Step configuration.Its Flesch-Kincaid grade was 12.52 versus 13.87, closer to the clinician reference at 10.72.
4. Discussion
The system achieved competitive results across evidence identification, answer generation, and answer-evidence alignment using answer-grounded classification, constraint correction, and self-consistency voting. However, performance metrics did not capture a persistent readability and stylistic gap, and configuration effects were not cleanly isolated.
- Main Findings: The answer-first pipeline ranked third on evidence identification, while self-consistency voting ranked fifth on answer-evidence alignment.Reported scores were Strict Micro F1 62.90 and F1 79.81, respectively.
- Linguistic Findings: Model outputs remained harder to read and stylistically less similar to clinician answers despite matching word and sentence counts.The linguistic analysis found 8 of 45 significantly different features and a 3.2-point higher Flesch-Kincaid grade.
- Evidence Identification: Answer-grounded evidence classification improved evidence selection over direct classification baselines by providing generated-answer context.The ensemble gained approximately 3 F1 points over the best single-run submission.
- Limitations: The configuration sweep could not cleanly isolate individual factor contributions because reasoning effort and candidate count varied across runs.Ensemble diversity across model versions and k settings was identified as the primary driver of the test-set gain.
- Answer Generation: The answer-generation submission achieved Overall 31.90, but this aggregate combines nine heterogeneous metrics that may partially conflict.SARI was the highest-scoring individual metric at 57.24.
- Answer-Evidence Alignment: In answer-evidence alignment, lowering the vote threshold from 4 to 3 increased recall by 4.2 points while reducing precision by 2.4 points.This trade-off produced a net F1 gain of 1.34 points.
5. Conclusion
The system combined answer-first evidence classification, iterative constraint enforcement, and self-consistency voting across three grounded EHR question-answering subtasks. Results included strong evidence-identification and alignment rankings, while linguistic analysis exposed a persistent readability gap and motivated further evaluation.
- The system's primary contributions were answer-first evidence classification for Subtasks 2 and 3, iterative constraint enforcement, and self-consistency voting for Subtask 4.The system participated in three ArchEHR-QA 2026 subtasks.
- Third-ranked evidence identification achieved Strict Micro F1 62.90.The result came from an ensemble enabled by the answer-first pipeline.
- Fifth-ranked answer-evidence alignment achieved F1 79.81 using self-consistency voting with threshold t = 3.Vote count provided a reliable proxy for link precision.
- The linguistic analysis found that model outputs remained harder to read than clinician-authored answers despite matched word and sentence counts.The analysis covered 45 linguistic features and identified a persistent stylistic gap.
- Future work should evaluate stylistic divergences clinically, test answer-first grounding beyond clinical data, and reduce the latency cost of register correction.The paper also proposes extending self-consistency voting to evidence classification and answer generation.
6. Limitations
The study's evaluation is constrained by missing human or clinical assessment, deployment costs, limited dataset and model coverage, and a small linguistic-analysis sample.
- No human or clinical evaluation was conducted, so automatic metrics may miss patient- and clinician-relevant answer qualities.Examples include appropriateness, emotional tone, and clinical accuracy.
- The multi-stage pipeline requires at least three sequential API calls per case, while the three-step rewrite variant increases latency 9.5-fold.This latency may preclude deployment in time-sensitive clinical settings.
- Experiments used one ArchEHR-QA 2026 dataset, limiting evidence about generalization across EHR systems, note types, specialties, and patient populations.
- The system depends on proprietary GPT-5.1 and GPT-5.2 models through Azure OpenAI, so results may not transfer to open-weight models.API behavior may also change across versions without notice.
- The linguistic analysis used 20 development cases, limiting statistical power for features with small effect sizes.The strongest readability divergences were reported as robust, but near-threshold findings require caution.
7. Ethical Considerations
The study used credentialed access and institutional safeguards for clinical notes and model services, while excluding protected health information from prompt development and tuning.
- Clinical notes were accessed through credentialed PhysioNet accounts under a signed data use agreement.
- The system used Azure OpenAI Services under a university account with a Business Associate Agreement and modified abuse monitoring.The modified monitoring disabled provider human review of prompts and completions.
- No protected health information was used during prompt development or tuning.
B. Linguistic Feature Analysis
The appendix tables document constraint-compliance rates by correction attempt and compare 45 linguistic features between the submitted configuration and clinician references.
- Table 5 reports cumulative constraint compliance by correction attempt across 35 development and 10 test experiments.
- Table 6 lists all 45 linguistic features for the submitted configuration versus the clinician reference, sorted by Cohen's d magnitude.It also defines adjusted-p-value significance markers and not-applicable entries.