Source-linked AI summary
SentryLine: Evidence-Grounded Question Answering over Evolving Documents in Oncology Care
Tampu Ravi Kumar, Gaurav Najpande, Muhammad Ali Khan, Kaneez Zahra Rubab Khakwani, Karan Kathuria, Shorya Azriel Moses, Yuvraj Kalia, M Bassam Sonbol, Irbaz Bin Riaz, Vivek Gupta
TL;DR
Rapidly changing ASCO guidelines make oncology QA vulnerable to outdated recommendations and hallucinated answers. SENTRYLINE combines hierarchical guideline retrieval, role-specific generation, and independent temporal and factual verification, then evaluates the approach on ASCOBENCH. Across generation backbones and baselines, it reports consistent gains, especially for reasoning and role-specific questions, while remaining limited by backbone quality, explicit update language, and narrow cancer-type coverage.
Problem
Rapidly evolving ASCO recommendations create a need for oncology QA that can track guideline versions and avoid outdated or hallucinated guidance.
Method
SENTRYLINE retrieves ASCO passages hierarchically, generates clinician- or patient-facing answers with citations, and applies factual, temporal, citation, and drift verification.
Results
SENTRYLINE shows consistent gains across three generation backbones and five baselines, including strong improvements on reasoning and role-specific questions.
Takeaways & Limitations
Verified, role-conditioned answers from a shared living-guideline evidence base can improve oncology QA across factual, temporal, reasoning, and communication demands.
Takeaways & Limitations
Performance depends on the generation backbone, drift detection requires explicit update language, and evaluation covers only ASCO breast and prostate guidelines.
Abstract
from arXiv · showhide
Oncology care operates at constant pressure of absorbing rapidly evolving evidence base in biomedicine. The American Society of Clinical Oncology (ASCO) addresses this through living guidelines, but the format introduces a new burden: any recommendation can change at any point, across multiple versioned documents. We present SENTRYLINE, a living guideline-aware clinical question answering system. SENTRYLINE retrieves guideline passages through a vectorless hierarchical RAG pipeline and returns a role-specific answer with inline citations, factual and temporal verification reports, and drift detection notes that surface when a guideline has been updated. We construct ASCOBENCH, a benchmark of 405 three-turn conversations across four question categories with gold answers from expert annotators(clinicians), and use test set to evaluate SENTRYLINE against five baselines under an LLM-as-judge framework. Experiments across three generation backbones show consistent improvements over four retrieval baselines and ASCO's guideline assistant, with particularly strong gains on Reasoning and Role-Specific questions where multi-hop synthesis and register adaptation are required
1 Introduction
SENTRYLINE addresses the risk of outdated or hallucinated oncology guidance by combining living-guideline retrieval, role-specific generation, and verification. The paper introduces ASCOBENCH and reports consistent gains across generation backbones and baselines.
- Living ASCO guidelines can reverse recommendations, while outdated guidance and clinical LLM hallucinations threaten safe oncology question answering.
- SENTRYLINE retrieves current ASCO passages through hierarchical RAG and returns role-specific answers with citations, verification reports, and drift notes.
- ASCOBENCH contains 405 three-turn conversations across four categories, with clinician-annotated gold answers for evaluation.
- SENTRYLINE shows consistent gains in factual grounding, temporal drift handling, and role-conditioned generation across three backbones and five baselines.
2 Related Work
Prior clinical QA systems either store knowledge in model weights or use retrieval without fully addressing evolving guideline structure and temporal drift. SENTRYLINE is positioned against these gaps and evaluates open-ended, guideline-grounded oncology QA.
- Medical LLMs store clinical knowledge in weights, making outputs difficult to trace and stale as guidelines change.
- Retrieval-augmented systems improve grounding, but existing oncology evaluation focuses mainly on exam-style multiple-choice questions.
- SENTRYLINE preserves hierarchical guideline structure during retrieval and addresses gaps in temporal awareness, structure preservation, and role adaptation.
- No prior benchmark targeted temporal guideline drift, motivating evaluation designed around evolving recommendations and open-ended oncology QA.
3 SENTRYLINE: System Architecture and Workflow
SENTRYLINE separates retrieval, role-conditioned generation, and independent verification into a workflow for producing auditable answers from ASCO guidelines. Its components preserve conversation context, citations, temporal status, and evidence links through answer delivery.
- Core workflow: The workflow parses clinical entities, retrieves hierarchical ASCO passages, routes context to clinician or patient templates, and verifies answers before delivery.
- Retrieval: PageIndex navigates guideline PDF trees through LLM reasoning instead of embedding-based nearest-neighbor search, while preserving verbatim document and page citations.
- Context handling: Conversation history enriches multi-turn retrieval, with sessions beyond three turns compressed into a running summary.
- Role conditioning: Clinician and patient templates use identical retrieved context, differing in generation framing, citations, register, and care-team redirection.
- Evidence interface: The evidence panel preserves citations across the conversation and opens the exact grounded paragraph in the cited PDF.
- Verification: Independent factual, page-level, temporal, and citation-grounding checks identify unsupported claims, wrong pages, stale recommendations, and unmatched citations.
4 ASCOBench
ASCOBENCH evaluates living-guideline QA across factual precision, reasoning, temporal tracking, and role-appropriate communication. It uses ASCO breast and prostate guideline documents, multi-turn conversations, and clinician-reviewed reference answers.
- ASCOBENCH tests factual precision, multi-step reasoning, temporal guideline tracking, and role-appropriate communication.
- The benchmark indexes 43 breast cancer and 14 prostate cancer ASCO guideline PDFs, with questions grounded to specific source pages.
- Its four categories cover factual thresholds and dosing, clinical reasoning, outdated-versus-current recommendations, and patient-facing questions at a 6th-8th grade reading level.
- Model-generated question-answer pairs are expanded into 405 three-turn conversations split into train, validation, and test sets.
- Two clinicians review the 105-conversation test split, accepting, revising, or rewriting responses to produce grounded gold answers.
- Evaluation reports fully correct, partially correct, and incorrect accuracy distributions using an LLM judge calibrated against expert ratings.
5 Evaluation Setup
The evaluation compares SENTRYLINE with retrieval, prompting, and production-assistant baselines across three generation backbones on ASCOBENCH. Performance is reported by question category using fully correct and incorrect answer rates.
- Baselines: SENTRYLINE is compared with Direct CoT, Sparse RAG, Dense RAG, Naive RAG, and ASCO’s Guidelines Assistant.The baselines use prompting, sparse retrieval, dense retrieval, raw passage matching, or the production tool.
- Backbones and scoring: Results span Gemini 3.1 Pro, Llama 3.3 70B, and DeepSeek-R1 generation backbones.Each response is evaluated against clinician gold answers using an LLM judge calibrated against expert ratings.
- Dataset and metrics: The test split contains 105 conversations and 315 questions, grouped by Factual, Reasoning, Contrasting, and Role-Specific categories.Each category reports percentages of fully correct, partially correct, and incorrect answers.
- Dataset and metrics: Evaluation reports FC, PC, and Inc distributions for each question category.Table 2 identifies the highest FC and lowest Inc values per column.
6 Results and Analysis
Across ASCOBENCH analyses, SENTRYLINE improves factual grounding, drift handling, role-specific generation, and comparison performance, while maintaining useful multi-turn behavior. Its strongest gains occur on Reasoning and Role-Specific questions, although performance varies by backbone and turn.
- Structured retrieval: SENTRYLINE achieves the highest Factual FC under Llama at 53.8% and Gemini at 56.4%, while DeepSeek Dense RAG reaches 66.7% versus SENTRYLINE’s 59.0%.Under DeepSeek, SENTRYLINE still reduces the incorrect rate from 5.1% to 2.6%.
- Drift detection: 78.1% FC under DeepSeek and 77.1% under Gemini on Reasoning exceed the respective RAG baselines’ 46.9–51.0% and 27.1–31.2%.SENTRYLINE also records 0.0% incorrect under DeepSeek and 2.1% under Gemini on Reasoning.
- Role-conditioned generation: 65.3% FC under Gemini and 61.1% under DeepSeek on Role-Specific questions exceed the next-best baselines’ 43.1% and 29.2%.The role router targets a 6th–8th grade reading level, while baselines retain a clinical register.
- Comparison with ASCO’s assistant: SENTRYLINE outperforms ASCO’s assistant across all four categories: 56.4% vs. 20.0% Factual, 77.1% vs. 10.0% Reasoning, 59.3% vs. 13.3% Contrasting, and 65.3% vs. 8.3% Role-Specific.The largest gaps are +67.1 percentage points on Reasoning and +57.0 percentage points on Role-Specific.
- Multi-turn consistency: FC falls from Turn 1 to Turn 2 for every backbone, while Gemini and DeepSeek recover at Turn 3 more than Llama.The reported Turn 1→Turn 2→Turn 3 FC sequences are Gemini 82.9%→50.5%→63.8%, DeepSeek 77.1%→48.6%→61.9%, and Llama 63.8%→23.8%→35.2%.
- Failure modes: SENTRYLINE has the lowest incorrect rate in 11 of 12 backbone-category cells.The results indicate that shortfalls more often become partially correct rather than outright incorrect, consistent with verifier checks catching unsupported claims.
7 Demo and Reproducibility
SENTRYLINE is released as an installable, MIT-licensed package with code, templates, ingestion adapters, and inspection tools for reproducible use. The release supports both clinician- and patient-facing workflows.
- Release: The open-source package includes setup instructions, full pipeline code, and an MIT license.Users can install it locally for guideline-based clinical question answering.
- Release: The release provides clinician and patient prompt templates, guideline PDF ingestion adapters, and the PageIndex hierarchical retrieval module.These components expose the main role-conditioned and hierarchical-retrieval workflow.
- Inspection: Users can inspect verification reports, evidence panels, and drift-detection annotations for generated answers.The package supports submissions under either declared user role.
8 Conclusion
SENTRYLINE combines current-version guideline grounding, role-conditioned generation, and factual and temporal verification for oncology QA. ASCOBENCH provides expert-annotated evaluation across four question categories, supporting reproducible study and planned clinical piloting.
- System contribution: SENTRYLINE grounds answers in current ASCO breast and prostate guidelines for clinicians and patients through one evidence base.Answers are checked for factual support and temporal currency before reaching users.
- Benchmark contribution: ASCOBENCH contains 405 conversations with expert clinician annotations across factual, reasoning, temporal-drift, and role-appropriate communication categories.The benchmark establishes results across three generation backbones.
- Scope and next step: The system targets answers based on guideline versions already superseded by living standards and is designed for auditable clinical use.The authors plan a clinical pilot at oncology practice as a next deployment step.
- Scope and next step: ASCOBENCH will be released to support reproducible evaluation of guideline-grounded clinical QA.The conclusion also identifies clinical piloting as the next step toward deployment.
Limitations
SENTRYLINE’s performance and scope remain constrained by its generation backbone, explicit update-language dependence, limited guideline coverage, and constructed rather than real-world conversations.
- Limitations: SENTRYLINE’s answer quality depends on the generation backbone, with DeepSeek showing a case where chunk-based RAG achieved a higher Factual FC rate.Verification narrows but does not remove the gap associated with a weaker backbone.
- Limitations: Drift detection cannot flag superseded recommendations when revised guidelines omit explicit update language.
- Limitations: The evaluation covers only ASCO breast and prostate cancer guidelines, not other cancer types, medical societies, or non-oncology guidelines.
- Limitations: ASCOBENCH uses model-generated, clinician-reviewed conversations rather than real clinical sessions, limiting coverage of authentic phrasing and follow-up patterns.
Ethical Considerations
The paper frames SENTRYLINE and ASCOBENCH as transparent, auditable decision-support research using public guidelines, constructed conversations, expert annotation, and controlled evaluation procedures.
- Ethical Considerations: SENTRYLINE is designed as transparent, auditable decision support rather than an autonomous clinical decision maker.
- Ethical Considerations: ASCOBENCH uses publicly available ASCO guidelines and constructed conversations, without protected health information or patient records.
- Dataset Construction: 405 base records become 405 three-turn conversations, with later turns conditioned on prior model responses rather than gold answers.
- Dataset Construction: Conversations are stratified by disease, question category, and user role across train, test, and validation splits.
- Annotation: The 105 test conversations yield 315 annotated turns reviewed by clinical annotators across factual, grounding, appropriateness, communication, citation, temporal, and safety dimensions.
- Evaluation: Results are reported per fixed generation backbone, without additional fine-tuning, using a judge disjoint from the generator pool.
C Ablation Results
Ablations show that factual and temporal verification reduce incorrect answers, while the temporal verifier trades some fully correct answers for fewer incorrect ones; multi-turn FC rates vary by backbone.
- Verifier Ablations: Removing the factual verifier increases DeepSeek Factual incorrect rate from 2.6% to 10.3%.
- Verifier Ablations: Removing the temporal verifier increases DeepSeek Role-Specific incorrect rate from 5.6% to 26.4%.
- Verifier Trade-offs: On DeepSeek Factual, the temporal verifier converts 10.2 pp of FC into PC while removing 2.5 pp of Inc.
- Verifier Trade-offs: On DeepSeek Role-Specific, the temporal verifier removes 20.8 pp of Inc at a cost of 8.3 pp of FC.
- Conversation Turns: All backbones’ FC rates drop from Turn 1 to Turn 2, while Gemini and DeepSeek recover at Turn 3 and Llama recovers only partially.