CRISS: A Retrieval-Augmented AI Chatbot for Assisting Cancer Registrars

The registry work context

• ODSs reconcile ambiguous clinical documentation, integrate information from multiple sources, and apply complex, evolving coding standards.Relevant guidance is spread across national manuals and site-specific guidelines that are periodically revised, so a single case may require consulting multiple references.Source: §1 · p. 2

Gap in existing AI support

• Prior AI approaches largely prioritized automation over support, while registry RAG work also reported hallucinations and factual inconsistencies.These systems were typically evaluated for output accuracy rather than whether they improved user understanding or guideline comprehension.Source: §1 · p. 2

CRISS approach

• CRISS is a conversational assistant that lets registrars ask questions during abstraction and receive explanations supported by explicit citations.It is framed as complementary to autonomous coding tools and prioritizes transparency, traceability, and human oversight.Source: §1 · pp. 2–3

Study aims

• The study evaluates citation-supported guidance, access to and interpretation of guidelines, and training/helpdesk use while preserving human oversight of final decisions.It addresses the augmentation of human expertise rather than its replacement.Source: §1 · p. 3

2 Materials and Methods

• CRISS assists registrars in navigating complex guidelines as a knowledge and educational support tool, not an autonomous coding engine.Its architecture combines registry knowledge-base retrieval with LLM-assisted response generation, while final coding decisions remain with the human user.Source: §2 · pp. 3–4, Figure 1 · p. 3

2.1 Construction of a Registry-Specific Knowledge Base

• National standards and registry guidelines are parsed with their document structure preserved, then segmented into overlapping 1,500-character chunks with 200-character overlap and source metadata.Preserving headings, rule identifiers, page numbers, tables, and notes retains context for exceptions and supports source citation.Source: §2.1 · p. 4

2.1.1 Semantic Retrieval of Relevant Guidelines

• Dense embeddings support semantic search; retrieved passages are reranked for applicable guidance, and iterative multi-hop retrieval seeks additional evidence when needed.The default retrieval is Top-k with k=6, and the retrieval layer can be updated independently of answer generation.Source: §2.1.1 · p. 4

2.2 Citation-Grounded Response Generation

• The LLM is prompted to answer from retrieved registry guidance, explain its reasoning, and cite sources; when evidence is insufficient, it should identify missing case information rather than infer an answer.Typical responses include a direct answer, explanation, supporting references, and context for why a rule applies.Source: §2.2 · pp. 4–5

3.1 Evaluation Design

• CRISS was evaluated on retrieval and response quality across expert-annotated easy, medium, and hard questions, with frontier GPT and Gemini models with web access as non-RAG baselines.Easy questions involve direct lookups, medium questions require multi-step reasoning, and hard questions reflect ambiguous or incomplete real-world cases.Source: §3.1 · p. 5

3.2 Retrieval Evaluation

• Inference-time retrieval metrics captured confidence, relevance, faithfulness, hop count, context length, reasoning trace, sources used, and cited-guideline year.These measures were used to characterize retrieval behavior and the temporal relevance of referenced rules.Source: §3.2 · p. 5

3.3 Response Evaluation

• Response quality used ROUGE, BERTScore F1, cosine similarity, and a six-dimension LLM-judge rubric; GPT-4o and Gemini 3.1 Pro Preview scores were averaged.Large judge disagreements exceeding 0.5 points triggered qualitative review; LLM judging was selected in part because expert annotation at the evaluation scale was prohibitively expensive and time-consuming.Source: §3.3 · pp. 5–6

4.1 Retrieval Quality

• Composite retrieval scores exceeded the 0.70 reference threshold for nearly all evaluated models, and retrieval quality was generally preserved as question difficulty increased.GPT-4o-mini scored 0.97/0.98/0.97 across easy/medium/hard tiers; stability varied across models.Source: Figure 2 · p. 6, §4.1 · p. 6

4.2 Response Quality

• Grounding scores were 0.62/0.56/0.59 for RAG versus 0.29/0.26/0.29 for No-RAG across easy/medium/hard tiers.The reported improvements were 114%, 118%, and 103%, respectively.Source: §4.2 · p. 6, Figure 3 · p. 7
• RAG Local models had the highest mean ROUGE, BERTScore F1, and cosine similarity, followed by RAG Proprietary models, with baselines lowest.RAG Local recorded ROUGE 0.105, BERTScore F1 0.580, and cosine similarity 0.496.Source: Figure 4 · p. 7, §4.2 · p. 7

4.3 LLM-as-a-Judge Assessment

• LLM-as-a-Judge scores favored proprietary RAG models on easy and medium tiers, while RAG Local models ranked highest on the hard tier.On hard questions, Qwen3.5-122B-A10B scored 2.94, OSS-120B 2.91, and Qwen3.6-35B-A3B 2.84.Source: §4.3 · pp. 7–8, Figure 5 · p. 8

5.1 Principal Findings

• RAG configurations generally maintained stronger grounding, semantic similarity, and judge scores than non-RAG approaches as difficulty increased.Some non-RAG models remained competitive on easier questions, but declined more sharply on medium and hard cases.Source: §5.1 · p. 8

5.2 Ablation Study: Model Behavior and Response Caution

• Proprietary RAG models performed strongly on easy and medium questions, while local RAG models ranked ahead on hard questions, where they were less likely to acknowledge uncertainty or decline a definitive answer.The authors caution that the local models’ hard-tier advantage does not demonstrate greater clinical reliability.Source: §5.2 · p. 8, Figure 6 · p. 9

5.3 Reliability of Automated Evaluation

• On hard questions, local RAG had fewer multi-part misses (48% vs. proprietary 90%, baseline 96%) and clinical-safety flags (58% vs. 87%, 88%), but hallucination flags remained high (70% vs. 77%).The flag analysis counted a flag only when both judge models agreed; manual review found that some hallucination flags were false positives.Source: §5.3 · pp. 8–9, Figure 7 · p. 9

5.4 Limitations

• Automated metrics may favor complete or reference-similar responses even when unsupported, and retrieval metrics were partly LLM self-reported, potentially introducing evaluation bias.The study evaluated 22 configurations on 208 questions, producing approximately 4,600 responses, without exhaustive subject-matter-expert review.Source: §5.4 · p. 10
• The system sometimes retrieved a manual edition that did not match a case’s diagnosis year, and year detection with recency-aware re-ranking did not fully prevent historical-case mismatches.The authors suggest exploring explicit diagnosis-year filtering during retrieval.Source: §5.4 · p. 10
• The single-institution question bank may not capture diverse registry workflows, and safety and caution thresholds are evaluation references rather than established deployment standards.Source: §5.4 · p. 10
• CRISS has not yet been evaluated prospectively with practicing cancer registrars in real-world workflows.The authors identify such evaluation as important for assessing usability, reliability, and practical value.Source: §5.4 · p. 10

• Across the evaluation, RAG consistently provided stronger grounding and overall response quality than non-RAG baselines, with the advantage becoming more important as question difficulty increased.Source: §6 · p. 10
• CRISS is intended to help registrars locate, interpret, and apply national standards and guidelines while preserving human oversight and supporting learning and verification.Source: §6 · p. 10
• The conclusion presents domain-specific retrieval, citation-grounded response generation, and multi-model evaluation as a practical foundation for assisting registrars with complex and evolving coding guidelines.Source: §6 · p. 10

• The Missouri Cancer Registry and Research Center supported the work, funded in part through a CDC–Missouri Department of Health and Senior Services cooperative agreement.Source: Funding · p. 10
• Support also included a surveillance contract between the Missouri Department of Health and Senior Services and the University of Missouri.Source: Funding · p. 10
• The authors state that their ideas and opinions do not necessarily reflect those of the named agencies, university, or other funding agencies.Source: Funding · p. 10

A.1 User Interface

• Users submit natural-language questions, and the interface displays generated answers with supporting citations and streams the model’s thinking process.The backend checks a semantic cache first; when no similar answer exists, it retrieves relevant vector-database content, sends it with the question to the LLM, and caches the response.Source: §A.1 · p. 14, Figure 8 · p. 14

B Knowledge Base Construction Parameters

• Manual text is split into overlapping 1,500-character chunks with 200-character overlap and tagged with document, section, page, and available rule identifiers.The metadata supports disambiguating similar passages during retrieval and providing source citations during response generation.Source: B Knowledge Base Construction Parameters · p. 14
• The indexed reference documents are official publications from authoritative cancer registry bodies, with year-specific documents tagged by applicable diagnosis-year range.Table 1 lists the indexed manuals and reference documents.Source: B Knowledge Base Construction Parameters · p. 14, Table 1 · p. 15

C.1 Automatic Metrics: Definitions and Rationale

• The automatic metric suite measures lexical overlap, semantic similarity, and answer relevance as a scalable, reproducible, judge-free first-pass quality signal.ROUGE variants capture lexical overlap, while BERTScore and sentence-level cosine similarity capture semantic similarity; relevance measures alignment with the question.Source: §C.1 · p. 15

C.3 LLM-as-a-Judge Rubric: Dimensions and Flags

• The rubric scores responses across correctness, faithfulness, completeness, reasoning, citation accuracy, and tone, with correctness serving as the primary technical quality signal.Faithfulness distinguishes grounded rules from hallucinated content; citation accuracy checks whether cited documents and pages support the conclusion.Source: §C.3 · p. 17
• Four binary flags identify hallucinated or irrelevant rules, clinically meaningful errors, missed parts of compound questions, and unresolved rule conflicts.The rubric also specifies that a flag is assigned only when both judges identify the same issue; agreement was higher for clinical safety and multi-part misses than for hallucination.Source: §C.3 · pp. 17–18, Figure 13 · p. 18

C.4 LLM-as-a-Judge Prompt Template

• The prompt compares each generated response with its ground-truth answer, scores six dimensions, assigns four reliability and safety flags, and requests a brief rationale.The same structured prompt was applied across models and difficulty tiers.Source: §C.4 · p. 18, Figure 14 · p. 19

C.6 Example Question by Difficulty Tier

• The appendix also provides representative easy, medium, and hard cancer registry coding questions used in the evaluation.The table presents example questions by difficulty level.Source: §C.6 · p. 20, Table 7 · p. 21

C.2 Automatic Metrics: Results

• BERTScore F1 rose from 0.507 on easy questions to 0.579 on hard, while cosine similarity rose from 0.386 to 0.446 across tiers.For medium questions, BERTScore F1 was 0.517 and cosine similarity was 0.388; these averages cover all models.Source: §C.2 · p. 16, Figure 9 · p. 16
• ROUGE-1 was 0.096/0.111/0.179, ROUGE-L was 0.077/0.080/0.108, and ROUGE-avg was 0.071/0.075/0.108 for easy/medium/hard tiers.ROUGE-2 remained comparatively low and stable at 0.041/0.032/0.038 across the same tiers.Source: §C.2 · p. 16, Figure 9 · p. 16
• Several local RAG models had much higher hard-tier ROUGE-avg scores, although lexical overlap alone does not establish correct coding reasoning.The authors relate this pattern to local models more often giving direct and complete answers on difficult cases.Source: §C.2 · p. 16, Figure 10 · p. 16
• Several local RAG models reached their highest BERTScore on hard questions, while easy-to-medium differences were smaller for many proprietary and baseline configurations.The authors caution that semantic similarity can coexist with unsupported details or incorrect coding-rule application.Source: §C.2 · p. 16, Figure 11 · p. 17
• On hard questions, several local RAG models had substantially higher cosine similarity, while some proprietary and baseline models remained stable or declined.The authors interpret similarity alongside grounding and safety measures because relevant terminology does not guarantee correct rule application.Source: §C.2 · p. 17, Figure 12 · p. 17

C.5 Ablation Study: Caution Flags

• RAG Proprietary models had a 23.1% caution/refusal rate, versus 15.3% for RAG Local and 15.2% for the No-RAG baseline.The authors note that the baseline’s similar rate to local RAG means this behavior does not appear to be caused by retrieval alone.Source: §C.5 · p. 19, Table 4 · p. 20
• Proprietary RAG models acknowledged ambiguity in 17.8% of responses, versus 10.3% for local RAG and 6.3% for the baseline.Explicit refusals were 3.4% versus 0.7% for local RAG, while local models more often requested information (2.7% versus 1.6%); baseline conditional answers were 6.9%.Source: §C.5 · p. 20, Table 5 · p. 20
• RAG Local models had the highest containment score (0.401) and a 15.3% refusal rate, while RAG Proprietary models had 0.288 containment and a 23.1% refusal rate.The authors report that models with higher caution or refusal rates tended to include less expected ground-truth content.Source: §C.5 · p. 20, Table 4 · p. 20
• Qwen3.6-27B, Qwen3.5-122B, and Qwen3.6-35B had containment scores of 0.457, 0.444, and 0.435, respectively.These models also produced relatively long responses and had comparatively low refusal rates; the authors caution that overlap may reflect response length and willingness to commit, not better clinical reasoning.Source: §C.5 · p. 20, Table 6 · p. 20