Taking the PULSE of Medical Reasoning: Step-Level Retrieval Timing for Knowledge-Grounded Inference

1 Introduction

• Medical reasoning may require external biomedical evidence, particularly when specialized or rapidly evolving knowledge is needed.Prior RAG work has focused substantially on improving the relevance and quality of that evidence.Source: §1 · p. 1
• Step-level retrieval timing addresses whether internal knowledge is reliable after each step, since errors may arise only at specific points in a reasoning chain.The broader aim is balancing reasoning accuracy against retrieval frequency rather than retrieving indiscriminately.Source: §1 · pp. 1–2
• Erroneous reasoning steps exhibit substantially higher inter-layer volatility than correct ones, motivating an intrinsic signal for retrieval timing.The introduction describes this as internal knowledge instability.Source: §1 · p. 2
• PULSE detects step-level factual errors from geometric evolution across hidden-state layers and selectively triggers biomedical retrieval and step regeneration.A lightweight probe operates on compact trajectory features after each step completes.Source: §1 · p. 2
• 72% fewer retrieval calls let PULSE match exhaustive per-step retrieval in downstream accuracy across four medical QA benchmarks.The same trigger transfers zero-shot to an unseen legal domain.Source: §1 · p. 2

2 Related Work

• Exhaustive per-step retrieval can add latency and distracting evidence, while omitting retrieval at deficient steps can let errors propagate.Interleaved methods such as IRCoT and ReAct address retrieval within multi-step reasoning; PULSE is described as orthogonal to retrieval-pipeline improvements.Source: §2 · p. 2
• Output-level triggers use token probabilities, entropy, logit-based regressors, heuristics, or reflection tokens, but remain surface-level proxies for knowledge deficiency.These signals are described as cheap to compute.Source: §2 · pp. 2–3
• Prior internal-state methods examine hidden-state uncertainty or geometry, but the cited approaches operate on complete queries or responses rather than individual steps.The passage names SeaKR, CoE, and work on knowledge recall and self-verification.Source: §2 · p. 3
• PULSE applies layer-wise geometric trajectory analysis at step level to detect factual errors and provide domain-transferable retrieval triggers.Its trajectory features are fed to an MLP probe, with flagged steps triggering retrieval.Source: §2 · p. 3, Figure 2 · p. 3

3.1 Posterior Trajectory Analysis

• Erroneous-step trajectories show greater mid-to-late-layer volatility than correct steps, and this gap persists across matched complexity strata.The separation arises from the sequential evolution across layers, not any single layer's activation.Source: §3.1 · p. 4

3.2 Prior Error Detection

• PULSE represents trajectory volatility with angular-change and magnitude-change sequences, then concatenates them as features for a two-hidden-layer MLP that flags steps exceeding a threshold.The probe uses a contiguous mid-to-late-layer range; the study validates its boundaries in an ablation.Source: §3.2 · p. 4

3.3 Detection Experiments

• The probe is evaluated on held-out MedMCQA and out-of-distribution MMLU-Pro Biology, against output-level indicators and hidden-state methods including ReProbe.The label distribution is imbalanced, so the study reports PR AUC alongside AUROC.Source: §3.3 · p. 5
• On Llama's out-of-distribution MMLU-Biology set, AUROC reaches 0.7933 versus 0.7406 for ReProbe (+5.3 pp) without re-training; PULSE also achieves the highest MedMCQA AUROC on both backbones.The reported MedMCQA AUROC values are 0.7385 on Qwen and 0.7528 on Llama.Source: §3.3 · p. 5

4.1 Error-Triggered Retrieval Framework

• After each step, the probe scores its trajectory; scores above threshold trigger retrieval and evidence injection, while other steps continue using parametric knowledge alone.PULSEfront prepends retrieved passages before the question; PULSEstep inserts them before the flagged step, and both regenerate that step while preserving preceding steps.Source: §4.1 · p. 6

4.2 Retrieval Experiments

• The retrieval experiments cover unseen probe-training questions from MedMCQA, JAMA Clinical Challenge, and MMLU-Pro Health and Biology, comparing methods that span different retrieval frequencies.The baselines include no-retrieval CoT, every-step IRCoT, and adaptive retrieval methods.Source: §4.2 · p. 6
• With Qwen, PULSEfront exceeds IRCoT by +4.96% on JAMA and +3.63% on MMLU-Pro Biology using 2.6–3.0 calls; on remaining benchmarks it stays within 2–4% at roughly one-third the retrieval budget.IRCoT retrieves at every step, using 5–9 calls per instance.Source: §4.2 · p. 7
• Both injection variants improve over unassisted CoT across all benchmarks and backbones while using fewer retrieval calls than IRCoT, with relative accuracy varying across settings.The study uses accuracy and average retrieval calls per instance to assess the trade-off.Source: §4.2 · pp. 6–7

5.1 Ablation

• 76.43% downstream accuracy and 0.7273 AUROC at 2.62 calls result from combining magnitude and angle features, while magnitude alone reaches 0.6565 AUROC and 2.51 calls and angle alone reaches 0.6866 AUROC and 2.94 calls.The authors interpret the two feature dimensions as complementary.Source: §5.1 · pp. 7–8, Table 3 · p. 8
• AUROC 0.7588 and F1 0.5474 for interval [7,25] exceed the all-layer configuration's AUROC 0.7559 and F1 0.5378, using 36 of 54 features.[7,25] is adopted as the default interval; Llama 3.1 uses [6,31] following the same selection logic.Source: §5.1 · p. 8, Figure 4 · p. 8

5.2 Cross-Domain Generalization

• 42.23% accuracy with 2.36 retrieval calls per instance on MMLU-Pro Law, versus 28.52% for CoT and 39.96% at 4.48 calls for IRCoT, using a medical-trained probe without adaptation.The comparison supports transfer of triggering decisions to an unseen domain.Source: §5.2 · p. 8, Table 4 · p. 8

• Erroneous reasoning steps exhibit a distinctive geometric signature in middle-layer hidden states.PULSE monitors this trajectory at the step level to time retrieval, matching or surpassing exhaustive per-step retrieval with 28–54% of calls across four medical benchmarks and transferring zero-shot to legal reasoning.Source: §6 · p. 8
• Future work may test whether the trajectory signature persists at larger model scales and extend trajectory-based error signals as process-level rewards in agentic settings.Source: §6 · p. 9

Limitations

• PULSE requires intermediate hidden states, limiting applicability to open-weight models, and assumes explicit step boundaries; free-form or single-paragraph outputs may need flexible segmentation.The authors name semantic chunking and clause-level boundary detection as possible strategies for defining steps.Source: Limitations · p. 10

A.1 Data Sources and Splits

• Probe training combines MedMCQA and AfriMedQA; evaluation uses a held-out MedMCQA set for in-distribution testing and MMLU-Pro Biology for out-of-distribution testing.The appendix says Tables 5 and 6 summarize question- and step-level statistics, respectively.Source: §A.1 · p. 13, Table 5 · p. 13, Table 6 · p. 13

A.2 Step Segmentation Protocol

• The step-segmentation protocol prompts for steps beginning with “Step” and extracts segments matching “Step [N]:”; nonconforming steps are discarded.The appendix reports that the vast majority of outputs conform to the prescribed format.Source: §A.2 · p. 13

• A step is a factual error when it asserts at least one incorrect or unsupported atomic factual claim, and each step is evaluated independently.Repeating an earlier false claim remains an error, but a valid conditional inference from an erroneous premise is not automatically labeled erroneous.Source: §A.3 · p. 13
• GPT generated step-level annotations using the question, correct answer, and full reasoning chain, and the protocol annotated 4,348 training steps and 5,740 evaluation steps.Figure 5 is described as an example of a segmented reasoning chain with step-level annotations.Source: §A.3 · p. 13, Figure 5 · p. 14
• Human verification found above 95% step-labeling accuracy in a sample of correct-answer chains, but used division-of-work review rather than independent double annotation.Reviewers checked atomic factual claims against medical or biological references; all chains with an incorrect final answer were reviewed.Source: §A.3 · pp. 13–14

A.4 Class Distribution and Training Considerations

• 26.0% of training examples are errors; probe training uses positive-class weight 3, with results robust to weights from 2 to 5.The error ratio is approximately 1:2.8 positive-to-negative.Source: §A.4 · p. 14
• Error prevalence is 36.4% on ID test and 22.1% on OOD test; AUROC and PR-AUC assess performance across differing prevalence conditions.AUROC is threshold-invariant, while PR-AUC is sensitive to class prevalence.Source: §A.4 · pp. 14–15

B.1 Baselines for Factual Error Detection (Section 3.3)

• PPL, maximum softmax probability, and entropy provide output-level step scores, with thresholds selected on validation data using Youden’s J statistic.PPL uses average token-level negative log-likelihood; maximum probability averages token confidence, while entropy averages normalized next-token entropy.Source: §B.1 · p. 15
• CoE-R and CoE-C compress layer-transition magnitude and angle changes into training-free scores, whereas ReProbe classifies concatenated final-token hidden states with a Transformer.ReProbe does not explicitly model the layer-by-layer sequence.Source: §B.1 · p. 15

B.2 Baselines for Retrieval Experiments (Section 4.2)

• Retrieval baselines include no-retrieval CoT, retrieval at every step with IRCoT, entropy-triggered DRAGIN, covariance-triggered SeaKR, and ReProbe as a learned trigger.IRCoT serves as an indiscriminate full-retrieval strategy; ReProbe is integrated with PULSE’s evidence injection strategy and corpus.Source: §B.2 · pp. 15–16

B.3 Robustness to Explicit Uncertainty Expressions

• PULSE’s AUROC changes from .746 to .744 on MedMCQA and from .712 to .704 on Biology after explicit uncertainty expressions are filtered.ReProbe also retains predictive performance and attains higher PR-AUC on these subsets.Source: §B.3 · p. 16, Table 7 · p. 17

B.4 Paired Statistical Analysis on JAMA

• Under strict final-answer extraction, PULSE-step and PULSE-front exceed ReProbe by 2.45 and 2.71 points, respectively, with both 95% intervals excluding zero.All four Holm-adjusted tests remain below .05; the paired analysis uses 1,511 JAMA questions.Source: §B.4 · p. 16, Table 8 · p. 17

C External Knowledge Sources and Retrieval Setup

• The appendix examines how external-corpus coverage and retrieval configuration interact with retrieval timing, including public-corpus experiments and the main experiments’ knowledge preparation and retrieval setup.The public-corpus experiments include a failure case from insufficient corpus support and a follow-up with a better-aligned corpus.Source: C External Knowledge Sources and… · pp. 16–17

C.1 Evaluation with Public Medical Corpora

• The general textbook corpus does not reliably cover fine-grained JAMA facts, and medically related but question-irrelevant passages can mislead; retrieval timing cannot recover absent knowledge.The authors attribute the JAMA result to insufficient coverage and retrieval noise.Source: §C.1 · p. 17
• On 600 MedQA questions, PULSE has the highest numerical accuracy, exceeding CoT by 1.83 percentage points and IRCoT and SeaKR by 2.17 points, while reducing average retrieval calls versus IRCoT by 82.8% and SeaKR by 73.8%.PULSE skips retrieval for 230 questions (38.3%). On 500 JAMA questions with the MedRAG Textbook corpus, all retrieval-augmented methods fall below no-retrieval CoT, and PULSE degrades less than IRCoT and SeaKR.Source: §C.1 · pp. 17–18, Table 9 · p. 17, Table 10 · p. 18

C.2 Knowledge Preparation

• For the main experiments, GPT-5.1 generates benchmark-specific factual knowledge collections offline from questions, gold answers, and generated reasoning, then answer-neutralization and quality-control steps filter the statements.The reference answers and reasoning traces are not supplied as additional inputs during evaluation; the process includes rule-based filtering, a separate LLM screen, and a random audit.Source: §C.2 · pp. 18–19

C.3 Retrieval Configuration

• At triggered steps, BM25 retrieves top-3 candidate passages using content-word queries, and the system selects the first passage not previously used for that question.The fixed retrieval configuration is intended to reduce variation when comparing trigger strategies.Source: §C.3 · p. 20

C.4 Computational Resources

• PULSE uses a two-layer MLP with hidden dimensions 128 and 64, low-dimensional magnitude and angular features, and tens of thousands of trainable parameters; the backbone remains frozen.Experiments were conducted on NVIDIA A100 GPUs with 80 GB.Source: §C.4 · p. 20

Analysis

• In the NF2 case, PULSE’s Step 2 retrieval supplies missing diagnostic knowledge and leads to the correct diagnosis, while ReProbe’s earlier management evidence and later surgical evidence lead to an incorrect choice.The case illustrates that factually relevant evidence can be premature for the current reasoning need and shift the trajectory toward treatment selection.Source: Analysis · p. 20

E.1 Risks and Safety

• PULSE is a benchmark medical-QA research framework, not medical advice; false positives and negatives and incomplete, outdated, or irrelevant evidence remain risks.Use beyond benchmark research requires task-specific validation, up-to-date evidence sources, and qualified professional oversight.Source: §E.1 · p. 21

E.2 Data Privacy

• The study uses public medical QA benchmarks and biomedical text corpora, without collecting new patient data, accessing electronic health records, or processing identifiable protected health information.The evaluation does not involve interaction with patients or human subjects.Source: §E.2 · p. 21

E.3 Use of AI Assistants

• LLMs support step-level factuality annotation and data-quality filtering, but their automatically produced labels may contain errors and are not authoritative medical judgments.The prompts, label schema, and filtering procedures are described in Appendices A.3 and C.2.Source: §E.3 · p. 21