SOPO: Socratic Guided Policy Optimization for Span-Level Hallucination Detection

1 Introduction

• Research has moved from SFT toward outcome-based RL; earlier work focused largely on case-level detection, while RL4HS applied CAPO to span-level detection.The passage identifies FaithLens as a case-level example and RL4HS as a span-level approach.Source: §1 · p. 1
• Prior guided-RL and iterative-refinement approaches have structural mismatches for objective fine-grained detection, including self-evaluation role conflicts, post-hoc hints, and ground-truth-leaking corrections.The passage attributes these concerns to [25],, and, respectively.Source: §1 · pp. 1–2
• A scalar reward must capture boundary alignment, semantic completeness, and factual discrimination, so a one-token boundary error can drop F1 from 1.0 to 0.2.The passage links this coarse signal to training instability and notes that RL4HS introduced CAPO to mitigate reward imbalance.Source: §1 · p. 2
• Hard examples can receive zero rewards across epochs, while increasing group size creates zero-advantage groups on easy samples and wastes computation.The authors associate this trade-off with performance plateauing and reduced training efficiency.Source: §1 · p. 2
• RLVR reasoning can inflate as PPO/GRPO-style objectives favor longer outputs when correctness is difficult, increasing inference latency and computational costs.The passage characterizes redundant filler tokens as length inflation without accuracy improvement.Source: §1 · p. 2
• SOPO uses a teacher LLM to give step-wise analysis and guiding questions about incorrect reasoning without revealing ground truth, turning sparse outcome rewards into denser supervision.The authors describe this as iterative reasoning refinement that prevents shortcut learning and encourages self-correction.Source: §1 · p. 2
• Turn-based Advantage Scaling penalizes verbosity and reliance on multi-turn feedback to encourage efficient, autonomous reasoning.It addresses reasoning-length inflation during RL training.Source: §1 · p. 3
• 4.4 points in overall F1 on the RAGTruth test set separates SOPO (4B) from the previous state-of-the-art, and SOPO also outperforms larger proprietary models including GPT-5 and o3.The authors also report superior performance on out-of-distribution benchmarks.Source: §1 · p. 3

2.1 Task

• The detector identifies response spans that contradict or are unsubstantiated by retrieved context, representing each span with start and end character indices; no hallucinations correspond to an empty set.The setting includes RAG summarization and knowledge-intensive question answering.Source: §2.1 · p. 3

2.2 Metric

• Span-level F1 evaluates localization using character-level overlap between predicted and ground-truth hallucination positions.Precision and recall are calculated from the overlap of the predicted and ground-truth character-index sets.Source: §2.2 · p. 3
• The metric rewards comprehensive coverage of hallucinated content while discouraging inclusion of factual text.These goals correspond to high recall and high precision, respectively.Source: §2.2 · p. 3

3 SOPO: Socratic Guided Policy Optimization

• SOPO frames hallucination detection as a bounded multi-turn process in which the student generates reasoning and predictions from the input and interaction history.The trajectory records student outputs, rewards, and teacher feedback until success or the turn limit.Source: §3 · p. 3, §3.1 · p. 4

3.1 Formalization of Socratic Interaction

• The teacher evaluates each student prediction against ground truth and, when feedback is provided, gives step-wise analysis and a guiding question without revealing the answer.Interaction can end when the span-level F1 reward exceeds a threshold; feedback for lower-scoring responses is stochastic.Source: §3.1 · p. 4

3.2 GRPO with Turn-based Advantage Scaling

• Turn-based Advantage Scaling reduces rewards for trajectories using more turns, addressing the failure of unpenalized rewards to distinguish equally scoring trajectories with different interaction lengths.The scaling factor decreases with turn count before rewards are normalized within the group.Source: §3.2 · p. 5

3.3 Span-Level F1 Reward

• The training reward is based directly on span-level F1, with a turn-averaged reward intended to encourage early error correction.The turn-averaged component is also intended to discourage deliberately failing early turns to solicit hints.Source: §3.3 · p. 5

Dataset

• RAGTruth experiments cover summarization, question answering, and data-to-text, with 15,090 training samples and 2,700 test samples.Each sample pairs source documents and an LLM-generated response with human-annotated hallucination spans.Source: §4 · pp. 5–6

Model

• SOPO uses Qwen3-4B-2507 as its policy model and a frozen Qwen3-Next-80B-A3B-Instruct model for feedback.Source: §4 · p. 6

Cold-start

• SOPO and all baseline models share an SFT initialization built from 5,090 RAGTruth samples, while the remaining 10,000 samples are used for RL training.Gemini-2.5-Flash generates candidate SFT responses, which are selected by span-level F1 against ground truth.Source: §4 · p. 6

Evaluation

• The reported result tables compare span-level and case-level hallucination detection on RAGTruth, marking best scores in bold and second-best scores with underlining.Source: Table 1 · p. 6, Table 2 · p. 6

Inference

• All methods are evaluated on RAGTruth with single-turn inference, producing reasoning chains and predicted hallucination spans without teacher feedback.Source: §4 · p. 7

5.1 Main Results

• SOPO (4B) reaches 63.3 overall F1 on RAGTruth, beating LettuceDetect-Large’s 58.9 and outperforming larger proprietary models and the SFT, GRPO, and CAPO baselines across three tasks.The passage reports a 4.4-point advantage over the previous best framework and robust performance on highly constrained hard examples.Source: §5.1 · p. 7
• SOPO generalizes to case-level detection with 84.8 Overall F1, and ten-run inference gives mean overall span-F1 of 63.34 versus 59.75 for CAPO and 59.49 for GRPO.The case-level result surpasses dedicated methods and standard RL baselines; the span-F1 differences versus CAPO and GRPO are reported as statistically significant under paired bootstrap tests.Source: §5.1 · p. 7
• On hard examples, SOPO improves scores for overly short hallucinated spans, dense multi-span occurrences, and top-10% longest contexts, with a positive score-distribution shift versus GRPO.The passage attributes this result to transforming sparse outcome rewards into dense Socratic guidance.Source: §5.1 · p. 7
• SOPO’s mean primary score exceeds GRPO’s on all four out-of-domain benchmarks: PsiloQA, Math-Shepherd-614, LLM-AggreFact, and HaluEval.The reported pairs are 73.90 vs. 72.51 span-F1, 89.02 vs. 85.68 span-F1, 75.93 vs. 72.55 mean balanced accuracy, and 69.77 vs. 67.52 accuracy, respectively.Source: §5.1 · p. 7
• SOPO reaches CAPO’s roughly 600-step task reward by about 200 steps and matches GRPO’s peak performance within two epochs.Its first-turn performance is consistently higher than other baselines, including in the first epoch.Source: §5.1 · p. 7, Figure 2 · p. 8, Figure 4 · p. 8
• SOPO maintains roughly 1,700 tokens per turn as performance rises, while its entropy stays above GRPO and CAPO as their policies approach entropy collapse.The model reduces average token generation by 39.9% and inference latency by 47.0%, with a one-time training FLOPs overhead of 18.9%.Source: §5.1 · p. 8, Figure 3 · p. 8, Figure 5 · p. 9

5.2 Feedback Scheme

• The full SOPO feedback scheme performs best among the compared styles; analysis-only scores lowest, while question-only is comparable to SOPO.Ground-truth leakage produces inferior test performance despite nearly identical final training rewards, and Direct-Feedback scores 60.81 span-F1 versus SOPO’s 63.31.Source: §5.2 · p. 8, Table 3 · p. 9

5.3 Model Scale and Heterogeneity

• SOPO improves F1 across student scales, with the Qwen3-4B-2507 model gaining +3.1; teacher results favor stronger intrinsic reasoning and heterogeneous guidance over parameter count or a homogeneous setup.The 1.7B model’s marginal improvement is linked to limited long-context capacity for long multi-turn trajectories, while the legacy 8B model is described as having weaker intrinsic reasoning than the newer 2507 series.Source: §5.3 · p. 9, Table 4 · p. 9, Table 5 · p. 9

6 Comparison with Offline Chain-of-Thought Distillation

• SOPO uses teacher feedback conditioned on evolving student reasoning during on-policy RL, whereas 80B-CoT-Distill trains from fixed offline reasoning traces and gold lists.The distillation baseline uses 10,000 examples for approximately four epochs; neither method uses the teacher or gold answers at inference.Source: §6 · p. 9
• 13.25 span-F1 points (nominal paired 95% CI [10.26, 16.28]) separated SOPO from 80B-CoT-Distill on RAGTruth.The results are reported in Table 6, which gives overall RAGTruth F1 across selected inference runs.Source: §6 · p. 10, Table 6 · p. 10
• 5.60 case-F1 points (nominal paired 95% CI [3.73, 7.49]) separated SOPO from 80B-CoT-Distill.The comparison reports nominal paired bootstrap statistics.Source: §6 · p. 10

7 Conclusion

• SOPO combines Socratic guidance with Turn-based Advantage Scaling to address reward sparsity and promote concise reasoning.The conclusion presents the approach as a framework for robust span-level hallucination detection.Source: §7 · p. 10
• The 4B SOPO model achieves state-of-the-art performance on RAGTruth and exhibits strong generalization across diverse out-of-domain benchmarks.Source: §7 · p. 10
• For robust span-level hallucination detection, SOPO integrates Socratic guidance with Turn-based Advantage Scaling to resolve reward sparsity and ensure concise reasoning, substantially reducing inference latency through enhanced reasoning efficiency.Source: §7 · p. 10

8 Limitations

• SOPO leaves advanced credit assignment, including integration with Process Reward Models, for future work; its multi-turn penalty may suppress trial-and-error exploration in open-ended domains.Reported comparison uncertainty uses repeated inference on fixed checkpoints, not significance tests across independent RL training seeds.Source: §8 · p. 10

A Broader Impacts

• Hallucination detectors risk automation bias and dual use, since users may overtrust outputs and adversaries could refine misinformation using diagnostic feedback.Source: A Broader Impacts · p. 13
• The paper recommends assistive, human-in-the-loop deployment and periodic retraining because real-world knowledge changes while models capture a static snapshot.Source: A Broader Impacts · p. 13

B Prompt

• The Socratic prompt directs the teacher to identify the student’s first reasoning error without revealing the answer, then ask a context-based guiding question.A separate prompt variant permits revealing specific corrective facts rather than using only indirect Socratic guidance.Source: B Prompt · pp. 14–15

D Computational Cost

• The computational-cost evaluation compares training FLOPs and test-time inference latency under matched Dynamic Sampling.Source: D Computational Cost · p. 16

D.1 FLOPs Calculation

• The FLOPs estimate uses parameter and token counts; GRPO generates all trajectories before filtering updates, while SOPO adds teacher forward generation and updates only a filtered student-trajectory fraction.The paper estimates forward generation at 2ND and backward updates at 6ND.Source: §D.1 · pp. 16–17

D.2 Empirical Cost Analysis

• 18.9% more training FLOPs and 10.6% more GPU hours accompanied 43.2% shorter single-turn responses, while wall-clock training time fell 11.5%.The cost analysis used 20 GPUs: 16 for RL training and four for teacher deployment.Source: §D.2 · p. 18, Table 10 · p. 18

D.3 Training Efficiency and Convergence

• SOPO reaches GRPO’s peak performance in 50% of the total training steps.The paper attributes the convergence advantage to dense Socratic guidance reducing exploration variance.Source: §D.3 · p. 18

D.4 Test-Time Cost and Deployment Latency

• 39.9% fewer average tokens and 47.0% lower inference latency were reported for SOPO on RAGTruth.Table 11 reports average inference latency and token counts at concurrency 32; Figure 7 shows the output-length distributions.Source: §D.4 · p. 18, Table 11 · p. 18, Figure 7 · p. 18

• 531-step and GPU-hour-matched checkpoints approximately match GRPO’s 632-step FLOP and GPU-hour budgets, respectively.The matching uses average per-step costs; the SOPO full-budget costs are 24.95 ExaFLOPs and 316.40 GPU-hours, compared with GRPO’s 20.98 ExaFLOPs and 286.08 GPU-hours.Source: E Inference-Time Significance on… · p. 19
• Scores were evaluated in 10 full-set inference runs on the fixed 2,700-example RAGTruth test set.Tables 1 and 2 report single-run scores, while Table 12 summarizes repeated-inference results; differences were computed before rounding the displayed means.Source: E Inference-Time Significance on… · p. 19
• 3.47 and 3.25 points: SOPO improved overall span-F1 over GRPO at the FLOP- and GPU-hour-matched checkpoints, respectively.The corresponding improvements over CAPO were 3.21 and 2.99 points; all four nominal paired 95% bootstrap intervals excluded zero (one-sided p ≤0.0057).Source: E Inference-Time Significance on… · p. 19

F Evaluation on Additional Benchmarks

• The full-training and FLOP-matched SOPO checkpoints were evaluated using each benchmark’s primary metric, with means and standard deviations across 10 inference runs.The GPU-hour-matched checkpoint was not included; Table 14’s paired comparisons use median-scoring runs from method pools and need not equal Table 13 mean differences.Source: F Evaluation on Additional Benchmarks · pp. 19–20, Table 13 · p. 20, Table 14 · p. 20

F.1 Span-Level: PsiloQA

• SOPO averaged 73.90 span-F1 on PsiloQA, versus 72.67 for CAPO and 72.51 for GRPO; SOPO-FLOP averaged 74.17.Source: §F.1 · p. 20

F.2 Case-Level: LLM-AggreFact

• SOPO averaged 75.93 unweighted mean balanced accuracy on LLM-AggreFact, versus 75.29 for CAPO and 72.55 for GRPO.The evaluation excluded the RAGTruth subset; SOPO’s nominal paired 95% interval versus GRPO was positive.Source: §F.2 · p. 20

F.3 Case-Level: HaluEval

• SOPO averaged 69.77 overall accuracy on HaluEval, versus 69.19 for CAPO and 67.52 for GRPO.Its nominal paired 95% intervals were above zero against both baselines.Source: §F.3 · p. 20

F.4 Out-of-Domain: Math-Shepherd

• SOPO averaged 89.02 span-F1 on Math-Shepherd-614, compared with 85.89 for CAPO and 85.68 for GRPO.The fixed set retains 614 of 1,000 sampled instances after step-label judgments; SOPO’s nominal paired 95% interval versus CAPO was above zero.Source: §F.4 · p. 21

G Analysis of Hard Case on RAGTruth Test Set

• SOPO had lower probability density than GRPO at span-F1 0.0–0.1 and higher density in 0.8–1.0, especially on QA and Data-to-Text.The analysis covers all 2,700 RAGTruth test samples and reports fewer complete localization failures.Source: G Analysis of Hard Case on RAGTruth… · p. 21, Figure 8 · p. 21

G.1 Performance on Short Spans

• For 0.5–1% logical-error spans, full SOPO scored 0.557 F1, versus 0.402 for Guidance-only and 0.309 for GRPO.For 0–0.5% numerical-error spans, the passage says the full Socratic chain offers limited marginal benefit.Source: Figure 9 · p. 21, §G.1 · p. 22

G.2 Performance on dense and sparse spans

• With exactly one hallucinated span, Guidance-only scored 0.556 F1 versus GRPO’s 0.559, while full SOPO scored 0.584.Full SOPO also achieved the highest performance in the ≥4-span setting; the passage attributes the sparse-case difficulty to suppressing false positives in otherwise factual text.Source: Figure 10 · p. 21, G Analysis of Hard Case on RAGTruth… · pp. 21–22, §G.2 · p. 22

G.3 Performance on Long Context

• Across context-length percentiles 50%–99.5%, full SOPO consistently outperformed GRPO and all ablated variants, while performance generally degraded as context length increased.Source: Figure 11 · p. 21, §G.3 · p. 22, G Analysis of Hard Case on RAGTruth… · p. 22

H Advantage Scaling

• Heuristic scaling after group normalization can break the zero-mean advantage property and introduce baseline bias; SOPO instead scales raw rewards before normalization.The paper reports severe instability around step 100 for post-normalization scaling.Source: H Advantage Scaling · pp. 22–23

I Calibration of RL4HS Scores

• RL4HS’s macro-averaged F1 is not directly comparable to the benchmark’s global micro-average, which aggregates counts across tasks.The passage notes that RAGTruth subtasks differ in instance counts and hallucination densities, and that RL4HS used 875 QA test instances rather than the official 900.Source: I Calibration of RL4HS Scores · p. 23

J Synergy of the Dual-Penalty Mechanism for Verbosity Control

• Turn-averaged span-F1 penalizes poor early responses, while pre-normalization advantage scaling favors shorter trajectories with similar average scores.Together, these mechanisms target early-turn reward solicitation and redundant interaction length.Source: J Synergy of the Dual-Penalty… · pp. 23–24