Source-linked AI summary
SelfCue: Making a 3D CT Report Generator Say What It Already Knows
Renjie Liang, Yang Yang, Jinqian Pan, Zhengkang Fan, Chengkun Sun, Jie Xu
TL;DR
CT report generators’ reports omit abnormalities that remain encoded in their hidden states, with the drop appearing when hidden states become text. SelfCue steers decoding with a predicted abnormality condition, and SelfCue-KD distills that behavior into a single-pass student.
Problem
CT report generators can get abnormalities wrong, while abnormality information remains in their hidden states rather than reaching the report.
Source: §1 · p. 1Method
SelfCue contrasts decoding with and without a predicted abnormality condition, then distills its reports into SelfCue-KD, which decodes without a condition.
Source: §1 · p. 1Results
Macro AUROC falls from 0.848 in hidden states to 0.739 in reports; SelfCue reaches 0.481 macro F1@0.5 and 0.510 GREEN on COLIPRI with Llama-3.1-8B.
Source: §1 · pp. 1–2Takeaways & Limitations
SelfCue-KD moves the correction into the weights, retaining most of the gain while using a single forward pass without an abnormality condition.
Source: §7 · p. 9SelfCue’s correction is bounded by its abnormality readout: with ground-truth labels as the condition, the same rule reaches 0.869 macro F1@0.5 against SelfCue’s 0.480 on COLIPRI with Llama-3.1-8B.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Progress in 3D CT report generation is usually sought in increasingly sophisticated architectures and larger pools of training data. We find instead that a 3D CT report generator already holds what its report leaves out, and loses it when the hidden state becomes tokens. Over the 18 CT-RATE abnormalities, this hidden-to-report surfacing gap is reflected by a drop in macro AUROC from 0.848 in the hidden states to 0.739 in the generated report. We propose SelfCue based on contrastive decoding. It promotes what the hidden state already supports and suppresses what it does not. It raises clinical efficacy F1 to 0.481 and the LLM-judged GREEN score to 0.510. Distilling that behaviour into the weights gives SelfCue-KD, a student that keeps most of the gain, needs nothing extra at inference, and drops into any pipeline already serving the baseline. Code is available at https://github.com/renjie-liang/SelfCue-CT.
1 Introduction
The generator retains abnormality information that its reports omit, and SelfCue uses that information to steer decoding. SelfCue-KD distills the correction into a student that decodes in one pass without a condition.
1 Introduction
- SelfCue uses contrastive decoding to promote abnormalities supported by the hidden state and suppress those it does not support.The condition is predicted for each volume rather than read from labels.Source: §1 · p. 1
- SelfCue-KD is trained on SelfCue-generated reports and decodes with a single forward pass, without a condition or contrast at inference.This moves the correction into the student’s weights.Source: §1 · pp. 1–2
- Macro AUROC drops from 0.848 in hidden states to 0.739 in generated reports, locating the surfacing gap at the transition from hidden states to text.The same pattern appears across two CT visual encoders and two language models.Source: §1 · p. 1, Figure 1 · p. 2
- The study measures abnormality information across the reporting pipeline and localizes its loss to text generation across two visual encoders and two language models.SelfCue-KD keeps most of SelfCue’s performance gain without depending on the abnormality condition.Source: §1 · p. 2
- SelfCue reaches 0.481 macro F1@0.5 and 0.510 GREEN on COLIPRI with Llama-3.1-8B.It contrasts output distributions with and without a predicted abnormality condition.Source: §1 · p. 2
2 Related work
Related work covers 3D CT report generation, contrastive decoding and representation intervention, and knowledge distillation. SelfCue combines a prompt-versus-no-prompt decoding rule with a student that does not receive the teacher’s condition.
2.1 3D CT report generation
- Prior CT report-generation work changes the visual pathway or adds retrieved studies or abnormality predictions; SelfCue starts from abnormality prediction and changes only the decoding rule.The section also distinguishes CT-AGRG, which generates one sentence per predicted abnormality.Source: §2.1 · p. 2
2.2 Contrastive decoding and representation intervention
- SelfCue contrasts a prompt with its absence, addressing a hidden-state-to-report gap while leaving the model’s states unchanged.The section contrasts this output-distribution approach with interventions that reshape the residual stream.Source: §2.2 · p. 3
2.3 Knowledge distillation
- Unlike cited distillation approaches whose students retain the condition at inference, SelfCue-KD receives no condition and distills a decoding rule rather than a prompt.It follows sequence-level distillation rather than token-distribution matching.Source: §2.3 · p. 3
3 Surfacing gap
The study scores abnormality information at each stage of the CT reporting pipeline using a shared ruler. The measured gap appears when hidden states become report text and recurs across alternative encoders and language models.
3.1 Probing every stage
- The evaluation uses AUROC over the same 18 CT-RATE abnormalities, with macro averages and a CheapCT probe for visual tokens and hidden states.Report probabilities come from RadBERT; the probe is trained on the training split and results are reported on CT-RATE validation.Source: §3.1 · p. 3
- The pipeline is scored at every second baseline-generator layer, averaging hidden states across visual and text prompt positions.AUROC depends on volume rankings and is unchanged by monotone transformations of scores.Source: §3.1 · p. 3
3.2 The gap along the pipeline
- Macro AUROC falls from 0.848 at the COLIPRI embedding to 0.739 in the report, while probed language-model layers remain between 0.839 and 0.848.The report-to-hidden-state gap is 0.109; it is 0.100 with VISD+CT-CLIP and 0.112 with MedGemma-27B.Source: §3.2 · p. 3
4 Method
SelfCue uses probe-derived textual abnormality conditions to steer contrastive decoding, then distills its reports into SelfCue-KD, a student that generates in a single pass.
4 Method
- SelfCue turns probe-recovered abnormalities into a textual condition and uses it to steer contrastive decoding; distilling its reports yields a single-pass student.The condition is predicted rather than read from labels.Source: §4 · p. 3
4.1 Self-cued decoding
- A thresholded probe readout becomes a short abnormality sentence; Mcond is trained with no-condition, ground-truth, and probe-accuracy prompts.The readout may come from hidden states or visual embeddings, and the condition is noisy rather than ground truth.Source: §4.1 · p. 4
- At each token, conditional and plain branches share the partial report, and contrastive logits promote tokens favored by the condition while demoting those favored by the plain branch.The contrast strength α is set to 0 when the condition mentions no abnormality; this means the readout found none, not that the volume is normal.Source: §4.1 · p. 4
4.2 SelfCue-KD
- SelfCue-KD uses out-of-sample probe conditions to generate teacher reports, then trains a condition-free student on complete reports with autoregressive cross-entropy for single-pass inference.Five disjoint folds ensure each training volume’s teacher condition comes from a probe that did not see it.Source: §4.2 · p. 4
5 Experiments
Experiments evaluate SelfCue on CT-RATE’s 18 abnormalities across encoder–backbone pairs, comparing clinical and report-quality metrics. SelfCue improves clinical efficacy over Mcond, while SelfCue-KD retains much of the gain with single-pass inference; further experiments examine decoding settings, condition sources, and held-out performance.
5.1 Setup
- Experiments use CT-RATE’s 18 abnormalities, two visual embeddings, and Llama-3.1-8B-Instruct or MedGemma-27B; the visual encoder and compressor are frozen during training.Clinical efficacy is measured with macro F1@0.5 and F1max, alongside macro AUROC and GREEN.Source: §5.1 · pp. 4–5
5.2 Main results
- On COLIPRI with Llama-3.1-8B, F1@0.5 rises from 0.395 to 0.430 to 0.481, while recall rises from 0.365 to 0.452 and precision moves from 0.590 to 0.577.SelfCue also holds GREEN at Mcond’s level on all three encoder–backbone pairs; its reports mention 2.40 abnormalities per volume versus Mcond’s 2.01.Source: Table 1 · p. 5, §5.2 · p. 6
- SelfCue-KD reaches 0.462 F1@0.5 and 0.534 F1max on COLIPRI with Llama-3.1-8B, and SelfCue exceeds eligible works on F1max at 0.542.SelfCue-KD retains most of its teacher’s gain and runs with a single forward pass; SelfCue is on par with the strongest eligible works on F1@0.5 and GREEN.Source: §5.2 · p. 6
5.3.1 Contrast strength and condition threshold
- Across all 28 α–τ settings, SelfCue exceeds Mbase and same-condition Mcond on F1max, and exceeds Mcond on F1@0.5.F1max stays flat through τ = 0.75 and falls beyond it; GREEN can fall as contrast strength increases, including 0.475 versus Mcond’s 0.491 at τ = 0.60.Source: §5.3.1 · p. 6, Figure 2 · p. 7
5.3.2 Gate on empty conditions
- The empty-condition gate sets α to 0 for volumes with no listed abnormality; it leaves F1@0.5 and F1max nearly unchanged while its GREEN effect grows with α and τ.On COLIPRI with MedGemma-27B, the gate raises GREEN by 0.027 at the operating point.Source: §5.3.2 · p. 7, Table 2 · p. 7
5.3.3 Condition source
- Changing the condition probe from hidden states to visual embeddings changes each measure by at most 0.008, while using cross-fitted conditions to train Mcond lowers F1max by 0.021.Training the student on teacher reports generated with injected noise lowers its F1max by 0.063.Source: §5.3.3 · p. 7, Table 3 · p. 8
5.3.4 Sequence-level distillation
- Sequence-level distillation is ahead of token-level distillation by 0.010 F1@0.5 and 0.002 F1max, but behind by 0.002 GREEN.SelfCue mentions 0.90 of listed abnormalities that are present and 0.86 of listed abnormalities that are absent; SelfCue-KD mentions 0.11 of present abnormalities omitted from the condition, versus Mcond’s 0.04.Source: §5.3.4 · pp. 7–8, Table 4 · p. 8
5.4 Selection on held-out data
- With 12% less training data, held-out runs retain Table 1-level performance, and SelfCue gains 0.059 F1@0.5 and 0.031 F1max over Mcond.Across three seeds, standard deviation is at most 0.012 on every measure.Source: §5.4 · p. 8
6 Discussion
SelfCue closes part of the gap between information in a generator’s hidden state and its report, but its readout still bounds what can be surfaced. Its benefits and limits reflect the different scope of clinical F1 and GREEN, while evaluation remains centered on CT-RATE’s 18 labelled abnormalities.
6 Discussion
- 0.869 macro F1@0.5 with ground-truth conditions versus 0.480 for SelfCue shows that the readout still bounds what a report can say.SelfCue closes about 60% of the distance between the report and the hidden-state probe, leaving some of the gap open to decoding.Source: §6 · p. 8
- SelfCue raises clinical content, especially for rarer abnormalities, while keeping report agreement with the reference level on GREEN.Macro F1 weights the 18 abnormalities equally; GREEN counts all report content, of which those abnormalities are a small part.Source: §6 · p. 8
- SelfCue closes part of the gap between what a model holds and says, building on activation-based gap measurements [Burns et al., 2023, Orgad et al., 2025, Goel et al., 2026].The discussion frames this as a setting where generated text is the clinical product, rather than an answer to be checked.Source: §6 · p. 8
- The evidence is limited to CT-RATE and its 18 labelled abnormalities; other imaging modalities and reader studies remain future work.The discussion also identifies richer annotations and iterative training as directions, while whether iterative rounds add gains remains open.Source: §6 · p. 9
7 Conclusion
The conclusion locates missing abnormality information at the transition from hidden state to text and presents SelfCue as a way to steer decoding from the model’s own state. SelfCue-KD moves the correction into model weights for single-pass deployment.
7 Conclusion
- Information about abnormalities survives the model’s layers and is lost when the hidden state becomes text.The conclusion describes a 3D CT report generator as holding more abnormality information than its report expresses.Source: §7 · p. 9
- SelfCue reads abnormality information from the generator’s hidden state and uses it to steer decoding.The conclusion characterizes this as raising clinical efficacy.Source: §7 · p. 9
- SelfCue-KD moves the correction into the weights, so the deployed model decodes in a single pass.The conclusion presents surfacing what a model already knows as complementary to stronger encoders and larger datasets.Source: §7 · p. 9
Supplementary Material SelfCue: Making a 3D CT Report Generator Say What It Already Knows
The supplementary terminology defines the prompts, model variants, and abnormality condition used to describe SelfCue and its distillation. The condition is an inference-time prediction, not a ground-truth annotation.
A Terminology
- The abnormality condition is a sentence listing abnormalities predicted present for a volume; its inference-time prediction is noisy, not ground truth.Source: A Terminology · p. 16
- A plain prompt is a report-generation instruction drawn per volume from CT-RATE’s 42 questions, without an abnormality condition.Source: A Terminology · p. 16
- A conditional prompt prepends that volume’s abnormality condition to its plain prompt.Source: A Terminology · p. 16
- Mbase is the baseline generator trained on CT volumes and reports under plain prompts; Mcond is warm-started from Mbase and trained with conditions on 70% of examples.The remaining 30% of Mcond’s training examples carry no condition, keeping the plain prompt in distribution.Source: A Terminology · p. 16
- Mteacher writes the reports used to train Mstudent, or SelfCue-KD, which uses plain prompts and no abnormality condition at inference.The plain and conditional branches are Mcond’s forward passes under the respective prompts.Source: A Terminology · p. 16
B.1 Probe implementation
The supplementary material describes how probe pooling, threshold selection, and a single-volume case study inform interpretation of the pipeline analysis. The threshold comparison concerns both system ranking and operating level, while the case illustrates where the condition limits correction.
B.1 Probe implementation
- The three probe-pooling approaches agree within 0.010 macro AUROC at their best layers, and the paper averages over all prompt positions.The approaches average all visual and text prompt positions, visual prompt positions alone, or use the last prompt position; causal attention lets generated positions attend to the whole prompt.Source: §B.1 · p. 16, Table 6 · p. 16
B.2 Thresholds and F1max
- System ordering survives global probability cuts from 0.01 to 0.95 and per-abnormality cuts, but F1max’s extra 0.03 to 0.05 requires varying cuts.The F1max-optimal cuts mostly sit far below 0.5, so F1max bounds ranking quality rather than marking an operating point; Figure 3 shows threshold effects and abnormality-specific curves.Source: §B.2 · p. 17, Figure 3 · p. 17
C.1 Case study
- In one validation volume, Mbase and Mcond omit consolidation, while SelfCue and SelfCue-KD mention it; hiatal hernia is absent from the condition and all four reports.The example shows the correction stopping where the readout does.Source: §C.1 · p. 17, Figure 4 · p. 17
C.2 Training curves
SelfCue’s gain over Mcond remains positive across checkpoints, while the size of the gain varies; F1@0.5 is less stable than F1max.
C.2 Training curves
- SelfCue improves over Mcond on F1@0.5 and F1max at every checkpoint.The checkpoints shown are for the conditional runs; open circles mark the checkpoint used in the paper’s other results.Source: §C.2 · pp. 17–18, Figure 5 · p. 18
- Across Mbase checkpoints, F1@0.5 varies three to four times as much as F1max on all three pairs.On COLIPRI with Llama-3.1-8B, the standard deviations are 0.037 for F1@0.5 and 0.011 for F1max.Source: §C.2 · p. 18
- F1@0.5’s fixed threshold tracks how often a report mentions an abnormality, whereas F1max chooses a threshold for each abnormality.Source: §C.2 · p. 18
C.3 Clinical efficacy
SelfCue’s clinical-efficacy gains come mainly from recall and are concentrated among rare abnormalities; the other-pair sweep and one-pair oracle run probe the method’s settings and readout.
C.3 Clinical efficacy
- On COLIPRI with Llama-3.1-8B, SelfCue raises recall from 0.338 to 0.452 over Mbase, mostly without mentioning more abnormalities per volume.Its precision and recall land near the condition’s own, indicating it recalls different abnormalities rather than simply listing more.Source: §C.3 · p. 18, Table 7 · p. 19
- On two Llama-3.1-8B pairs, SelfCue matches more findings and misses fewer than Mbase, while on COLIPRI with MedGemma-27B its fewer false findings are offset by fewer matches and more misses.GREEN rises on the two Llama pairs but remains unchanged on COLIPRI with MedGemma-27B; report length moves with the conditional stage rather than the contrast.Source: §C.3 · p. 18, Table 8 · p. 19
- SelfCue’s gain over Mbase concentrates on the six rarest abnormalities: 0.105 versus 0.039 and 0.037 on COLIPRI with Llama-3.1-8B.Across the three pairs, prevalence and gain have rank correlations from −0.51 to −0.65; the surfacing gap is also widest for the rarest group, but separates groups less.Source: §C.3 · p. 18
C.4 The sweep on the other pairs
- On both other pairs, every α–τ grid cell clears its own Mbase and α = 0 F1max baselines, and τ moves the metric more than α.The peak is at τ = 0.70 on VISD+CT-CLIP, versus 0.75 on the other two pairs.Source: §C.4 · p. 18
C.5 The oracle ceiling
- On one pair, replacing the probe condition with ground-truth labels raises Macro F1@0.5 from 0.480 to 0.869, while GREEN stays unchanged.The decoding rule, checkpoint, and gate are unchanged, pointing to the readout rather than the rule as the method’s limit in this run.Source: §C.5 · p. 19
C.6 Student training choices
Student training is relatively insensitive to initialization, while teacher-report selection has a larger effect at the low-scoring end; these ablations use an earlier teacher corpus.
C.6 Student training choices
- The student’s starting point changes F1@0.5 by less than 0.01, and training from scratch comes within 0.01 of starting from Mcond.Source: §C.6 · p. 19, Table 11 · p. 21
- Keeping only the teacher’s worst half costs 0.016 against the full corpus, while keeping only its best half adds no gain.Dropping teacher reports scored below 0.2 improves over the full corpus by 0.003 F1@0.5 and 0.005 F1max.Source: §C.6 · p. 19, Table 11 · p. 21
- These student variants were trained on an earlier teacher corpus written at α = 0.5 with no threshold, so their rows compare with one another, not the paper’s reported systems.Source: §C.6 · p. 19