Source-linked AI summary
Photon: Speedup Volume Understanding with Efficient Multimodal Large Language Models
Chengyu Fang, Heng Guo, Zheng Jiang, Chunming He, Xiu Li, Minfeng Xu
TL;DR
Scaling multimodal language models to 3D medical imaging remains costly, and prior slice-based or fixed-length approaches can compromise volumetric detail. Photon uses instruction-conditioned scheduling and surrogate-gradient propagation to adaptively compress 3D volume tokens during training and inference, achieving state-of-the-art medical visual question answering performance with lower computational overhead. Prospective clinical validation and human-centric studies remain future needs.
Problem
3D medical visual question answering is hindered by high computational demands, while slice-based and fixed-length compression approaches can disrupt volumetric continuity and obscure subtle findings.
Method
Photon models complete 3D volumes with variable-length tokens, using instruction-conditioned token scheduling and surrogate-gradient propagation for adaptive differentiable token reduction.
Results
Photon achieves state-of-the-art performance on multiple medical visual question answering tasks while significantly accelerating training and inference.
Takeaways & Limitations
Photon reduces computational and memory costs while preserving clinically critical information across medical visual question answering benchmarks.
Takeaways & Limitations
Prospective clinical validation and human-centric user studies with medical practitioners remain necessary future work.
Abstract
from arXiv · showhide
Multimodal large language models are promising for clinical visual question answering tasks, but scaling to 3D imaging is hindered by high computational costs. Prior methods often rely on 2D slices or fixed-length token compression, disrupting volumetric continuity and obscuring subtle findings. We present Photon, a framework that represents 3D medical volumes with token sequences of variable length. Photon introduces instruction-conditioned token scheduling and surrogate gradient propagation to adaptively reduce tokens during both training and inference, which lowers computational cost while mitigating the attention dilution caused by redundant tokens. It incorporates a custom backpropagation rule with gradient restoration to enable differentiable optimization despite discrete token drop. To stabilize token compression and ensure reliable use of visual evidence, Photon further applies regularization objectives that mitigate language-only bias and improve reliability. Experiments on diverse medical visual question answering tasks show that Photon achieves state-of-the-art accuracy while reducing resource usage and accelerating both training and inference.
1 Introduction
Photon addresses the cost and fidelity challenges of 3D medical visual question answering with variable-length, instruction-aware token reduction. Its scheduling and surrogate-gradient mechanisms support efficient training and inference while retaining strong clinical-task performance.
- 3D volume modeling is computationally demanding, while slice-based pipelines can disrupt spatial continuity, remove volumetric detail, and introduce frame-selection bias.
- Photon represents complete 3D medical volumes with variable-length token sequences, avoiding slice sampling and fixed-length token compression.
- Instruction-conditioned Token Scheduling estimates instruction-aware saliency and sample-specific thresholds to adaptively retain informative visual tokens.
- Photon removes unimportant tokens, caches, and positional encodings during forward computation while regularization mitigates language-only hallucination bias.
- Surrogate Gradient Propagation combines discrete token reduction in the forward pass with surrogate-gradient optimization in the backward pass, enabling differentiable training.
- Photon significantly accelerates training and inference while achieving state-of-the-art performance on multiple medical visual question answering tasks.
2 Related Work
Prior medical vision-language systems commonly use slice-based processing or fixed-length visual embeddings, limiting volumetric context, scalability, and fine-grained spatial detail. Visual-token reduction methods improve efficiency but often overlook instruction-specific information needs.
- Many medical vision-language models retain slice-based processing, which undermines volumetric context and risks spatial discontinuity.
- OmniV-Med supports variable-length sequences but uses L1 similarity of slice features, which can remove subtle clinically important pathologies.
- Existing visual-token reduction methods reduce redundancy and improve efficiency, yet often ignore that different instructions require different amounts of information.
3 Methodology
Photon models full 3D medical volumes with instruction-conditioned, variable-length token sequences. Its scheduling, surrogate-gradient, and regularization mechanisms adaptively prune tokens while preserving trainability and visual grounding.
- Instruction-conditioned Token Scheduling: Photon dynamically adjusts 3D visual-token sequences using instruction-aware saliency and instance-specific thresholds.The framework applies Instruction-conditioned Token Scheduling to adapt retention to both the instruction and each scan's saliency distribution.
- Instruction-conditioned Token Scheduling: Instruction-aware saliency weights visual tokens by their alignment with central instruction tokens, producing normalized rankings for adaptive thresholding.Instruction-token centrality is computed from positive affinities, then used to score visual-token relevance.
- Instruction-conditioned Token Scheduling: A learned threshold converts normalized saliency into retention probabilities and an approximately binary mask, enabling instance-specific token selection.The threshold predictor summarizes saliency distributions and maps them to a smooth probabilistic threshold before binarization.
- Token Selection: Photon hard-drops unselected tokens and their associated caches, attention masks, and positional embeddings to reduce the forward-pass sequence.The resulting compressed sequence follows discrete selection while removing the full computational state of discarded tokens.
- Surrogate Gradient Propagation: Surrogate Gradient Propagation restores gradients through retained activations and retention probabilities, making discrete pruning trainable.The straight-through path preserves hard forward selection while allowing the threshold predictor and token-selection process to receive optimization signals.
- Surrogate Gradient Propagation: Gradient-based proxy scores connect token activations to loss gradients, guiding retention probabilities toward task-related visual evidence.The proxy values are standardized, clipped, and mapped to relative retention tendencies for surrogate-gradient construction.
- Objective: Robustness regularization discourages correct predictions under inconsistent visual retention, reducing language-only hallucination bias.With perturbed volumes or masks, the objective encourages uncertainty when visual evidence is insufficient and reinforces reliance on the intended pruning structure.
4 Experiments
Photon improves medical visual question answering across multiple benchmarks while reducing computational and memory costs. Its adaptive token pruning preserves clinically relevant information, and ablations show that instruction-conditioned scheduling, surrogate gradients, and regularization contribute to performance and efficiency.
- Comparative Evaluation: Photon achieves the best finetuned results across all 3D-RAD tasks, with gains of about 14.0% in anomaly detection and image observation over the best baseline.Medical measurement improves by about 7.3%, while longitudinal temporal diagnosis rises by around 3.0%.
- Comparative Evaluation: On DeepTumorVQA, Photon leads both multi-choice and free-text settings, improving total averages by about 3.6% and approximately 11.5%, respectively.Measurement subtypes improve by more than 35.3%, and visual reasoning subtypes show margins exceeding 20.7%.
- Visualization Analysis: Photon retains question-relevant structures while discarding non-essential context, adapting its pruning to clinical focus such as pleural effusion or kidney cysts.Figure 3 marks reduced tokens in white and retained clinically relevant regions with purple boxes.
- Ablation Study: Fully finetuning the vision encoder and MLP aligner causes overfitting, degraded performance, and inconsistent answers, whereas Phase 1 alignment improves key metrics.Phase 1 improves BLEU by 70% on anomaly detection and 134% on image observation, while existence detection accuracy rises by 44%.
- Ablation Study: Removing ITS and SGP lowers static temporal diagnosis accuracy by more than 3%, increases retained tokens, and slows training.Removing regularization also reduces accuracy, destabilizes training, and undermines result reliability.
- Efficiency: Compared with finetuned Qwen2.5-VL, Photon reduces inference memory usage by about two-thirds and achieves over a five-fold speedup.Inference-speed gains are more modest when using KV Cache but remain steady.
5 Discussion and Conclusion
Photon is an efficient 3D medical volume-understanding framework that combines adaptive token reduction with differentiable training. Across medical visual question answering benchmarks, it achieves state-of-the-art performance with fewer tokens and lower computational overhead.
- Discussion and Conclusion: Photon directly operates on 3D scans and combines Instruction-conditioned Token Scheduling with Surrogate Gradient Propagation for adaptive token reduction during training and inference.ITS selects informative tokens using instruction-vision interactions, while SGP supports backpropagation through discrete token dropping.
- Discussion and Conclusion: Photon achieves state-of-the-art performance across medical visual question answering benchmarks while using substantially fewer tokens and less computational overhead.
A Methodology
Photon uses a lighter two-stage training strategy and combines instruction-conditioned token scheduling with surrogate-gradient optimization for adaptive, differentiable token pruning. Its threshold features, gradient dynamics, and token-importance proxy are designed to preserve informative visual evidence while reducing computation.
- Training strategy: Photon replaces the usual three-stage pipeline with a lighter two-stage strategy that first aligns visual embeddings and then fine-tunes the complete task model.Phase 1 trains the 3D patch embedding layer while retaining pretrained backbone and decoder parameters; Phase 2 adds Photon modules and fine-tunes all modules.
- Token scheduling: Instruction-conditioned token scheduling constructs normalized saliency scores and instance-specific thresholds from complementary distributional features.The feature construction uses normalized-score statistics, raw-logit statistics, and a signed-log descriptor before thresholding and selection.
- Token scheduling: The scheduling mechanism uses scale and variability statistics from raw logits together with compressed signed-log indicators to characterize token-retention behavior.Raw-logit features preserve absolute level and variability, while the signed-log view summarizes large magnitudes without discarding sign.
- Surrogate optimization: Photon applies discrete token dropping in the forward pass and surrogate gradient optimization in the backward pass, enabling differentiable training.A straight-through mechanism restores gradients to retention probabilities, while saturation control prevents vanishing gradients near probability extremes.
- Surrogate optimization: The surrogate dynamics reinforce informative tokens toward retention and progressively prune less informative tokens using activation-gradient signals.The method models token removal by zeroing hidden states, approximates loss variation with a first-order Taylor expansion, and uses the activation-gradient product as a proxy.
- Optimization details: The implementation standardizes and clips token-level proxy signals across active visual positions for stable optimization and applies FlashAttention2 throughout attention layers.The reported configuration uses AdamW, BF16 precision, Deepspeed ZeRO2, and fixed hyperparameters for the surrogate-gradient procedure.
B.2 Tasks, Datasets and Metrics
The evaluation combines large 3D CT VQA benchmarks with medical imaging and video-oriented datasets, using task-specific generation, classification, and numerical metrics. Experiments examine training stability, clinical metrics, and the behavior of Photon’s retention mechanism.
- Datasets: Phase 2 training uses 3D-RAD and DeepTumorVQA, covering descriptive, recognition, measurement, and reasoning-oriented diagnostic scenarios.The benchmarks include both open- and closed-ended questions under clinically relevant protocols.
- Datasets: 3D-RAD contains 16,188 CT scans from 11,255 patients and provides six tasks with 33,910 curated benchmark QA pairs and 136,195 training pairs.Its tasks span anomaly detection, observation, measurement, existence detection, and static and longitudinal temporal diagnosis.
- Datasets: The evaluation also includes SLAKE, NExT-QA, and MedFrameQA to test bilingual medical VQA, causal and temporal video reasoning, and multi-image clinical reasoning.MedFrameQA contains 2,851 items built from 9,237 frames across 3,420 instructional medical videos.
- Metrics: Free-text tasks use BLEU, ROUGE, and BERT Score, categorical tasks use Accuracy, and numerical answers additionally use Mean Relative Accuracy.The metrics target fluency, lexical overlap, semantic similarity, classification correctness, and numerical prediction quality.
- Training stability: The soft retention band mainly acts in extreme cases by pulling abnormal retention ratios toward a stable range and preventing collapse.Under normal conditions, most retention probabilities remain within the expected range and receive little corrective pressure.
- Training stability: Removing robust regularization produces instability because limited 3D medical visual priors can cause systematic underestimation of important visual tokens.The figure comparison examines retention-band triggers and kept-token counts under different regularization settings.
- Training stability: The staged schedule keeps retention at 1.0 during warmup, introduces soft masking before reduction, and then stabilizes hard-pruning behavior after an adaptation period.This schedule is intended to prevent premature pruning while the threshold predictor learns.
- Clinical evaluation: Photon’s clinical evaluation reports sensitivity, specificity, and accuracy, with repeated seeds and bootstrap intervals used to assess stability and uncertainty.The authors report that Photon exceeds the strongest baseline across ten seeds and that 95% bootstrap intervals remain narrow.
B.6 Zero-shot and out-of-domain evaluation
Photon is evaluated under institution-level distribution shift and in a training-free zero-shot setting. It retains most accuracy under out-of-domain evaluation while reducing the number of visual tokens substantially in both settings.
- Institution-level OOD evaluation: The out-of-domain evaluation redivides DeepTumorVQA by medical institution, inducing changes in population, devices, and imaging protocols.Results are reported for both free-text and multiple-choice variants together with average retained visual tokens.
- Institution-level OOD evaluation: Accuracy drops by approximately 0.038 and 0.049 under the OOD split while Photon retains only 0.34K–0.40K visual tokens per sample.The authors interpret this as preservation of most accuracy under institution-level distribution shift without destabilized cross-domain generalization.
- Zero-shot evaluation: The training-free zero-shot experiment attaches token scheduling to Qwen2.5-VL 3B without additional medical fine-tuning and compares it with the unmodified backbone.The evaluation uses the full MedFrameQA dataset.
- Zero-shot evaluation: Photon reduces the average zero-shot token keep rate from 7.00K to 3.28K while maintaining almost identical accuracy.Because the scheduling module operates independently of backbone parameters, the result supports preservation of zero-shot behavior without task-specific fine-tuning.
B.7 More Ablation Study
Additional experiments test pruning sensitivity, noisy-threshold robustness, transfer across backbones and modalities, and typical failure modes. Photon generally reduces visual tokens substantially while preserving comparable task performance, but rare structures and precise small-lesion measurements remain difficult.
- Ablations: Varying τce, β, and ϵsat mainly changes the number of retained tokens while leaving performance stable across the tested 3D-RAD subtasks and generative metrics.The ablation supports the robustness of the default configuration across pruning-related hyperparameters.
- Robustness: Random threshold perturbations of ±10% cause only mild degradation, indicating tolerance to moderate threshold-estimation errors.The noisy-threshold experiment tests multiplicative perturbations at inference time.
- Scaling: Photon transfers across Qwen2.5-VL backbone sizes and to Qwen3-VL 2B while maintaining consistent pruning behavior without training collapse.Extending from 3B to 7B improves performance across tasks with similar retention ratios.
- Cross-modal evaluation: On MedFrameQA and SLAKE, Photon reduces average keep rates to about 0.43 and 0.38, respectively, while keeping accuracy nearly unchanged.On SLAKE, the pruned model has only one fewer correct case out of 489 than the non-pruned baseline.
- Video reasoning: On NExT-QA, Photon cuts the average token keep rate by more than half while maintaining essentially unchanged accuracy across causal, descriptive, and temporal questions.The video setting removes the medical warmup stages and robustness regularization, training the pruning module directly with the task loss.
- Cross-modal evaluation: Across CT, MRI, multimodal medical frames, and videos, Photon consistently reduces visual tokens by more than half while keeping accuracy comparable to non-pruned baselines.This cross-benchmark pattern supports transfer across diverse modalities and task formats.
- Failure modes: Photon’s qualitative failures concentrate on fine-grained quantitative estimation and rare structural patterns, including small-nodule measurement and invisible-lung cases.A reported nodule estimate is 4×3 mm versus the 5×3 mm ground truth, while rare invisible-lung anatomy can receive lower saliency.
- Failure modes: Rare or anatomically unusual findings are harder to retain reliably because long-tail imbalance makes uncommon structures less salient during attention and pruning.In 3D-RAD existence detection, some findings have only a few hundred positive cases despite more than one hundred thousand total cases.
C.2 Statistical Analysis on Token Pruning
Photon’s token pruning varies with instruction and task semantics rather than using a fixed global rate, while retained regions change noticeably across organ-related prompts and are evaluated against lung anatomy.
- Table 11 reports task-conditioned lung token retention alongside organ-wise Jaccard dissimilarity.
- Medical Measurement retains the most lung tokens, whereas Existence Detection retains the fewest, demonstrating task-dependent rather than globally fixed pruning.The comparison concerns Image Observation, Anomaly Detection, Medical Measurement, and Existence Detection for lung-related instructions in 3D-RAD.
- Retained-region masks show structured instruction-dependent changes, with organ-pair Jaccard dissimilarity values ranging from 0.52–0.69.Jaccard dissimilarity is defined as 1 minus mask overlap.
- Photon’s lung-related pruning is assessed by measuring block-level overlap between retained tokens and reference lung segmentations on 3D volumes.Reference masks come from pretrained segmentation models and are resampled to the pruning module’s token grid.
D Limitations and Future Works
The paper identifies prospective clinical validation and human-centered studies as necessary future work, alongside broader applications and extensions beyond current VQA tasks.
- Limitations: Prospective clinical validation and human-centric studies with medical practitioners remain necessary despite Photon’s robust empirical performance.
- Future Works: Future research may integrate temporal and longitudinal imaging to model disease progression coherently.
- Future Works: Photon could extend beyond VQA to cross-institutional generalization, interactive diagnostic support, and multimodal pretraining across diverse modalities.
- Future Works: Future extensions include auxiliary diffusion modules for more intuitive visual results and combining tasks such as segmentation.
- Data and Ethics: The study used authorized, de-identified medical datasets under applicable licensing and ethical standards.
F Reproducibility Statement
The study documents its architecture, objectives, procedures, optimization settings, experimental configurations, and evaluation details to support reproducibility.
- The authors provide model, training, optimization, hyperparameter, evaluation, and implementation details for replication.Source code and evaluation agreements are planned for release after institutional review and authorization.