Source-linked AI summary
RefineRank: Joint Box Refinement and Ranking for Surgical Spatio-Temporal Grounding
Linzhe Jiang, Jiayuan Huang, Changhao Zhang, Chunyang Jiang, Zhehua Mao, Mobarak I. Hoque
TL;DR
Surgical STG needs precise boxes that answer procedural questions, but language models and open-set detectors provide complementary and incomplete capabilities. RefineRank uses RefineNet to correct and score detector candidates while keeping both backbones frozen, and its built-in joint ranking is the strongest evaluated selection policy. The method achieves the highest displayed STG score in the cited MedVidBench rankings and improves controlled selection performance, while remaining limited by detector coverage and evaluation scope.
Problem
Surgical STG requires precise object boxes at question-specified times, but MedVLM coordinates can be imprecise and detector confidence does not measure question relevance.
Method
RefineRank connects a frozen MedVLM and frozen GroundingDINO through trainable RefineNet, which predicts candidate corrections and quality scores before fixed joint-pool decoding.
Results
RefineRank’s built-in decoding rule is the strongest evaluated selection policy, while separately trained selectors on fixed RefineNet outputs do not improve it.
Takeaways & Limitations
A compact box-level module can combine question understanding with detector localization without retraining either backbone.
Takeaways & Limitations
RefineRank cannot recover targets absent from GroundingDINO proposals, and the comparison does not establish superiority over other learned fusion designs.
Abstract
from arXiv · showhide
Surgical spatio-temporal grounding (STG) requires locating, at each video time specified by a procedural question, the object that the question asks about. Existing approaches face a trade-off: vision language models understand the question context but produce imprecise coordinates, whereas open-set detectors provide localized candidate boxes whose confidence does not reflect which box answers the question. We introduce RefineRank, which closes this gap at the candidate-box level. A compact trainable module, RefineNet, combines the language and regional features of a frozen medical vision language model with the proposals of a frozen open-set detector: it predicts a bounded coordinate correction and a quality score for every candidate box, and a fixed decoding rule returns the original or refined box with the highest score. On the MedVidBench Official Rankings (Verified), RefineRank records 0.421 STG mIoU, the highest displayed STG score, while its global multi-metric rank is 11. In a controlled evaluation on separate training and evaluation videos, coordinate correction raises the candidate oracle upper bound from 0.6772 to 0.7302, and ranking the joint pool of original and refined candidates by their RefineNet scores improves STG mIoU from 0.2719 to 0.4534, whereas separately trained selectors over the same pool reach at most 0.4186. These results show that a small box-level module can reconcile question understanding with precise localization without retraining either backbone. Code is available at [https://github.com/linzhe001/RefineRank](https://github.com/linzhe001/RefineRank).
1 Introduction
Surgical STG must connect procedural language and requested times with precise object localization, but MedVLMs and open-set detectors each solve only part of this problem. RefineRank addresses the gap by refining and ranking detector candidates using frozen backbones and a trainable box-level module.
- Problem: Surgical STG returns one object box at each requested video time, including for small, occluded, or visually similar surgical targets.The task jointly involves spatial, temporal, and language interactions.
- Motivation: MedVLMs interpret complex clinical questions, whereas detector confidence measures query matching rather than whether a candidate answers the complete question.These model families therefore provide complementary capabilities.
- Motivation: Candidate boxes provide a compact connection between frozen models because they already contain coordinates, detector scores, and corresponding MedVLM regional features.This avoids learning dense alignment across incompatible feature spaces.
- RefineRank: RefineNet jointly predicts a bounded coordinate correction and a question-relevant quality score for every detector candidate.The MedVLM and GroundingDINO backbones remain frozen.
- RefineRank: RefineRank decodes the highest-scoring candidate from the joint pool of original and refined boxes using a fixed rule with no separate selector.The rule uses RefineNet scores to combine refinement and selection.
- Evaluation: RefineNet corrections raise the candidate localization upper bound, while its scores improve final selection over detector confidence and separately trained selectors.A gap to the candidate oracle remains.
2 Related Work
Prior work develops direct spatio-temporal grounding, grounded vision-language models, medical video grounding, and localization-quality methods. RefineRank instead combines two frozen models through RefineNet, which refines and ranks detector boxes before fixed decoding.
- Spatio-temporal video grounding: Spatio-temporal grounding systems learn dense video-language interactions or directly predict tubes, whereas RefineRank selects boxes at known requested timestamps.It does not predict an entire tube directly from video tokens.
- Grounded vision language models: Grounded vision-language models train spatial grounding inside a single model using location tokens, coordinates, or continuous region features.RefineRank instead connects two frozen models after detector proposal generation.
- Medical video grounding: MedGRPO introduced MedVidBench and its medical vision-language checkpoint, which RefineRank uses alongside a separate frozen GroundingDINO detector.The MedVLM + GroundingDINO baseline ranks proposals by detector confidence without trained fusion.
- Localization quality and box refinement: Localization-quality methods distinguish proposal localization from ranking, a distinction also central to surgical STG.Examples include progressive proposal refinement and localization-confidence prediction.
- RefineRank: RefineRank’s pipeline uses frozen MedVLM and GroundingDINO components, trainable RefineNet, and fixed decoding over original and refined candidates.The ranking head scores both versions, while the box head predicts corrections.
3 Method
RefineRank connects frozen MedVLM and GroundingDINO backbones through trainable RefineNet, which refines candidate boxes and scores both original and refined alternatives. A fixed decoding rule selects from the retained joint candidate pool without a separate learned selector.
- Architecture: RefineRank keeps MedVLM and GroundingDINO frozen while RefineNet uses question, regional, and detector features to produce box corrections and quality scores.The model scores original and refined versions of each detector proposal.
- Box refinement: Each refined box applies bounded center offsets and logarithmic scale offsets to an original box before conversion to clipped corner coordinates.Center offsets lie in [-0.5, 0.5], while scale offsets lie in [-log 2, log 2].
- Training: RefineNet trains separate logits for original and refined candidates using IoU-based ranking objectives, with box supervision applied to up to eight highest-IoU original boxes.The total objective combines two ranking losses with a box loss, while padded boxes are masked.
- Candidate pool and decoding: Inference retains both versions of candidates, keeps 16 original boxes, and greedily fills a pool of up to 48 candidates using quality and diversity.The diversity term favors candidates that do not duplicate already selected locations, with λq = 0.7 and λd = 0.3.
- Candidate pool and decoding: At each requested time, a fixed rule returns the retained original or refined candidate with the highest corresponding RefineNet score.The rule has no learned parameters and uses no separate selector module.
4 Experimental Protocol
The controlled study uses video-separated training and evaluation data, with frozen detector and MedVLM outputs prepared before RefineNet training. Comparisons evaluate direct MedVLM prediction, detector-confidence selection, RefineRank, oracle candidate quality, and separately trained selectors.
- Controlled split: The controlled study spans 30 videos and 3,300 examples, divided into 23 training videos with 2,346 examples and 7 evaluation videos with 954 examples.The split covers CholecTrack20, CoPESD, and EgoSurgery.
- Precomputation: Detector queries, original boxes, and MedVLM features are precomputed before RefineNet training, and the same frozen inputs are used throughout comparisons.Padding is excluded from evaluation, and refined boxes are added without another backbone pass.
- Baselines and selectors: The comparison includes direct MedVLM coordinates, detector-confidence selection from original boxes, RefineRank scoring of the joint pool, and separately trained selector analysis.No additional selector is trained for RefineRank’s prediction.
- Evaluation metrics: STG mIoU is measured at annotated requested times, while the candidate oracle selects the available box with greatest IoU for diagnostic upper-bound analysis.The oracle uses original boxes for the baseline and the union of original and refined boxes for RefineRank.
- Public benchmark: The public comparison uses the MedVidBench Official Rankings (Verified) snapshot and orders verified entries by STG mIoU while also reporting global multi-metric rank.The snapshot was accessed on 15 August 2026.
5 Results
RefineRank achieves the highest displayed STG mIoU in the verified MedVidBench snapshot and improves both candidate localization potential and box selection in controlled evaluation. Its joint-pool ranking outperforms detector-confidence selection and separately trained selectors, although an oracle gap remains.
- 5.1 MedVidBench Official Benchmark: 0.421 STG mIoU places RefineRank first among the five highest verified MedVidBench entries, while its global leaderboard rank is 11.The global ranking averages ranks over ten metrics.
- 5.2 Does Refinement Create Better Candidate Boxes?: 0.7302 candidate-oracle STG mIoU follows refinement, up from 0.6772 using original GroundingDINO boxes alone.The oracle selects the best available box with annotations at each requested time.
- 5.3 Does the Learned Scoring Function Select Better Boxes?: 0.2719 STG mIoU from detector-confidence selection trails the 0.6772 candidate oracle over the same original boxes.Useful boxes are often present but not ranked highest for the complete surgical question.
- 5.3 Does the Learned Scoring Function Select Better Boxes?: 0.4534 STG mIoU from RefineNet scoring the joint original-and-refined pool improves the baseline by 0.1815, while a 0.7302 oracle gap remains.The largest improvements occur on CholecTrack20 and CoPESD.
- 5.4 Do Separately Trained Selectors Improve RefineRank?: 0.4186 is the strongest result among separately trained selector ablations, below RefineRank’s 0.4534 on the same RefineNet-generated candidate pool.The selector study uses the same joint pool and argmax decoder, with RefineNet frozen for every row.
6 Ablations and Qualitative Analysis
The ablations separate candidate localization from selection and show that RefineRank’s joint candidate pool and fixed scoring rule are supported by complementary correction and ranking behaviors. Qualitative and diagnostic analyses also identify boundaries around response-map interpretation, proposal coverage, and evaluation scope.
- Ablations: RefineNet-generated corrections raise the candidate oracle upper bound, while its scores improve final selection over detector confidence and separately trained selectors.The candidate oracle measures available localization potential, whereas the final prediction still reflects both localization and ranking errors.
- Input Feature Study: 0.4044 is the reported average after adding qlast, up from 0.2767, with intermediate visual features providing a smaller additional gain.Block 23 has the highest equally weighted average in this diagnostic study, but blocks 7, 15, and 23 remain closely grouped and vary by dataset.
- Input Feature Study: Table 5 re-scores the joint pool of original and refined boxes with a separately trained MLP, excluding RefineNet scores from both inputs and selection.The study changes one feature input at a time after RefineNet is trained and frozen.
- Spatial Response Across Depth: Figure 2 shows localized responses around displayed tools in intermediate blocks, while block 31 is less spatially differentiated in these examples.The maps are independently normalized and descriptive rather than calibrated confidence, attention, or causal explanations.
- Qualitative Analysis: Figure 3 illustrates complementary roles: refinement improves one hand localization, whereas higher scores select better original detector boxes in several examples.The two shared failures indicate that incorrect or insufficient proposals can prevent correct selection, and the examples do not estimate behavior frequency.
- Limitations: Evaluation uses one fixed video-separated split, and no learned MedVLM–GroundingDINO fusion baseline was trained or evaluated.Thus, the controlled comparison does not establish superiority over other learned fusion designs; additional splits are left for future work.
- Limitations: RefineRank cannot recover targets absent from GroundingDINO proposals, and refined-box scores use features pooled only at the original detector coordinates.Candidate-oracle values use target annotations and represent an upper bound.
7 Conclusion
RefineRank combines frozen MedVLM question understanding and frozen GroundingDINO proposals through a compact trainable RefineNet. Its fixed ranking rule selects from original and refined boxes, while separate selectors do not improve it and the remaining oracle gap reflects localization and ranking errors.
- Conclusion: RefineRank combines frozen MedVLM and GroundingDINO backbones with a compact trainable RefineNet that jointly learns box scores and corrections.The decoder ranks the joint pool of original and refined candidates and returns the highest-scoring box without an additional selector module.
- Conclusion: The built-in RefineRank decoding rule is the strongest evaluated selection policy, while separately trained selectors do not improve fixed RefineNet outputs.The remaining gap to the candidate oracle reflects both localization and ranking errors.
- Conclusion: Further evaluation should cover other video sets and learned fusion baselines.These directions address the scope boundaries identified for the current evaluation.