Source-linked AI summary
Retrieval-Augmented Visual Prompting: Guiding Foundation Models in Two-Photon Imaging
Salvatore Calcagno, Marco Finocchiaro, Giovanni Bellitto, Daniela Giordano, Concetto Spampinato, Federica Proietto Salanitri
TL;DR
Two-photon calcium imaging challenges foundation models because appearance varies across recordings, annotations are scarce, and rapid adaptation is needed. The paper introduces RAVP, which injects retrieved annotated exemplars into SAM 3’s input instead of adapting model weights. On the Allen Brain Observatory, exemplar-augmented inference strengthens zero-shot neuron detection and instance segmentation, with one carefully selected exemplar outperforming multiple retrieved examples.
Problem
Robust neuron segmentation is difficult because neuronal appearance and recording conditions vary substantially, while annotations are scarce and target-specific prompting is limited.
Method
RAVP augments each target tile with retrieved annotated exemplar crops whose bounding boxes serve as concept prompts for SAM 3, without modifying model weights or architecture.
Results
Exemplar-augmented inference improves zero-shot detection and segmentation, while a single carefully selected exemplar can recover much of the gap to fine-tuned models and outperform multiple retrieved examples.
Takeaways & Limitations
Inference-time visual memory injection provides a competitive alternative to parameter adaptation for specialized biomedical imaging.
Takeaways & Limitations
The strongest Recall Predictor variant requires exemplar-library annotation, SAM 3 inference for recall supervision, and predictor training, imposing non-negligible cost.
Abstract
from arXiv · showhide
Two-photon calcium imaging presents a challenging setting for foundation models: image appearance varies substantially across recordings and experimental conditions, annotations are scarce, and rapid adaptation is often needed. Rather than adapting model weights through fine-tuning, we ask whether a foundation model can be guided at inference time by injecting external visual memory directly into its input. We implement this idea with SAM 3 and introduce Retrieval-Augmented Visual Prompting (RAVP), a framework in which each target tile is augmented with a retrieved annotated exemplar whose bounding box is used as a concept prompt. RAVP turns retrieval into a form of visual prompting and enables adaptation through input design alone. We study multiple exemplar selection strategies, including fluorescence-guided heuristics and a lightweight recall predictor trained to estimate which exemplar is most informative for a target tile. Experiments on the Allen Brain Observatory show that exemplar-augmented inference consistently strengthens zero-shot neuron detection and instance segmentation. Ablation studies further show that a single carefully selected exemplar is more effective than prompting with multiple retrieved examples. These results position inference-time visual memory injection as a simple and effective alternative to parameter adaptation for foundation models in specialized biomedical imaging.
1 Introduction
Two-photon calcium imaging requires robust neuron segmentation, but variable neuronal appearance and recording conditions make generalization difficult. The paper introduces RAVP, which guides SAM 3 at inference time by composing target tiles with retrieved annotated exemplars, avoiding weight or architecture changes.
- Neuron segmentation is a critical preprocessing step for assigning pixels to cells and conditioning downstream neural-dynamics analyses.
- Bright active neurons contrast with nearly invisible weakly active or silent neurons, while recording and biological factors create substantial domain shift.
- SAM 3 supports few-shot Promptable Concept Segmentation from visual exemplars or text prompts, but direct transfer to two-photon imaging is unreliable.
- RAVP augments each target tile with crops retrieved from an external annotated library, using their bounding boxes as concept prompts without changing SAM 3 weights or architecture.
- The study compares fluorescence-guided retrieval and a recall predictor that estimates which exemplar may maximize detection recall.
- RAVP is positioned as inference-time visual memory injection, supported by a systematic analysis of selectors and Allen Brain Observatory evaluation.
2 Related Work
Prior neuron-analysis systems range from batch statistical source separation and online dictionary learning to learned detectors and promptable foundation models. RAVP differs from weight adaptation and separate retrieval-fusion architectures by embedding retrieved exemplars directly into SAM 3’s image-plane input.
- CNMF provides accurate spatial-temporal source separation but is computationally demanding and assumes access to the full recording for batch optimization.
- Suite2P improves scalability through clustering-based initialization, while OASIS enables online temporal deconvolution for fast spike-train inference.
- OnACID and realSEUDO support online source discovery, but dictionary-learning methods can remain limited by warm starts or accumulated statistics in short or cold-start sequences.
- Deep-learning approaches learn spatial and morphological priors from annotated data, including real-time localization, self-supervised detection, and neuron-segmentation systems.
- The SAM family established promptable segmentation, while medical adaptations commonly modify model weights through full or parameter-efficient fine-tuning.
- Few-shot and visual in-context segmentation use support examples at inference, and RAVP extends this direction by embedding retrieved exemplars directly into SAM 3’s image plane.
3 Method
RAVP formulates neuron detection as prediction on image tiles augmented with annotated exemplars retrieved from an external library. It composes these exemplars around each target tile, uses their projected bounding boxes as SAM 3 concept prompts, and studies heuristic and learned selection strategies.
- Problem Formulation: The task predicts neuronal ROI detections, each represented by a bounding box, binary segmentation mask, and confidence score.
- Problem Formulation: RAVP assumes a few-annotation regime with an annotated support library of neuron-centered crops, masks, and fluorescence statistics.The library is built from a held-out split disjoint from training and evaluation data.
- Retrieval-Augmented Image Composition: Retrieved exemplar crops are tiled along the target tile’s top and left borders to form a composite image without modifying SAM 3 weights or architecture.Target and exemplar annotations are projected into shared canvas coordinates.
- Retrieval-Augmented Image Composition: Projected exemplar bounding boxes prompt SAM 3’s Promptable Concept Segmentation head, whose predictions are mapped back and restricted to the target region.Target annotations are used only for quantitative evaluation; prompts come from retrieved library crops.
- Retrieval-Augmented Image Composition: The method processes images with sliding-window tiles because SAM 3 cannot account for all neurons simultaneously in a full image.
- Exemplar Selection Strategies: Selection strategies include random, fixed-fluorescence, image-adaptive, and learned recall-based retrieval of exemplars.Image-adaptive selection matches library fluorescence to the target tile’s mean intensity, while the recall predictor estimates candidate effectiveness from tile–exemplar pairs and is trained on SAM 3 inference outcomes.
4 Experimental Results
Experiments use the Allen Brain Observatory with subject-level splits and compare frozen or adapted SAM 3 under standardized inference and post-processing settings.
- Dataset and evaluation: The Allen Brain Observatory spans cortical areas, imaging depths, and transgenic lines, with maximum projections and instance masks capturing substantial recording variability.Subject-level splitting prevents sessions from the same animal entering training or library construction and test sets.
- Models and baselines: SAM 3 is evaluated against zero-shot prompting, full fine-tuning, encoder-only fine-tuning, and LoRA adaptation.SAM 3 is the reference because its Promptable Concept Segmentation interface accepts visual exemplars as concept prompts.
- Retrieval-augmented inference: Retrieval-augmented inference compares random, fixed-fluorescence, image-adaptive, and learned exemplar selection while keeping SAM 3 frozen.The learned recall predictor uses frozen ResNet-18 features and a two-layer MLP to estimate candidate exemplar recall.
- Inference pipeline: Sliding-window inference partitions images into 50%-overlapping tiles, processes them independently, and merges detections into full-resolution coordinates.Uniform post-processing removes boxes outside 100–1000 px2 and applies NMS at τNMS = 0.4.
- Exemplar configuration: All retrieval-augmented experiments use one exemplar crop per tile at a random top or left border location, with black padding elsewhere.The fixed-fluorescence selector targets f̄ = 0, consistently choosing the lowest-fluorescence library exemplars.
5 Results
Across detection and segmentation evaluations, parameter adaptation improves SAM 3 over zero-shot inference, while exemplar-based prompting strengthens zero-shot performance across evaluation splits at lower adaptation cost.
- Adaptation strategies: Zero-shot SAM 3 reaches AP50 = 0.28 on ABO in-domain detection, while encoder-only and LoRA fine-tuning reach AP50 = 0.69 and 0.68.Both constrained adaptation regimes outperform full fine-tuning at AP50 = 0.64 on the in-domain split.
- Adaptation strategies: On OOD detection, fine-tuned models remain above zero-shot SAM 3 despite precision loss, with encoder-only AP50 decreasing from 0.69 to 0.63 versus zero-shot AP50 = 0.12.On STNeuroNet, encoder-only and LoRA fine-tuning also reach AP50 values up to 0.75, versus 0.69 for full fine-tuning.
- Library mechanism: On ABO in-domain detection, random exemplar prompting raises AP50 from 0.28 to 0.45 and H from 0.41 to 0.53, while the Recall Predictor reaches AP50 = 0.53 and H = 0.62.On STNeuroNet, the Recall Predictor raises AP50 from 0.41 to 0.60 and H from 0.52 to 0.66.
- Library mechanism: On OOD detection, all Library variants improve zero-shot H from 0.20 to approximately 0.58–0.60, with similar performance across selection strategies.The reported trend suggests retrieval-augmented prompting itself provides the main benefit under domain shift.
- Adaptation cost: Retrieval-augmented prompting costs 42% of full fine-tuning and 55% of LoRA adaptation, after recall-supervision generation, feature extraction, and predictor optimization.After predictor training, SAM 3 remains frozen and requires no further parameter updates.
- Qualitative results: Qualitatively, zero-shot SAM 3 misses most neuronal structures, whereas the Fixed Fluorescence and Recall Predictor selectors provide the most comprehensive detections.These comparisons are shown on a representative test sample.
6 Ablation Study
Ablations vary Library configuration while keeping SAM 3 frozen, showing that one exemplar and a moderately sized library are preferred for detection.
- Ablation design: Figure 3 reports validation-split detection H as a function of exemplar count and library size while SAM 3 remains frozen.The ablation varies the Library mechanism rather than model parameters.
- Number of exemplars per tile: With one exemplar per tile, Library SAM attains H ≈0.55; increasing exemplars from k = 2 to 5 causes H to fall to approximately 0.33.Larger exemplar counts partially recover recall but remain below the single-exemplar setting.
- Number of exemplars per tile: A single carefully selected exemplar with sufficient spatial context outperforms multiple exemplars drawn from different fluorescence regimes.This ablation motivates using k = 1 in the main experiments.
- Library size: Increasing the exemplar library from 1 to 50 candidates raises H from approximately 0.52 to 0.59, while larger pools provide only marginal gains.The experiments therefore use 100 candidates as a performance–efficiency trade-off.
7 Conclusion
RAVP adapts SAM 3 for two-photon neuron detection and segmentation through image-plane exemplar composition, without updating model weights or architecture. Carefully selected single exemplars improve zero-shot performance, while adaptive selection generally outperforms random or fixed choices.
- RAVP adapts SAM 3 entirely at inference time through image-plane exemplar composition, without updating model weights or architecture.
- Adaptive exemplar selection generally outperforms random or fixed choices.
- A single exemplar from a moderately sized library captures most available gains and outperforms prompting with multiple retrieved examples.
- The strongest Recall Predictor variant requires SAM 3 inference for recall supervision and additional predictor training beyond annotating the exemplar library.This cost is non-negligible, whereas simpler selectors avoid the stage while remaining competitive.
A Additional Results: MedSAM 3
Additional experiments test whether RAVP's behavior extends from generic SAM 3 to MedSAM 3, which already incorporates medical-domain adaptation. MedSAM 3 starts substantially stronger, while Library augmentation still improves its zero-shot baseline and selection effects vary with domain shift.
- MedSAM 3 provides a substantially stronger zero-shot starting point under domain shift, with detection H increasing from 0.204 for SAM 3 to 0.580.
- Library augmentation consistently improves the MedSAM 3 zero-shot baseline, although with smaller margins than for SAM 3.
- Encoder-only and LoRA adaptation are generally preferable to full fine-tuning in the MedSAM 3 evaluation.
- The Recall Predictor performs best in-domain and on the STNeuroNet subset, whereas OOD results are less sensitive to selection strategy.
- Figure S1 qualitatively compares ground truth, MedSAM 3 zero-shot results without exemplar prompting, and results from different exemplar selection strategies.
B Additional Ablations
Additional controls isolate the source of the Library mechanism's gains in zero-shot detection. The results indicate that retrieved exemplar content, rather than canvas layout, padding, or bounding-box presence alone, is central.
- Library ablations show that gains are not explained by canvas layout, padding, or the mere presence of bounding boxes.
- H = 0.420 versus 0.407 when both exemplar and bounding box are replaced by black padding, indicating only a marginal change from zero-shot SAM 3.
- H = 0.615 for the complete Library mechanism, compared with H = 0.339 when bounding boxes are retained over black regions.
- H = 0.239 for random bounding boxes, which does not reproduce the complete Library mechanism's gain.