NeuroCBIR: A Fast and Accurate Image Retrieval System for Whole-Brain and Region-Specific MRI

• NeuroCBIR provides fast retrieval of both whole-brain and region-level 3D T1w MRI scans.Source: Highlights · p. 2
• Retrieval across 103 cortical and subcortical regions enables flexible anatomical search.Source: Highlights · p. 2
• The system reports state-of-the-art performance for MRI scan re-identification.Source: Highlights · p. 2
• The learned embeddings support zero-shot analyses without task-specific training.Source: Highlights · p. 2
• Code, pretrained models, and MRI embeddings are publicly released.Source: Highlights · p. 2

• CBIR retrieves images by intrinsic image features rather than metadata, offering access to comparable neuroimaging cases for clinical decisions and population-level analyses.Source: §1 · p. 3
• Prior brain-MRI CBIR spans handcrafted tumor-retrieval features and learned embeddings or classifiers for tumor and disease-related tasks (Onga et al., 2019; Nishimaki et al., 2022).Source: §1 · pp. 3–4
• DeepBrainPrint used 2D T1-weighted MRIs for subject re-identification, reporting mAP@3 of 95.5% on ADNI, but was limited to small 2D datasets.Source: §1 · p. 4
• Existing CBIR approaches are often limited by small datasets, restricted brain-region applicability, and little evaluation across heterogeneous MRI acquisition protocols.Source: §1 · p. 5
• NeuroCBIR evaluates retrieval across five large-scale datasets and supports whole-brain and 103-region searches using expert-curated clinical scans.Source: §1 · p. 5
• Its framework couples VAE-derived latent embeddings with contrastive learning and a multi-positive ranking loss to use multiple valid matches per query.Source: §1 · p. 5
• The study assesses zero-shot age estimation and disease-risk inference, and plans to release source code and precomputed features while respecting data-sharing constraints.Source: §1 · p. 5

2.1. Dataset description

• Five public T1w MRI datasets were used; FAED found ADNI, AIBL, MIRIAD, and OASIS3 relatively comparable, while younger SLIM was more distant.FAED was calculated from features of a pretrained 3D ResNet-10 using 200 randomly selected samples per dataset.Source: §2.1 · pp. 5–6, Figure 1 · p. 6

2.2. Imaging prepossessing

• Preprocessing included Talairach registration, intensity rescaling, cropping, and FreeSurfer/FastSurfer segmentation into 103 cortical and subcortical regions.Regional crops were resampled to [64, 64, 64], and scans with evident preprocessing failures were excluded after visual inspection.Source: §2.2 · p. 7

2.3. Dataset setup

• ADNI and OASIS3 were split by subject into approximately 80% training, 10% validation, and 10% testing, while AIBL, MIRIAD, and SLIM were reserved for external testing.Training used subject identity rather than diagnostic class labels, and partitions were stratified by dementia and healthy-subject proportions.Source: §2.3 · p. 7

2.4. Framework of NeuroCBIR

• The shared NeuroCBIR workflow first extracts VAE latent representations, then applies an encoder–projector trained contrastively to create compact embeddings for top-k retrieval.Intra-subject samples are brought together and inter-subject samples pushed apart.Source: §2.4 · p. 7

2.4.1. VAE-GAN

• Separate VAE-GANs were trained for whole-brain and regional data, compressing inputs into latent representations with input-specific tensor shapes.The autoencoder objective combines reconstruction, KL, perceptual, and adversarial losses; latent shapes are [8, 20, 22, 26] for whole-brain inputs and [8, 8, 8, 8] for regional inputs.Source: §2.4.1 · p. 7

2.4.2. Contrastive Learning

• A convolutional encoder and projection head train on VAE latent means with batch size 128, defining positives by subject identity for whole-brain data and subject–region identity for regional data.The learned embeddings are L2-normalized; MPRCL distinguishes hard positives, soft negatives, and hard negatives.Source: §2.4.2 · p. 9
• MPRCL organizes embeddings by pulling same-subject positives together while ranking close different-subject soft negatives and distant hard negatives apart.Negatives are mined using VAE-based cosine similarities, with percentile thresholds computed per anchor; regional losses are computed independently for sampled brain regions.Source: §2.4.2 · pp. 9–10

2.4.3. Inference

• After training, the VAE and encoder–projector form a Q2E module that embeds dataset images into a reference database for retrieval.At inference, the query embedding is compared with the precomputed database to rank the most similar images.Source: §2.4 · p. 7, Figure 2 · p. 8, §2.4.3 · p. 10
• MPRCL defines hard positives through shared identity labels and selects soft and hard negatives using percentile thresholds on VAE latent similarity.Soft negatives exceed the upper threshold πsoft, while negatives fall below the lower threshold πhn.Source: Figure 4 · p. 10
• MPRCL applies anchor-range-scaled margins to rank hard positives closest, soft negatives at intermediate similarity, and negatives furthest from the anchor.The scaling factors are αid for identity-based constraints and αsoft for soft-negative versus negative constraints.Source: Figure 4 · p. 10
• For a new query, Q2E generates an embedding, ranks reference embeddings by pairwise cosine similarity, and retrieves the top-k images.Source: §2.4.3 · p. 10

2.5.1. Re-identification metrics

• mAP@k measures the average proportion of relevant images among each query’s top-k results, while mS@k measures the fraction of queries with at least one relevant match.A hit is a retrieved image from the same subject as the query.Source: §2.5.1 · pp. 10–11
• Each image in the selected query subset serves once as a query against the full image collection, with cosine similarity ranking the retrieved images.Evaluations cover all images, training and test partitions, and ADNI, OASIS3, AIBL, MIRIAD, and SLIM separately.Source: §2.5.1 · p. 11
• Queries from subjects with fewer than k relevant database images are excluded, so each top-k evaluation has enough possible matches.Source: §2.5.1 · p. 11

2.5.2. Embedding-space similarity

• Spearman’s rank correlation measures the monotonic relationship between embedding cosine similarities and MS-SSIM reference similarities, without assuming linearity.For each dataset, 100 image pairs are compared; ρ ≈−1 indicates close agreement because lower ranks correspond to higher similarity, while ρ ≈0 indicates no correlation.Source: §2.5.2 · p. 11

2.6. Interpretability

• Region-level retrieval links similarity directly to anatomically defined regions, whereas whole-brain embeddings make it difficult to identify which regions drive retrieval.Source: §2.6 · p. 11
• For 100 randomly selected whole-brain queries, the analysis compares region-specific cosine similarities with whole-brain rankings among the top-50, top-200, and top-1000 retrieved cases.Same-subject scans are excluded from retrieved cases.Source: §2.6 · p. 11
• A region’s Spearman correlation with whole-brain similarity indicates how closely that region alone explains the whole-brain retrieval ranking.Correlations are calculated separately for each top-k setting.Source: §2.6 · p. 11
• The coefficient of variation summarizes variability in regional correlations across regions for each query and across queries for each region.Low CV across regions indicates consistent regional contributions for a query; high CV indicates varying influence, while low CV across queries indicates a region consistently contributes across queries.Source: §2.6 · p. 11

2.7. Bias evaluation

• Dataset populations and acquisition protocols differ, and scanner manufacturer and field strength also vary within datasets, potentially introducing systematic retrieval-performance biases.Source: §2.7 · p. 11
• Queries are grouped by field strength (1.5T or 3T) and manufacturer (GE, Philips, or Siemens), while every group retrieves against the full database.This keeps the candidate pool constant across query subsets.Source: §2.7 · p. 11
• Prec@5 distributions are compared between subsets using the Mann–Whitney U test and Cliff’s δ effect size.Prec@5 is the proportion of relevant images among the top five retrieved results for each query.Source: §2.7 · p. 11

2.8. Comparison with pretrained models

• NeuroCBIR embeddings are benchmarked against 3D ResNets, CNN-AD, VoComni-L, UMAP-projected VAE embeddings, BrainIAC, and FOMO2JOMO.The compared embeddings are evaluated for compression rate, feature count, mAP@5, mS@5, and ρMS−SSIM on test data.Source: §2.8 · p. 12

2.9. Computational complexity

• Inference-time efficiency is assessed through loading the Q2E module, generating a query embedding, computing cosine similarities against precomputed embeddings, sorting, and retrieving matches.Execution times are reported on a Linux x86_64 system with an Intel Xeon Gold 6226R CPU at 2.90 GHz; PyTorch used 1 and 4 CPU threads.Source: §2.9 · p. 12

2.10. Application-specific experimental setups

• Application-specific evaluations test whether the embeddings capture biological and disease-related patterns without task-specific training or fine-tuning.The section frames these as zero-shot analyses, including age estimation and disease-risk inference.Source: §2.10 · p. 12

2.10.1. Zero-shot similarity-based age inference

• Age inference retrieves similar cognitively normal scans and aggregates their ages without fitting a regression model, using inverse-frequency weights to address uneven age sampling.The reference pool is restricted to subjects older than 50; queries include cognitively normal and Alzheimer’s disease subjects, and performance is measured with MAE, Pearson’s r, and regression slope.Source: §2.10.1 · p. 12

2.10.2. Zero-shot similarity-based disease risk estimation

• Disease risk is estimated from the diagnoses of retrieved neighbors, weighted inversely by their class proportions, with the highest weighted class score selected as the prediction.The retrieval pool includes available scans except those from the query subject; performance is assessed with balanced accuracy and macro-F1.Source: §2.10.2 · pp. 12–13

3.1. Hyperparameter optimization

• Removing soft-negative loss reduces mAP@5 from 98.6% to 79.0%, while removing hard-negative loss shifts ρMS−SSIM from −0.71 to −0.01 despite retaining high retrieval accuracy.The selected whole-brain configuration uses Nf = 32, αhn = 0.2, αsn = 0.2, λsn = 0.2, and πhn = 5.Source: §3.1 · p. 13, Table 3 · p. 13

3.2. Re-identification and embedding-space analysis

• Whole-brain retrieval exceeds 95% mAP and mS for most testing datasets, while mS@1 exceeds 98% in most datasets for large regions such as hippocampus and lateral ventricles.Limited longitudinal samples challenge retrieval in AIBL and SLIM; performance also varies with region size and repeated scans.Source: §3.2 · pp. 13–14, Table 4 · p. 14, Table 5 · p. 15

3.3. Interpretability evaluation

• For top-1000 neighbors, cerebral white matter and lateral ventricles show CV < 1, while several cortical regions exhibit higher query-dependent variability.The analysis uses CV of Spearman correlations between whole-brain and region-based similarity.Source: §3.3 · p. 14, Figure 8 · p. 16

3.4. Bias assessment

• Most dataset comparisons are statistically significant but have negligible-to-small effect sizes, while class, scanner-field-strength, and manufacturer comparisons generally show minimal effects.The authors interpret these patterns as evidence of robust generalization across datasets and acquisition conditions.Source: §3.4 · p. 15, Table 6 · p. 15

3.5. Comparison with pretrained models

• NeuroCBIR reaches mAP@5 of 98.3% and mS@5 of 99.8% with 32 features, outperforming evaluated baselines and achieving ρMS-SSIM = −0.74.The negative correlation indicates agreement between embedding-based rankings and perceptual image similarity.Source: §3.5 · p. 15, Table 7 · p. 16

3.6. Computational complexity

• Whole-brain embedding extraction takes 18.7 seconds on four cores and peaks at 6.8 GB, versus 1.2 seconds and 1.4 GB for region-specific extraction; top-5 search takes <0.01 seconds.Embedding extraction, rather than similarity search, dominates runtime.Source: §3.6 · p. 15, Table 8 · p. 16

3.7. Application-specific evaluation

• Whole-brain zero-shot age estimation achieves MAE 4.3 years and r = 0.54; disease-risk estimation reaches BAcc 56.3% and MF1 53.1%, above the 33% chance level.Removing hard-negative loss yields MAE = 5.2 years and r = 0.09 for age, and lowers three-class disease BAcc to 38.4%.Source: §3.7.1 · p. 17, §3.7.2 · p. 17

4. Discussion

• Whole-brain re-identification reached mAP@5 ∼98.4, with strong regional results and robust generalization across datasets, field strengths, and scanner manufacturers.Embedding-space similarity also correlated with MS-SSIM, supporting preservation of perceptual and semantic similarity between inter-subject scans.Source: §4 · p. 17
• Same-subject scans serve as similarity ground truth, while VAE latent representations capture fine-grained anatomy and the contrastive objectives shape inter-subject relationships.Latent embeddings reduce the input from [160, 176, 208] voxels to [8, 20, 22, 26], supporting larger contrastive-learning batches than direct volumetric processing allows.Source: §4 · pp. 18, 20
• The hard-negative term shifted the MS-SSIM correlation from −0.01 to −0.71, while the soft-negative term raised mAP@5 from 79.0% to 97.0%.The hard-negative term primarily aligned embedding similarity with anatomical similarity; the soft-negative term refined retrieval relationships while separating hard negatives.Source: §4 · p. 18
• Zero-shot age estimation achieved MAE ∼4.3 years and Pearson correlation r ∼0.5 across evaluated regions, despite no age-specific training.The discussion reports an AD-related age gap when predictions use cognitively normal reference datasets.Source: §4 · p. 19
• Whole-brain embeddings achieved BAcc 56.3% and MF1 53.1% across CN, MCI, and AD; external AIBL BAcc was 75.2% whole-brain and 81.9% left amygdala for CN/AD.The AIBL results required no retraining or domain adaptation; most CN–AD misclassifications shifted toward MCI.Source: §4 · p. 19
• NeuroCBIR can complement systems such as TheHiveDB with image-similarity search, and new scans can be added without retraining.The discussion also proposes retrieval embeddings as a perceptual and reasoning component in future AI systems for cohort selection or patient-risk prioritization.Source: §4 · pp. 19–20
• The study is limited to T1w MRI, depends on skull-stripping and other preprocessing, and leaves robustness to substantially different preprocessing strategies unclear.The authors also call for more comprehensive investigation of clinical applications and additional tasks, including dementia-subtype stratification.Source: §4 · p. 20

5. Conclusions

• Across more than 26,000 scans from five datasets, whole-brain embeddings achieved mAP@5 ≥98.4%, with strong regional retrieval and generalization across heterogeneous imaging conditions.The embeddings also preserved meaningful perceptual and semantic similarities between inter-subject scans.Source: §5 · p. 20
• The unsupervised, task-agnostic embeddings supported subject re-identification, zero-shot age prediction, and multi-class pathology stratification, including disease-related structure and intermediate MCI representations.The authors describe retrieval-based predictions as interpretable and applicable to downstream neuroscience tasks without large labeled datasets.Source: §5 · p. 20
• Embedding extraction took ∼18.7s per scan on a 4-core CPU, while similarity search took <0.01s; pretrained models and features for >26,000 scans are publicly available.The authors present the release as a reproducible platform for benchmarking and extending neuroimaging CBIR.Source: §5 · p. 20

CRediT authorship contribution statement

• Félix Nieto-del-Amor is credited with conceptualization, methodology, software, formal analysis, investigation, visualization, writing, and data curation.He also contributed to writing and review.Source: CRediT authorship contribution statement · p. 20
• Jingru Fu, J-Sebastian Muehlboeck, Eric Westman, and Daniel Ferreira are credited with methodology and writing or review; Muehlboeck also contributed data curation.Rodrigo Moreno is credited with conceptualization, methodology, supervision, resources, funding acquisition, writing, and project administration.Source: CRediT authorship contribution statement · pp. 20–21

Funding

• Funding included a Digital Futures postdoctoral fellowship, Swedish Research Council grants, Digital Futures and MedTechLabs support, and multiple research, medical, and disease foundations.D. Ferreira’s support also included regional clinical-research agreements and Karolinska Institutet programs.Source: Funding · p. 21

Appendix A. Detailed Brain-Region Metrics

• The appendix provides a region-by-region view of performance across individual anatomical structures in the testing partition.It describes this as a detailed view at the level of individual regions.Source: Appendix A · p. 25
• Table A.11 reports mAP@5, mS@1, and mS@5 for the evaluated brain regions.The table caption identifies these metrics and their scope across the testing dataset.Source: Appendix A · p. 25, Table A.11 · p. 26
• Table abbreviations define CC as Corpus Callosum, Mid as Middle, lh/rh as left/right hemisphere, and ctx as cortex.Source: Appendix A · p. 26