Source-linked AI summary
Aggregating Neighbor Embedding Projection and Rank-Based Manifold Learning for Image Retrieval
Vinicius Atsushi Sato Kawai, Gustavo Rosseto Leticio, Lucas Pascotti Valem, Daniel Carlos Guimarães Pedronette
TL;DR
CBIR still struggles to rank visually similar images when high-dimensional distances miss contextual relationships and visual features remain separated from high-level concepts. The paper combines UMAP-based neighbor embedding projections with rank-based re-ranking through Borda Count aggregation. Across datasets and deep feature extractors, the approach improves retrieval in several scenarios, especially when baseline representations are weaker, while its benefits vary with dataset and feature quality.
Problem
Pairwise distances can miss contextual and geometric relationships in high-dimensional feature spaces, while the semantic gap and ranking bottleneck remain challenging for CBIR.
Method
The framework independently generates UMAP projection rankings and re-ranked original-feature lists, aggregates them with Borda Count, and optionally applies post re-ranking.
Results
The aggregation strategy improves retrieval effectiveness in several scenarios and often improves top-ranked positions across datasets and feature extractors.
Takeaways & Limitations
Combining projection-based and rank-based manifold learning can provide complementary contextual information for image retrieval.
Takeaways & Limitations
Benefits vary with the dataset and feature representation, particularly when the baseline already achieves very high performance.
Abstract
from arXiv · showhide
Content-based image retrieval (CBIR) has advanced significantly with deep learning, yet effectively ranking similar images remains challenging, particularly in high-dimensional feature spaces, where pairwise distances often fail to capture contextual relationships and the semantic gap between visual features and high-level concepts persists. Manifold learning and rank-based refinement methods have emerged as complementary strategies, respectively improving feature representations and exploiting contextual information embedded in ranked lists, such as neighborhood relationships among images. However, combining these projection-based and rank-based strategies to exploit their complementary properties remains a challenging research problem. To address this, we propose a framework that combines neighbor embedding projections with rank-based manifold learning through rank aggregation. Uniform Manifold Approximation and Projection (UMAP) generates alternative low-dimensional feature representations, and ranked lists obtained from UMAP projections and rank-based re-ranking methods are combined using the Borda Count aggregation strategy. Experiments were conducted on several public datasets using deep learning features extracted from ResNet152, Swin Transformer, and DINOv2 models. Results show that the proposed approach improves retrieval effectiveness in several scenarios, particularly when the baseline representation struggles to achieve high precision. The aggregation strategy also often improves the quality of top-ranked positions, leading to competitive Mean Average Precision (MAP) and Precision values across different datasets and feature extractors. These findings suggest that combining projection-based and rank-based manifold learning strategies through rank aggregation can provide complementary contextual information for image retrieval tasks.
1 Introduction
CBIR ranking remains difficult because pairwise distances can miss contextual and geometric relationships in high-dimensional spaces, while the semantic gap separates visual features from intended concepts. The paper addresses this by independently generating projection-based and rank-based rankings and aggregating them to exploit complementary information.
- Pairwise distance measures often fail to capture complex contextual and geometric relationships between images in high-dimensional feature spaces.
- Deep features improve representation, but ranking remains a bottleneck in fine-grained and unsupervised retrieval settings.The challenge is heightened by high inter-class similarity, substantial intra-class variability, and the absence of labeled data for guidance.
- Projection-based and rank-based strategies are individually effective, but combining their resulting rankings remains relatively underexplored.
- The proposed framework combines UMAP-based projection rankings with re-ranked original-space lists using Borda Count aggregation and optional post re-ranking.The framework uses independent ranking generation rather than a sequential projection-then-re-ranking formulation.
- Across multiple datasets and feature extractors, aggregation provides competitive performance and can improve top-ranked retrieval results.
2 Related Work
Related work develops three complementary directions for retrieval optimization: manifold-based representation, rank-based refinement, and rank aggregation. The paper builds on these directions by unifying neighbor embedding projections with rank-based re-ranking through ranking fusion.
- Manifold Learning: Manifold learning transforms high-dimensional data into lower-dimensional spaces while preserving essential geometric properties.The methods support retrieval, visualization, biological analysis, and unsupervised learning applications.
- Manifold Learning: Projection methods include graph-, neighborhood-, probability-, and diffusion-based approaches, each emphasizing different aspects of local or global structure.Examples include Isomap, LLE, Laplacian Eigenmaps, SNE, t-SNE, and Diffusion Maps.
- Neighbor Embedding Projection: UMAP constructs a fuzzy simplicial complex and offers computational efficiency and neighborhood consistency for compact feature representations.Its choice of projection technique influences how similarity is interpreted and affects ranking-based retrieval.
- Rank-Based Manifold Learning: Rank-based manifold learning refines retrieval by exploiting contextual relationships and ordering information embedded in ranked lists.Graph, hypergraph, reciprocal-neighbor, and diffusion methods model or propagate relationships among retrieved items.
- Rank Aggregation: Rank aggregation combines multiple ranked lists into a consensus ordering, using methods such as Borda Count, Markov Chain fusion, and agglomerative aggregation.The paper positions its approach as an integration of neighbor embedding projections and rank-based re-ranking through rank aggregation.
3 Proposed Approach
The proposed approach combines UMAP-based neighbor embedding projections with rank-based manifold learning, then aggregates their rankings using Borda Count. It is designed to combine geometric and contextual information for image retrieval while accounting for high-dimensionality challenges and UMAP’s parameter sensitivity.
- Rank-Based Manifold Learning: Rank-based manifold learning refines initial ranked lists in an unsupervised manner to obtain more representative rankings.These refinement methods are commonly categorized as re-ranking approaches.
- Motivation: The method addresses weaknesses of pairwise distances in high-dimensional spaces, where manifold structure may be poorly preserved and retrieval quality may deteriorate.The projection maps representations from R^d to a lower-dimensional space, typically with q = 2.
- Neighbor Embedding Projection: UMAP constructs a k-nearest-neighbors graph and optimizes a compact lower-dimensional representation intended to preserve essential data structure.The projection supports alternative feature representations for retrieval.
- Trade-offs: UMAP can make similarity comparisons faster through dimensionality reduction, but retrieval quality depends on selecting suitable n_neighbors and min_dist values.Poorly tuned parameters or varying local densities can produce embeddings that do not accurately reflect intrinsic structure.
- Overview: The framework generates rankings from UMAP projections and rank-based manifold learning, then combines them through rank aggregation.The approach uses independent ranking processes rather than a sequential projection-and-reranking pipeline.
- Rank Aggregation: Borda Count assigns points according to item positions across ranked lists and orders images by summed scores.The method gives greater weight to items that rank consistently highly across sources.
4 Experimental Evaluation
Across datasets and feature extractors, the framework generally provides competitive retrieval performance, with gains especially visible for weaker baseline representations and top-ranked precision. Results also show that Borda Count is competitive, hyperparameter performance is stable, and re-ranking contributes beyond simple aggregation.
- Dataset results: 78.01% MAP was achieved on Flowers with UMAP + RFE rank aggregation and LHRR post re-ranking, exceeding the intermediate configurations reported.UMAP only reached 73.32%, Re-Rank only with RFE reached 73.00%, and UMAP + RFE reached 75.07%.
- Dataset results: 88.74% MAP on Corel5k with ResNet152 was competitive but below LHRR only at 89.76%, while P@100 increased from 61.29% to 86.08%.The result indicates stronger preservation of ranking quality among the top 100 positions than of overall MAP.
- Dataset results: On Pets and CUB200 with SwinTf, UMAP alone achieved the highest MAP, while aggregation-based configurations remained close and improved top-k precision in selected settings.Pets reached 87.26% P@100 with UMAP + CPRR, and CUB200 reached 48.40% P@100 with CPRR post re-ranking.
- Dataset results: 34.80% MAP was obtained on CUB200 with ResNet152 using the full framework, compared with 22.77% for the baseline.UMAP only and CPRR re-ranking reached 31.29% and 32.56%, respectively.
- Dataset results: For Dogs with DINOv2, MAP increased from 55.18% baseline to 66.94% with UMAP and 68.29% with UMAP plus LHRR aggregation.Rank Aggregation alone achieved the highest MAP among the reported configurations, including those with post re-ranking.
- Robustness and efficiency: Borda Count was equal or better than RRF and CombSUM in most configurations, while MAP remained stable across the evaluated UMAP hyperparameter settings.Runtime increased with projection dimensions during ranking and with neighborhood size during projection, whereas aggregation runtime remained approximately constant.
- Ablation and robustness: Removing the re-ranking stage did not consistently improve retrieval, indicating that the strongest results depend on interaction between projection-based representations and manifold-based refinement.The simplified original-plus-UMAP Borda configuration produced lower effectiveness than the full framework for multiple features.
- Ablation and robustness: Across multiple UMAP executions, the proposed configuration exceeded the baseline for ResNet152, SwinTf, and DINOv2 with p < 0.001.Reported MAP comparisons were 82.39% versus 80.59%, 71.77% versus 69.70%, and 68.22% versus 66.87%, respectively; positive RBC values favored the proposed configuration.
5 Conclusion
The paper integrates UMAP-based projections and rank-based manifold learning through Borda Count aggregation, improving retrieval effectiveness and often top-ranked quality in several scenarios. Benefits vary with dataset and feature representation, motivating further exploration of dimensionality reduction, representations, and aggregation strategies.
- The framework combines UMAP-based dimensionality reduction with multiple re-ranking methods through Borda Count aggregation and optional post re-ranking.
- The aggregation strategy improves retrieval effectiveness across multiple datasets and feature extraction models, especially when baseline representations perform less strongly.
- The approach often improves the quality of top-ranked retrieval positions, which matters for practical image retrieval applications.
- Benefits vary by dataset and feature representation, particularly when baseline performance is already very high.
- Future work may examine alternative dimensionality reduction techniques, feature representations, rank aggregation strategies, and hyperparameter settings.
Declarations
The authors report individual contributions across conceptualization, software, methodology, data curation, supervision, funding, visualization, and manuscript preparation, and state that all authors approved the manuscript.
- Contributions span conceptualization, data curation, software, methodology, resources, validation, visualization, supervision, funding acquisition, project administration, and manuscript writing.
- All authors read and approved the final manuscript.
- The authors declare that they have no competing interests.
Funding
The study received financial support from Brazilian research agencies, Petrobras, and the University of São Paulo. The datasets, reranking framework, UMAP implementation, and complete source code are publicly available.
- Financial support came from FAPESP, CNPq, Petrobras, the University of São Paulo, and CAPES-Brasil.
- The publicly available datasets include Flowers, Corel5k, Pets, CUB200, and StanfordDogs.
- The reranking methods use pyUDLF, UMAP uses umap-learn, and the complete framework source code is available on GitHub.