Source-linked AI summary
AMBER: Multi-View Adaptive Budget Allocation for Listwise Vision-Language Reranking
Wenteng Chen, Jiachen Zhu, Rong Shan, Tianyi Xu, Yuxiang Chen, Congmin Zheng, Teng Wang, Junjie Wu, Weiwen Liu, Changwang Zhang, Weinan Zhang, Jun Wang, Jianghao Lin
TL;DR
VLM listwise reranking refines embedding-recalled candidates by comparing multiple images in a shared context, but token costs, context-length limits, and inference latency create a severe resource bottleneck. AMBER aggregates local listwise outputs as Elo-updated tournaments, then selects ambiguous candidate views and schedules queries by expected information gain. AMBER achieves strong overall performance among compared multi-call VLM reranking methods on CIRR, CIRCO, and PhotoBench under comparable budgets and remains effective in low-budget settings.
Problem
VLM listwise reranking refines embedding-recalled candidates by comparing multiple images in a shared context, but token costs, context-length limits, and inference latency create a severe resource bottleneck.
Source: §1 · p. 1Method
AMBER aggregates local listwise outputs as Elo-updated tournaments, then selects ambiguous candidate views and schedules queries by expected information gain.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
The analysis gives greedy query allocation a (1 −1/e) approximation guarantee under an idealized information-gain objective.
Source: §1 · p. 2The current hyperparameters are static; automatically tuning them during reranking remains future work.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Vision-language models (VLMs) are powerful listwise rerankers for multimodal retrieval, but high inference costs restrict them to evaluating small local candidate views. Existing multi-call strategies rely on fixed schedules, wasting expensive VLM calls on uninformative candidate pairs and easy queries. To address this, we propose Adaptive Multi-view Budgeted Elo Reranking (AMBER), an online, budgeted multi-view reranking framework that dynamically optimizes global resource allocation. AMBER treats fragmented listwise VLM outputs as local tournaments, using continuous Elo updates to maintain a lightweight global ranking state. Building on this, it allocates computation at two levels: dynamically constructing candidate views with high score ambiguity, and scheduling queries to maximize expected information gain. We show that each Elo update corresponds to a stochastic gradient ascent step on the Bradley-Terry log-likelihood, and provide a submodular information-theoretic motivation for the query-level allocation strategy. Experiments on CIRR, CIRCO, and PhotoBench demonstrate that AMBER achieves the strongest overall performance among the compared multi-call VLM reranking methods under comparable VLM-call budgets, while remaining effective in lower-budget settings. Our code is publicly available at https://github.com/wnlfc/AMBER.
1 Introduction
AMBER addresses the budget problem created by costly VLM calls that expose only small candidate views, combining local ranking evidence with dynamic allocation across candidates and queries. Its theoretical analysis and benchmark results support the framework’s allocation strategy and performance under comparable budgets.
1 Introduction
- Costly VLM calls can evaluate only small local candidate views, making it necessary to aggregate evidence across calls and allocate a limited budget across queries and candidates.Fixed schedules apply effort uniformly, while per-query adaptive methods cannot redistribute remaining budget across a global query pool.Source: §1 · p. 1
- AMBER treats each listwise VLM output as a local tournament and uses continuous Elo updates to maintain a lightweight global ranking state without an explicit pairwise preference matrix.The state integrates first-stage embedding priors with VLM evidence.Source: §1 · pp. 1–2
- AMBER builds candidate views from near-top, infrequently observed candidates with ambiguous scores, then schedules calls for queries whose uncertain comparisons matter to head-ranking precision.Query scheduling evaluates the proposed views’ expected information gain.Source: §1 · pp. 1–2
- AMBER achieves strong overall performance among compared multi-call VLM reranking methods on CIRR, CIRCO, and PhotoBench under comparable budgets and remains effective in low-budget settings.The introduction also reports a theoretical foundation for Elo updates and an idealized information-gain objective.Source: §1 · p. 2
- Elo updates are interpreted as online stochastic optimization of the Bradley–Terry model, while an idealized query-level information-gain objective is monotone submodular and gives greedy allocation a (1 −1/e) approximation guarantee.The guarantee is for the idealized objective.Source: §1 · p. 2
2 Preliminary
The preliminary section formalizes second-stage VLM reranking: an embedding retriever supplies candidates, and a listwise VLM refines a limited local view. The setup assumes a fixed global call budget and tracks query-specific reranking state.
2 Preliminary
- The task is to refine an embedding retriever’s ordered candidate pool so the most relevant candidates appear at the top.The paper frames this as second-stage VLM reranking for multimodal retrieval.Source: §2 · p. 2
- A listwise VLM ranks a small candidate subset at each round, so each call observes only a limited local view.Source: §2 · p. 2
- The setup assumes a fixed global VLM-call budget, with b denoting the average number of calls permitted per query.Source: §2 · p. 2
- For each query, AMBER tracks candidate Elo scores, observation counts, observed candidates, current ranks, and a query-level scheduling utility.Source: §2 · p. 2
3 Methodology
AMBER aggregates fragmented local VLM rankings into an online global Elo state, then uses that state to allocate candidate views and query-level calls. Its theoretical analysis connects Elo updates to Bradley–Terry likelihood optimization and gives a condition for top-K recovery.
3.1 Online Cross-View Elo Aggregation
- Each local VLM ranking becomes pairwise outcomes that update a shared Elo state, enabling incremental aggregation across different candidate views.Source: §3.1 · pp. 2–3
- Embedding ranks initialize Elo scores as a prior; after updates, observed candidates are sorted by Elo and unobserved candidates retain embedding order.Source: §3.1 · p. 3
3.2 Adaptive Candidate Selection
- Candidate sampling combines head importance, diminishing returns for frequently observed candidates, and ambiguity from small local Elo gaps.Source: §3.2 · p. 3
3.3 Entropy-Based Query Scheduling
- Query scheduling scores each proposed view by its pairwise uncertainty, scaled by head importance, to prioritize calls resolving critical uncertain comparisons.Source: §3.3 · p. 4
3.4 Overall Algorithm
- AMBER first gives every query one VLM call, then repeatedly allocates remaining calls to the highest-scoring eligible queries until the budget is exhausted.Source: §3.4 · p. 4
3.5 Theoretical Analysis
- Under the Bradley–Terry assumption, each Elo update is a stochastic gradient-ascent step on the log-likelihood, and the regularized estimates have a stated convergence rate.Source: §3.5 · p. 4
- With sufficient observations of every pair straddling the top-K boundary, the Elo ranking recovers the true top-K set with probability at least 1 − δ.Source: §3.5 · p. 4
13: Phase 2: Adaptive Allocation
AMBER’s adaptive allocation repeatedly builds candidate views, schedules queries by utility, and updates its ranking state. Candidate selection balances head importance, limited prior observation, and score ambiguity; its query-allocation theory is conditional on idealized assumptions.
13: Phase 2: Adaptive Allocation
- AMBER selects queries with the largest scheduling utility, reranks their next views, then updates candidate scores, observation records, ranks, and spent budget.The allocation loop continues while budget remains.Source: §13 · p. 5
- Candidate sampling prioritizes near-head candidates, discounts repeatedly observed candidates, and favors candidates whose scores are ambiguous relative to neighbors.The observation discount reflects diminishing reductions in estimation error; small Elo gaps indicate candidates requiring more observations.Source: §13 · p. 5
- Under Assumption 1, uncertainty reduction is monotone submodular, and exact-marginal greedy query allocation has a guarantee in the idealized setting.AMBER’s scheduling score is a tractable proxy for the theoretical marginal gain.Source: §13 · p. 5
- The theoretical analysis assumes conditional independence across calls, although AMBER’s Elo state couples successive observations.The passage also notes possible positional bias and context sensitivity in VLM outputs.Source: §13 · p. 5
4 Experiments
Across CIRR, CIRCO, and PhotoBench, AMBER leads the compared reranking baselines overall at comparable budgets, and remains competitive with reduced calls. The experiments evaluate common recalled candidate pools across composed-image and personal-photo retrieval settings.
4.1 Experimental Setup
- The evaluation covers single-target CIRR, multi-target CIRCO, and merged-album PhotoBench queries reported separately for Chinese and English.PhotoBench empty-ground-truth queries remain in evaluation and receive zero for all metrics.Source: §4.1 · p. 6
- All rerankers use the same recalled candidates from Qwen3-VL-Embedding-8B; experiments use Qwen3-VL-8B-Instruct without chain-of-thought prompting or supervised fine-tuning.Unless otherwise specified, the candidate pool has 50 items, the local view has 10, and the main and low-budget settings average 24 and 8 calls per query.Source: §4.1 · p. 6
4.2 Main Results
- AMBER achieves the strongest overall performance among compared reranking baselines across datasets under comparable call budgets and shared top-50 candidate pools.Table 2 summarizes the main 24-call setting.Source: §4.2 · p. 6, Table 2 · p. 7
- AMBER improves CIRR R@1 from 51.37 to 54.07 over TourRank and leads the reported CIRCO mAP cutoffs and PhotoBench NDCG@1 and NDCG@10 for both languages.On CIRR it also has the best R@1 and R@2 among all methods; on CIRCO, gains are more pronounced at mAP@25 and mAP@50.Source: §4.2 · pp. 6–7
4.3 Low-Budget Results
- With 8 calls per query, AMBER has the best average rank on all three datasets; it leads the listed CIRR and CIRCO metrics and four of five PhotoBench metrics.Sliding Window is slightly better on Chinese NDCG@10; AcuRank averages approximately 10.5–10.7 calls under native stopping.Source: §4.3 · p. 7
5 Related Works
The related-work discussion places AMBER among multimodal retrieval and listwise-ranking approaches, emphasizing the challenge of scaling expensive models across large candidate pools. It distinguishes AMBER’s global budget redistribution from fixed schedules and adaptivity confined to individual queries.
5.1 Multimodal Image Retrieval
- Multimodal retrieval spans unified architectures and methods for reasoning, query rewriting, and cross-modal interaction, while scaling heavy models across large candidate pools remains a bottleneck.The discussion also situates these developments within composed image retrieval.Source: §5.1 · pp. 7–8
5.2 LLMs for Listwise Ranking
- Prior multi-call listwise strategies use rigid schedules or single-query adaptivity; AMBER instead combines continuous Elo updates with globally adaptive allocation.Its global ranking state is used to prioritize informative query-candidate comparisons.Source: §5.2 · p. 8
6 Conclusion
AMBER dynamically allocates a limited VLM budget across candidate views and queries while maintaining a global ranking state from local listwise outputs. Its theory links Elo updates to Bradley–Terry optimization and motivates query scheduling through submodularity; experiments report strong performance across three benchmarks and low-budget effectiveness.
6 Conclusion
- AMBER is an online, budgeted multi-view reranking framework that dynamically optimizes global VLM resource allocation.It maintains a lightweight global ranking state by converting local listwise outputs into continuous Elo-style updates.Source: §6 · p. 8
- Its two-level strategy constructs candidate views around score ambiguity and schedules queries to maximize expected information gain.Theoretical analysis connects Elo updates to online maximum-likelihood optimization under Bradley–Terry and establishes submodular approximation guarantees for query allocation.Source: §6 · p. 8
- Experiments on CIRR, CIRCO, and PhotoBench show strong overall performance among compared multi-call rerankers at comparable budgets, with effectiveness retained at low budgets.These results cover composed image retrieval and personal photo retrieval benchmarks.Source: §6 · p. 8
7 Limitations
The discussion identifies static hyperparameters as a future direction and notes that the evaluation focuses on image retrieval. It also reports full experimental results and compares alternative cross-view aggregation rules.
7 Limitations
- Automatic hyperparameter tuning during reranking remains future work; the evaluation focuses on image retrieval, although the framework may generalize to video or audio.The authors also report using standard listwise prompting without CoT and outperforming strong baselines without its added computational overhead.Source: §7 · p. 9
A.1 Full Experimental Results
- CIRR results are fully reported in Table 2; this appendix provides additional full-metric results for CIRCO and PhotoBench.Source: §A.1 · p. 10
A.2 Alternative Aggregation Methods
- With cached candidate views and VLM outputs held constant, Elo achieves the best performance at all four CIRCO mAP cutoffs among the compared aggregation rules.The alternatives are batch BT-MLE, Borda Count, and average-rank aggregation.Source: §A.2 · p. 10
A.3 Sensitivity Analysis
The appendix examines initialization sensitivity, budget-performance trade-offs, practical costs, and an alternative VLM reranker. It also places AMBER within broader multimodal image-retrieval research.
A.3 Sensitivity Analysis
- At 8 calls per query, CIRCO performance remains stable across Elo initialization ranges of 100–800, with best overall results around 200–400.The embedding prior provides a starting point, while online Elo updates incorporate VLM preference evidence.Source: §A.3 · p. 10, Table 7 · p. 12
A.4 Budget-Performance Trade-off
- Performance improves consistently as AMBER’s CIRCO budget rises from 4 to 24 VLM calls per query, with particularly clear mAP gains; 8 calls already gives strong performance.Source: §A.4 · p. 11
A.5 Practical Efficiency
- Compared methods have similar per-call image counts, token usage, latency, and API costs; AMBER’s rating and observation-count updates are lightweight client-side operations.Source: §A.5 · p. 11
A.6 Results with an Alternative VLM Reranker
- On CIRCO with Qwen3.5-Flash, AMBER achieves the best R@5 and all four mAP metrics while remaining competitive on other Recall cutoffs.The first-stage retriever and candidate pools are unchanged; the results indicate effectiveness beyond the default VLM reranker.Source: §A.6 · p. 11
B.1 Multimodal Image Retrieval
- Related multimodal image-retrieval work spans unified MLLM representations, generative and conversational approaches, and composed-image retrieval methods including agentic systems.The cited examples include MM-Embed and Qwen3-VL, generative retrieval and query rewriting, and CoLLM, XR, and OSCAR.Source: §B.1 · pp. 11–12
B.2 LLMs for Listwise Ranking
Prior work developed LLM-based listwise ranking and efficiency strategies, but fixed schedules and within-query adaptivity leave shared global budgets unallocated across queries. AMBER combines cross-view Elo aggregation with candidate- and query-level allocation.
B.2 LLMs for Listwise Ranking
- LLM listwise ranking progressed from ChatGPT and proprietary rerankers to open-weight systems such as RankVicuna and RankZephyr.Other work considered ranking systems and scoring functions, and tournament-inspired ranking; positional bias remained a studied challenge.Source: §B.2 · pp. 12–13
- Efficiency work includes budget-constrained reranking, uncertainty-aware adaptive computation, passage compression, minimal reasoning, and embedding-based listwise reranking.Source: §B.2 · p. 13
- Fixed-schedule methods can spend computation on easy queries or uninformative comparisons, while adaptive methods primarily operate within individual queries rather than redistributing a shared global budget.Source: §B.2 · p. 13
- AMBER aggregates local listwise outputs into continuous Elo updates and couples candidate-view construction with query-level allocation toward uncertain comparisons relevant to head-ranking quality.Source: §B.2 · p. 13
C Theoretical Analysis
The theoretical analysis relates Elo aggregation to Bradley–Terry optimization, motivates candidate sampling through top-K recovery, and grounds query scheduling in submodular information gain. Its results rely on assumptions that may not hold for realistic VLM calls.
C.1 Elo Aggregation as Online Maximum Likelihood Estimation
- AMBER’s Elo update corresponds exactly to stochastic gradient ascent on the Bradley–Terry log-likelihood, and its aggregation is reported to estimate the true candidate ordering consistently as VLM calls accumulate.The analysis is stated under the Bradley–Terry model assumption.Source: C Theoretical Analysis · p. 13, §C.1 · p. 14
C.2 Sample Complexity and the Rationale for Head-Aware Observation-Margin Sampling
- Under a finite budget, top-K recovery motivates observing candidate pairs across the boundary; the sampling value combines head importance, observation penalty, and margin ambiguity.The stated error bound decreases as O(τ/√m), and the proof sketch uses it to motivate the required observations for recovery.Source: §C.2 · p. 14
C.3 Submodular Motivation for Query Scheduling
- Under the stated conditional-independence assumption, idealized query information gain is non-negative, monotone, and submodular; greedy allocation under a cardinality budget has a (1 −1/e) approximation guarantee.Direct marginal-gain computation requires maintaining a posterior over latent Bradley–Terry scores, which is impractical in the online setting.Source: §C.3 · p. 15
- AMBER uses a view-conditioned scheduling score as a tractable proxy for marginal information gain, estimating uncertainty for the next view and weighting comparisons consequential to head-ranking metrics.The analysis motivates greedy scheduling; the authors say empirical results validate the practical proxy.Source: §C.3 · pp. 14–15
- The theoretical results assume Bradley–Terry preferences and, for Proposition 4, conditional independence across calls, whereas VLM outputs may show positional bias and context sensitivity and Elo couples successive observations.Source: §C.3 · p. 15