PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Reranking Is a System One Decision

Zhuchenyang Liu, Yao Zhang, Yu Xiao

arXiv:2610.28435v1cs.IR

TL;DR

The paper asks whether a general VLM can rerank multimodal retrieval candidates as a System One decision rather than as a dedicated reranker. It tests choice-based scoring and decision tuning, finding that zero-shot reranking competes with dedicated models, while comparisons are confounded by model scale and generation.

  • Problem

    Multimodal rerankers are usually dedicated models trained on retrieval data, while whether a general VLM can rerank without such training remains open.

    Source: §1 · p. 2

  • Method

    The paper treats the query and top 16 retrieved candidates as a System One decision, ranking them by label probabilities from a general VLM and testing text-decision tuning.

    Source: §1 · p. 2, §3.4 · p. 4

  • Results

    Zero-shot Qwen3.8-27B beats the supervised Qwen3-VL-Reranker-2B on all three benchmarks and comes within 1.6 points of Reranker-8B on each.

    Source: §1 · p. 2

  • Takeaways & Limitations

    If the position holds, multimodal decision tuning offers a route to better multimodal reranking alongside dedicated rerankers.

    Source: §11 · p. 9

    The zero-shot model is larger and from a newer generation than the supervised rerankers, confounding the comparison.

    Source: §9 · p. 8

Abstract

from arXiv · show

Multimodal rerankers are usually dedicated models: a vision-language model (VLM) is trained on retrieval data to score the candidates of a query. Scoring options is not specific to reranking. System One decision models such as Jev map a state and a typed question to one probability per option without generating text, and general models can make such decisions with little or no training. Reranking is one such decision, with the query and its candidates as the state and the candidates as the options. We therefore argue that a general VLM can serve as a multimodal reranker through System One decisions, and that decision tuning can improve its reranking. We ask a VLM one choice question over the top 16 candidates of a first-stage retriever and rank the candidates by the probabilities of their labels, read from one forward pass. On MMEB-V2 image retrieval, this choice scores 5.4 to 7.3 hit@1 above a yes/no question per candidate, and the yes/no question falls below the first stage. With the choice, zero-shot Qwen3.8-27B beats the supervised Qwen3-VL-Reranker-2B on all three benchmarks and comes within 1.6 points of Reranker-8B on each. It matches Reranker-8B on MMEB-V2 visual documents (84.79 vs. 84.78 nDCG@5), at 1.1 to 1.5 times its latency. JEV-27B-VL, the same backbone tuned on text decisions only, improves page reranking by 0.6 to 1.1 points, with gains on most tasks (sign test, p <= 0.017), and leaves natural images unchanged. The zero-shot model is larger than the supervised rerankers and of a newer generation, so these comparisons are confounded; we discuss this and other alternative views, and the experiments that would test the position.

1 Introduction

The paper reframes multimodal reranking as a System One decision that a general VLM can make, and argues that decision tuning can improve reranking. It tests this position against dedicated rerankers and prior multimodal approaches.

1 Introduction

  • Reranking can be posed as a System One decision over a query and its candidates, allowing a general VLM to rerank and potentially benefit from decision tuning.The state holds the query and candidates; the question asks which candidate best matches the query.Source: §1 · p. 2
  • A VLM already served to generate answers could also rerank through one decision, avoiding a separately trained, served, and maintained reranker.The authors also propose that decision tuning across tasks could improve reranking without reranking-specific training.Source: §1 · p. 2
  • One choice over candidates improves first-stage image retrieval, while a yes/no question per candidate falls below the first stage; zero-shot page results also depend on model scale.Among Gemma 4 models, those with at least 12B parameters improve page retrieval, while the two smaller models lower it.Source: §1 · p. 2

2 Related Work

  • Prior work covers Jev-style decisions, text reranking, and multimodal rerankers, but this paper applies the readout without reranking training to retrieved images and pages.Closest, ZipRerank trains a VLM to rank pages by letter logits in one forward pass; its smaller zero-shot VLM baselines do not consistently improve over the first-stage retriever [33].Source: §2 · p. 3

3 Reranking as a System One Decision

The paper treats the query and retrieved candidates as a decision state and compares choice and yes/no questions for ranking them. It obtains probabilities from either an untuned VLM or a text-decision-tuned model.

3 Reranking as a System One Decision

  • A reranking request uses the query and retrieved candidates as its state, with a typed question and candidate options.The setting reranks up to 16 candidates; candidates below that cutoff retain the first-stage order.Source: §3 · p. 3, §3.1 · p. 3

3.1 Setting

  • Table 1 presents reranking as a Jev request, with both readouts using the same instruction, query, and candidates.The paper defines a choice question as returning one probability per option and a noul question as returning probabilities for “false” and “true”.Source: §3 · p. 3, Table 1 · p. 4

3.2 Choice readout

  • The choice readout places labeled candidates in one prompt and ranks them by probabilities restricted to their label tokens from one forward pass.Thinking is disabled for the untuned VLM.Source: §3.2 · p. 4

3.3 Yes/no readout

  • The yes/no readout scores each candidate by its “yes” probability against “no” at the first answer token.The paper applies the readout of supervised pointwise rerankers such as Qwen3-VL-Reranker [17] zero-shot.Source: §3.3 · p. 4

3.4 Two ways to obtain the probabilities

  • The untuned Qwen3.8-27B uses no trained weights, while JEV-27B-VL adds a LoRA adapter and decision head trained on text decisions only.The two models share every weight except the adapter and head, allowing comparisons that isolate text decision tuning or question type.Source: §3.4 · p. 4

4 Experimental Setup

The experiments use MMEB-V2 and ViDoRe V3 retrieval benchmarks, a shared first-stage retriever, and top-16 candidates. In the image-retrieval comparison, the choice readout beats the first stage, whereas yes/no scoring falls below it.

4 Experimental Setup

  • The evaluation covers MMEB-V2 image retrieval (12 tasks, hit@1), visual-document retrieval (24 tasks, nDCG@5), and ViDoRe V3 (8 datasets, nDCG@10).ViDoRe V3 pools queries across six languages within each dataset and averages over datasets; video retrieval is omitted because JEV-27B-VL accepts no video input.Source: §4 · p. 4
  • All rerankers receive the same top-16 candidates from Qwen3-VL-Embedding-2B, and the compared models use the same image budget.The image budget is at most 1800 visual tokens per image.Source: §4 · p. 5

5 Preliminary: Choice vs. Yes/No Format

  • 76.76 and 76.80 hit@1 with choice exceed the 74.85 first-stage score, while yes/no reaches 69.50 and 71.42 hit@1.Choice is higher on all 12 image tasks for both models; yes/no falls below the first stage on 11 tasks for Qwen3.8-27B and 9 for JEV-27B-VL.Source: §5 · p. 5, Table 2 · p. 5

6 Findings

A general VLM reranks through a single System One choice decision, competing with dedicated rerankers; tuning on text decisions improves page reranking while leaving natural images unchanged. Gemma 4 results vary by size, and the efficiency comparison measures latency on the same top-16 pages.

6.1 Finding 1: As a System One decision, a general VLM competes with dedicated rerankers

  • Zero-shot Qwen3.8-27B beats Qwen3-VL-Reranker-2B on all three benchmarks and matches Reranker-8B on MMEB-V2 visual documents.It is higher by 2.3 hit@1, 2.1 nDCG@5, and 1.8 nDCG@10 than Reranker-2B; on visual documents it scores 84.79 vs. 84.78 nDCG@5 against Reranker-8B.Source: §6.1 · pp. 5–6, Table 3 · p. 6

6.2 Finding 2: Text decision tuning improves reranking of pages

  • Text-only decision tuning improves page reranking and leaves natural images unchanged.JEV-27B-VL is higher by 0.63 nDCG@5 on MMEB-V2 visual documents and 1.09 nDCG@10 on ViDoRe V3; a two-sided task-level sign test gives p=0.017 and p=0.008, respectively. On natural images, the models tie overall, with JEV higher on 4 of 12 tasks.Source: §6.2 · p. 6

7.1 Gemma 4 models

  • Gemma 4 E2B and E4B lower the first stage on all three benchmarks, while models from 12B upward improve visual-document retrieval.Gains are 1.7 to 2.9 nDCG@5 on MMEB-V2 visual documents and 2.4 to 3.4 nDCG@10 on ViDoRe V3; on natural images, only 31B improves the first stage, by 2.6 hit@1. Gains are not monotonic with size, and every Gemma model trails Reranker-2B on both visual-document benchmarks.Source: §7.1 · pp. 6–7, Figure 2 · p. 7

7.2 Efficiency

  • On one H200, a Qwen3.8-27B choice pass takes 3.97 s per query, 1.1 times Reranker-8B’s 3.67 s using released sequential scoring.With Reranker-8B scoring the 16 pairs in one batch, Qwen3.8-27B takes 1.5 times as long; the comparison uses the same top-16 pages and includes preprocessing.Source: Table 4 · p. 7, §7.2 · p. 7

8 Discussion

The choice readout helps across text-containing image queries but loses on image-only queries; the authors hypothesize that its shared candidate context makes scores more comparable than separate yes/no prompts. Alternative views qualify the result through model-size, cost, comparison, and readout constraints.

8.1 Where the choice helps

  • +1.9 hit@1 on natural images, +5.6 nDCG@5 on MMEB-V2 visual documents, and +7.9 nDCG@10 on ViDoRe V3; image-only tasks lose.Text-containing queries gain, including CIRR (+7.1 hit@1), while MSCOCO image-to-text (−4.1), VisualNews image-to-text (−1.7), and NIGHTS (−0.7) lose.Source: §8.1 · p. 8

8.2 Why the choice beats the yes/no format

  • The choice scores all 16 candidates in one context with probabilities summing to one; separate yes/no prompts lack a shared probability scale.The authors hypothesize that this comparability explains why zero-shot yes/no reranking falls below the first stage.Source: §8.2 · p. 8

9 Alternative Views

  • The comparison changes supervision, model scale, and generation together, so matching a dedicated 8B reranker does not establish parity on the same backbone.The authors state that this confound limits the practical value of the general route.Source: §9 · p. 8
  • Qwen3.8-27B has more than three times Reranker-8B’s parameters and runs at 1.1 times its latency with released code or 1.5 times with batched scoring.A system already serving a VLM could use that model for reranking instead.Source: §9 · p. 8
  • The finding holds only for sufficiently large general VLMs; at small sizes, Reranker-2B improves both visual-document benchmarks while Gemma 4 E2B and E4B lower them.Source: §9 · p. 8
  • Decision tuning and dedicated listwise reranking can coexist: a broadly decision-tuned model is expected to retain other decisions while approaching dedicated reranking performance.Source: §9 · p. 9
  • The choice prompt holds at most 16 candidates under labels A–P, and listwise scores can depend on option order and label preference.Deeper lists may use sliding windows or repeated small-set choices; the experiments present candidates in first-stage order.Source: §9 · p. 9

10 Testing the Position

The conclusion reports strong zero-shot reranking results and a smaller gain from text-only decision tuning. The authors propose tests of multimodal tuning, backbone-matched rerankers, image budgets, and choice-order effects to further assess the position.

10 Testing the Position

  • The proposed multimodal decision-tuning test includes choice questions over images and pages and preference questions over candidates of differing relevance.The position would be refuted if tuning did not improve reranking, especially on natural images and image-only queries.Source: §10 · p. 9
  • A same-backbone comparison across zero-shot choice, multimodal decision tuning, and a dedicated reranker would test whether the gap shrinks with model size.Source: §10 · p. 9
  • Changing Qwen3.8-27B’s image budget or Gemma 4’s budget would test whether their performance gap comes from the backbone or image budget.Source: §10 · p. 9
  • Measuring choice probabilities across candidate orders and calibration, then correcting order effects, would test the choice readout’s robustness.Source: §10 · p. 9

11 Conclusion

  • Zero-shot Qwen3.8-27B beats supervised Qwen3-VL-Reranker-2B on all three benchmarks and comes within 1.6 points of Reranker-8B on each.It matches Reranker-8B on MMEB-V2 visual documents; the comparison is confounded because the zero-shot model is larger and newer.Source: §11 · p. 9
  • Text-only decision tuning improves page reranking by 0.6 to 1.1 points, gains on most tasks, and leaves natural images unchanged.Source: §11 · p. 9

A Prompts

The appendix specifies choice and yes/no prompts for Qwen3.8-27B, including their candidate formats and the next-token logits used for scoring.

A. Choice prompt, listwise (Qwen3.8-27B)

B. Yes/no prompt, pointwise (Qwen3.8-27B), one per candidate

C. JEV-27B-VL through its decision endpoint (request bodies)

JEV-27B-VL’s decision endpoint accepts choice or yes/no requests and returns option probabilities; the released Qwen3-VL-Reranker code uses a yes/no prompt and scores its answer logits.

C. JEV-27B-VL through its decision endpoint (request bodies)

D. Qwen3-VL-Reranker, released code (messages before the chat template)

B Implementation details

The implementation reads candidate probabilities from one forward pass, using either a choice prompt or a yes/no prompt. It then ranks candidates by those probabilities while preserving first-stage order below the reranked set.

B Implementation details

  • Candidates receive probabilities from the next-token logits for label tokens in the choice prompt, while the yes/no prompt scores candidates using the probability of “yes” or “Yes.”Both readouts use one forward pass without decoding; the choice readout applies a softmax to the returned label values.Source: B Implementation details · pp. 14–15
  • JEV-27B-VL uses a decision head tuned on text decisions, returning one probability per option; candidates are sorted by probability, with ties and candidates below the top K retaining first-stage order.No request failed in the reported runs.Source: B Implementation details · p. 15
  • Images are converted to RGB and downscaled above 1,843,200 pixels; choice prompts range from about 3.1k to 28.6k tokens across the listed retrieval settings.The image processor resizes inputs, and whitespace runs in text are collapsed to one space.Source: B Implementation details · p. 15

Server command (one H200)

The experiments use one H200 per run and specify the first-stage retriever, supervised reranker scoring, and benchmark metrics.

Server command (one H200)

  • The supervised reranker scores the 16 query-candidate pairs sequentially, truncates each pair to 10,240 tokens, and caps each image at 1,843,200 pixels.Its score is equivalent to applying a sigmoid to the difference between yes and no logits.Source: Server command (one H200) · p. 16
  • Qwen3-VL-Embedding-2B supplies the first-stage rankings, with the top 100 ViDoRe V3 pages selected by embedding inner product.The ViDoRe page and query instructions differ as specified in the evaluation setup.Source: Server command (one H200) · p. 16
  • Runs use one NVIDIA H200, with Qwen models on torch 2.8 and transformers 5.8, and 27B models on vLLM 0.30.0.MMEB-V2 visual-document results use nDCG@5; ViDoRe V3 nDCG@10 averages results over eight datasets after pooling six languages per dataset.Source: Server command (one H200) · p. 16

C Per-task results

Gemma 4 choice-readout results vary by model size and retrieval benchmark: visual-document gains appear from 12B upward, while natural-image improvement is limited to 31B.

C Per-task results

  • Gemma 4 visual-document retrieval is reported per task on MMEB-V2 using nDCG@5 with top-16 Qwen3-VL-Embedding-2B and a choice readout, and per dataset on ViDoRe V3 using nDCG@10 pooled across six languages.Source: Table 11 · p. 19, Table 12 · p. 19
  • Table 10 reports Gemma 4 performance by task on MMEB-V2 image retrieval, using hit@1 with top-16 of Qwen3-VL-Embedding-2B and a choice readout.Source: Table 10 · p. 18
  • The per-task result tables cover MMEB-V2 image retrieval at hit@1 and visual-document retrieval at nDCG@5, plus ViDoRe V3 datasets at nDCG@10.The ViDoRe V3 table pools all six languages.Source: Table 10 · p. 18, Table 11 · p. 19, Table 12 · p. 19

Related papers

Loading 2610.28435v1…