PaperlayerView exact revision on arXiv

Source-linked AI summary

MULTI3IR: A Benchmark for Multi-perspective Multi-domain Multi-modal Information Retrieval

Seokwon Song, Sohyeon Kim, Gunhee Kim

arXiv:2608.30949v1cs.IR

TL;DR

Existing IR benchmarks and retrievers provide limited coverage of open-ended queries whose perspectives span multiple domains and modalities, while diversification methods can require costly inference or document-level supervision. The paper introduces MULTI3IR and SPIN, which use perspective annotations and lightweight steering of a frozen retriever to improve coverage. Experiments show reduced single-perspective bias and transfer to unseen benchmarks.

  • Problem

    Existing IR benchmarks focus mainly on closed-ended or single-domain, text-only queries, while diversification methods can require inference-time LLMs or costly document-relevance labels.

  • Method

    MULTI3IR provides 104.9K annotated Stack Exchange queries, and SPIN learns noise vectors that steer a frozen retriever toward multiple perspective-aware embeddings using perspective descriptions.

  • Results

    SPIN improves perspective coverage across backbones and transfers to unseen benchmarks, with gains of 9.37 to 11.73 points on HC@10 for PIR.

  • Takeaways & Limitations

    The results indicate that multimodal retrievers’ single-perspective bias can be substantially mitigated through perspective-aware steering learned without document-level annotations.

  • Takeaways & Limitations

    SPIN uses a fixed number of perspective vectors despite substantial variation in the number of perspectives across queries, and automatic dataset construction may retain LLM-induced biases.

Abstract

from arXiv · show

Information retrieval (IR) increasingly targets open-ended queries that admit diverse perspectives. Existing IR benchmarks, however, focus primarily on closed-ended queries, while even open-ended benchmarks largely consist of queries whose supporting documents span a single subject domain and modality. We introduce Multi$^3$IR, a benchmark that evaluates how well retrievers cover the multifaceted perspectives of open-ended queries across diverse domains and modalities. It comprises 104.9K Stack Exchange queries, each annotated with perspective descriptions that capture the query's implicit viewpoints. We further propose SPIN, a parameter- and label-efficient method that learns noise vectors to steer embeddings toward diverse yet meaningful semantic directions. Experiments show that existing multimodal retrievers suffer from single-perspective bias, while SPIN substantially improves perspective coverage on Multi$^3$IR and generalizes well to unseen open-ended IR benchmarks. The dataset and experimental code are available at https://github.com/seokwon99/Multi3IR.

1 Introduction

Open-ended queries can require comprehensive retrieval across multiple perspectives, domains, and modalities, but existing benchmarks and methods provide limited coverage of this setting. MULTI3IR addresses this gap with a large benchmark, while SPIN targets efficient perspective diversification.

  • Motivation: Open-ended queries decompose into complementary perspectives spanning diverse subject domains and modalities.These perspectives collectively contribute to a more comprehensive answer.
  • Limitations of Existing Work: Most IR benchmarks emphasize closed-ended queries, while open-ended benchmarks commonly use single-domain, text-only supporting documents.This leaves comprehensive retrieval across heterogeneous knowledge sources insufficiently evaluated.
  • Limitations of Existing Work: Query expansion requires inference-time external LLMs, whereas multi-vector retrieval requires costly document-relevance labels for fine-tuning.These requirements create substantial overhead for retrieval diversification.
  • Contributions: MULTI3IR contains 104.9K Stack Exchange queries annotated with perspectives and supporting documents across multiple domains and modalities.Each query requires comprehensive information spanning 3.34 domains and 1.91 modalities on average.
  • Contributions: SPIN learns noise vectors to diversify a frozen retriever into multiple perspective-aware embeddings without costly document-level annotations.The method is described as parameter- and label-efficient and achieves higher perspective coverage than existing diversification methods.

2 Related Work

Prior open-ended IR work addresses multiple interpretations and retrieval diversification, but existing approaches face limitations in candidate coverage, inference cost, or training supervision. These limitations motivate a method that diversifies retrieval using perspective-aware representations efficiently.

  • Open-ended IR Benchmarks: AmbigQA introduced multiple valid interpretations grounded in Wikipedia documents but evaluated question answering rather than retrieval.PIR extended this setting to retrieval with explicitly specified perspectives, while BeRDS addresses implicit user perspectives.
  • Retrieval Diversification: Post-retrieval reranking promotes diversity only within an initial candidate pool, so documents missing from that pool remain unretrievable.This limits its ability to recover perspectives absent from first-stage retrieval.
  • Retrieval Diversification: Query expansion generates hypothetical documents or pseudo-answers before retrieval but requires an LLM call for every query.The approach changes the query to expose additional retrieval directions.
  • Retrieval Diversification: Multi-vector retrieval represents a query with multiple embeddings to expand the regions of the embedding space it can match.The related-work passage identifies this as a diversification strategy alongside query expansion.

3 The MULTI3IR Benchmark

MULTI3IR is constructed as a multi-stage benchmark in which Stack Exchange questions yield verified perspectives and supporting text and image documents. The resulting dataset contains 104.9K questions, and human verification produces a high-quality test split for evaluating perspective coverage.

  • 3.1 Task Formulation: The benchmark defines each open-ended query as a set of textual perspectives grounded in supporting document sets.Retrieval succeeds when the top-k results include at least one document supporting every perspective.
  • 3.2 Dataset Construction: The dataset pipeline selects multi-answer Stack Exchange posts from 77 sites across five categories.The query is formed from each post’s title and body, while its answer set supplies the source material for perspective extraction.
  • 3.2 Dataset Construction: Extracted perspectives are filtered for uniqueness using cosine similarity and for faithfulness using binary entailment judgments.The uniqueness procedure discards perspectives whose maximum similarity exceeds 0.8, while faithfulness checks entailment against the answer set.
  • 3.2 Dataset Construction: 104.9K questions and 521.7K perspectives remain after removing questions with fewer than four perspectives.The benchmark statistics also report 104,916 questions and 521,739 perspectives.
  • 3.2 Dataset Construction: Supporting documents come from Google Image Search and C4, with top-10 results from each source merged and checked for exclusive perspective support.The pipeline uses multimodal retrieval sources and verifies whether documents support the target perspective rather than another perspective.
  • 3.3 Human Verification: Human verification retains a test split of 1.0K queries, 4.8K perspectives, and 12.5K supporting documents.Nearly all perspectives were judged relevant (99.1%) and unique (96.2%), while 90.2% of documents exclusively supported their target perspective.

4 Approach

The approach diagnoses single-perspective retrieval bias and introduces SPIN, which steers a frozen retriever into multiple perspective-aware embeddings using learnable noise vectors. These embeddings are trained through probabilistic coverage of perspective descriptions and combined at inference by Round Robin retrieval.

  • Preliminary Analysis: Existing multimodal retrievers concentrate top-20 retrievals on one dominant perspective, leaving other perspectives with substantially lower scores.Perspective coverage checks whether each perspective has at least one supporting document retrieved.
  • Preliminary Analysis: Perspective descriptions used as queries with Round Robin aggregation recover previously missed documents and produce more balanced performance across perspective ranks.This localizes the bottleneck to query interpretation rather than solely document alignment.
  • Perspective-Guided Learning: SPIN injects learnable noise vectors at an intermediate layer of a frozen retriever to steer embeddings toward diverse perspectives.The method uses perspective descriptions and does not require document relevance annotations.
  • Perspective-Guided Learning: For each query, SPIN forwards noise-perturbed hidden states through the remaining layers to create steered embeddings and encodes perspective descriptions as target embeddings.The learnable noise vectors are optimized so steered embeddings align with the target embeddings.
  • Perspective-Guided Learning: SPIN formulates many-to-many alignment as probabilistic coverage, combining independent steered-to-target coverage probabilities with a noisy-OR.A target is considered covered when at least one steered embedding aligns with it.
  • Inference: At inference, each steered embedding independently retrieves a ranked list by Maximum Inner Product Search, and Round Robin interleaves the lists while skipping duplicates until k documents are selected.The aggregation cycles through rank positions across the lists in order.

5 Experiments

The experiments evaluate perspective coverage on MULTI3IR and unseen open-ended benchmarks using hard and soft coverage metrics. SPIN consistently improves perspective coverage, with gains over competing training approaches and transfer to unseen datasets.

  • Evaluation setup: Hard coverage checks annotated supporting-document retrieval, while soft coverage uses GPT-5-mini to recover relevant evidence missed by annotations.Hard coverage is reported at k ∈ {5, 10, 20, 100}; soft coverage is limited to smaller k because of API cost.
  • Results on MULTI3IR: Zero-shot naive retrievers reach only 28.83 HC@10 and 39.52 SC@10, while oracle perspective queries improve HC@10 by 15.58 to 35.31 points and SC@10 by 31.28 to 36.72 points.At HC@100, the oracle gap remains 19.45 to 35.60 points.
  • Results on MULTI3IR: Under perspective-level supervision, SPIN outperforms LLM-EXPANSION by 8.18 to 13.13 points on HC@10 and by 17.99 to 21.92 points on SC@10.Multiple query embeddings alone do not improve coverage: LLM-EXPANSION falls below the naive baseline on HC@10.
  • Results on MULTI3IR: SPIN outperforms ARE by 3.62 to 4.47 points on HC@10 and by 14.47 to 14.62 points on SC@10, using perspective-level rather than document-level supervision.Perspective-level annotations are also substantially cheaper because document relevance annotation dominates synthesis cost.
  • Results on Other Benchmarks: SPIN improves over the naive retriever across all three backbones on PIR and BeRDS without adaptation to the target distribution.On PIR, HC@10 gains range from 9.37 to 11.73 points; on BeRDS, GME-QWEN2 gains 20.10 points on HC@10.

6 Analysis

The analysis examines how SPIN's injection layer, parameterization, knowledge-source coverage, and document dispersion affect retrieval. Intermediate-layer steering is consistently effective, but domain coverage remains a larger gap than modality coverage.

  • Injection layer: Mid-layer injection at L=18 yields consistent improvements as the number of noise vectors increases, whereas late-layer injection at L=30 plateaus.The analysis attributes this pattern to deeper representations having already collapsed toward a single direction.
  • Adaptation method: SPIN achieves the best HC@100 of 63.4 with only 20.5K parameters, outperforming LoRA and adapter by 7.5 points.This uses over three orders of magnitude fewer parameters than those weight-based adaptation methods.
  • Injection layer: Perturbation-based methods benefit more from intermediate-layer injection: SPIN gains 29.1 points, compared with 1.7 and 2.0 points for reparameterization-based methods.Soft prompt gains 6.0 points under intermediate-layer injection.
  • Knowledge Source Coverage: At k=100, SPIN reaches 78.2 modality coverage versus the 81.1 oracle and closes 91% of the naive-to-oracle gap, but domain coverage reaches only 68.3 versus 84.8.SPIN closes only 35–47% of the naive-to-oracle gap for domain coverage.
  • Document dispersion: SPIN's largest dispersion-conditioned gain is +24.4 HC@100 at m=5 for the top quartile of dispersed queries, compared with +8.7 for tightly clustered queries.Queries are grouped by the average pairwise cosine distance of their positive documents.

7 Conclusion

MULTI3IR benchmarks multi-perspective retrieval across domains and modalities, while SPIN diversifies a frozen retriever using lightweight steering vectors. Experiments report reduced single-perspective bias, stronger retrieval, and generalization, alongside limitations in adaptive perspective counts, supervision, and automated data construction.

  • MULTI3IR contains 104.9K Stack Exchange queries annotated with perspective descriptions and supporting documents.
  • SPIN learns lightweight steering vectors that diversify a frozen retriever into multiple perspective-aware embeddings using only perspective descriptions.
  • Strong multimodal retrievers exhibit single-perspective bias, while SPIN substantially mitigates it, outperforms document-supervised baselines, and generalizes to external benchmarks.
  • The method uses a fixed number of perspective vectors despite substantial variation in the number of perspectives across queries.
  • MULTI3IR relies on automatic perspective extraction, document retrieval, and exclusive-support verification, so LLM-induced biases may affect absolute evaluation scores.

B.1 Raw Data Collection

The dataset is built from quality-filtered and uniformly sampled Stack Exchange questions, whose answers provide multi-perspective supervision. Perspectives and supporting documents are extracted and verified through automated models and human annotation, while SPIN reduces the dominant cost of document-level supervision.

  • Stack Exchange was selected because its multi-answer structure supplies multi-perspective supervision, voting and acceptance provide quality control, and cited links offer perspective-grounded evidence.
  • Questions come from 77 sites across five broad categories and are retained only with more than three answers and votes from more than three distinct users.
  • Category-level uniform sampling with per-category caps limits over-representation of dominant categories such as technology.
  • Perspectives are extracted with GPT-5-mini, deduplicated using a cosine-similarity threshold of 0.8, and checked for faithfulness against concatenated answers.
  • Human verification evaluates perspective meaningfulness, redundancy, and document–perspective support through separate annotation stages.
  • 13.6× more local compute and 2.5× higher API cost are required for document relevance annotation than for perspective-only supervision.

C.1 Evaluation Setup

The evaluation compares retriever training and inference configurations across existing open-ended benchmarks and multimodal corpora. It standardizes prompts and frozen document embeddings, while SPIN trains small intermediate-layer noise vectors on perspective supervision.

  • The evaluation includes PIR’s six sub-datasets with 4,187 questions and 10,019 perspectives, plus the ArguAna subset of BeRDS with 1,000 questions and 2,000 perspectives.
  • All retriever methods use the same instruction prompt, L2-normalized embeddings, precomputed frozen document embeddings, and a single reported run.
  • Fine-tuned methods share 72K MULTI3IR training queries and use AdamW with a cosine schedule and 3% warmup.
  • Naive fine-tuning uses InfoNCE with in-batch negatives, while LLM-Expansion generates five query reformulations at inference.
  • SPIN injects noise vectors at an intermediate layer with frozen encoder weights, using m=5 and a noisy-OR training objective.
  • SPIN, ARE, and zero-shot evaluation run on a single NVIDIA A6000 GPU, whereas Naive and LLM-Expansion use four GPUs.
Loading 2608.30949v1…