PaperlayerView exact revision on arXiv

Source-linked AI summary

WARP: Wasserstein-Aligned RAG for Population Opinions

Aman Singh Thakur, Aditya Agrawal, Alwarappan Nakkiran, Alex Karlsson

arXiv:2608.22859v1cs.IRcs.CL

TL;DR

Opinion-focused RAG can produce summaries that reflect relevant but homogeneous evidence rather than the population’s distribution of viewpoints. WARP expands candidate pools and uses Wasserstein-1 re-ranking to match a population target, reducing distributional error by at least 43% with sub-second reranking latency. The method’s main scope boundary is that its offline gains do not establish effects on live user outcomes, and Ppop may reflect corpus bias.

  • Problem

    Standard top-k retrieval ranks by query similarity rather than population representation, allowing minority opinions to disappear from synthesized answers.

  • Method

    WARP performs deficit-aware pool expansion followed by Wasserstein-1 re-ranking against a population opinion target, using variants for different pool densities.

  • Results

    At least 43% lower distributional distance than Top-k was achieved across three domains, with all variants under 330 ms p99 reranking latency.

  • Takeaways & Limitations

    WARP provides a training-free post-retrieval way to improve population-level opinion fidelity without changing the retriever or adding inference calls.

  • Takeaways & Limitations

    The evaluation is offline, so effects on live user trust, engagement, or decision quality remain untested; Ppop also does not remove underlying corpus bias.

Abstract

from arXiv · show

RAG systems are increasingly used to summarize what large collections of documents say. A user asks "What do people think about X?" and receives an answer that reads as consensus. But standard top-k retrieval ranks documents by query similarity, not by how faithfully they represent the population, so minority views quietly disappear. Existing fixes fall short. Diversity re-rankers like MMR and DPP spread retrieved documents apart, but with no target distribution to aim for. Calibration methods based on KL or JS divergence do target one, yet treat opinion bins as unordered: confusing strong positive with strong negative costs no more than an adjacent-bin miss. We introduce WARP, a family of post-retrieval algorithms that calibrate retrieved evidence to the population's opinion distribution. WARP first recovers underrepresented opinions that cosine ranking may bury, then uses Wasserstein-1 distance to select documents whose sentiment-intensity distribution matches the population target, capturing the ordinal structure ignored by KL and JS divergence. We develop three variants for dense, sparse, and variable candidate pools, trading off calibration quality and speed. Across three review domains spanning 35K documents, 156 queries, and 26 entities, WARP's domain-matched variants reduce distributional error by at least 43% with sub-second latency. These gains carry through to generation: a five-judge LLM panel prefers WARP-generated answers in 86% of decided comparisons at k <= 5.

1 Introduction

WARP addresses distributional distortion in opinion-focused RAG by calibrating retrieved evidence to the population’s opinion distribution. Its two-stage pipeline recovers buried sentiment poles, then selects evidence with Wasserstein-1 matching, with variants chosen by candidate-pool density.

  • Motivation: Standard top-k retrieval can produce relevant but homogeneous evidence that over-represents majority opinions and makes summaries factually grounded yet distributionally incomplete.Similarity ranking does not account for the corpus-wide spread of opinions.
  • Motivation: Opinion-query systems must preserve genuine disagreement rather than collapse it, while selecting a small evidence set under sub-second production constraints.The paper frames this as preserving aleatoric uncertainty and matching the population distribution.
  • WARP: WARP then applies Wasserstein-1 re-ranking to select final evidence whose empirical opinion distribution matches the population target.The approach requires no model fine-tuning, retriever changes, or additional inference calls.
  • WARP: WARP expands candidate pools to recover under-represented sentiment poles that cosine retrieval may bury, bypassing expansion when the pool is already balanced.Entity-gated and adaptive re-retrieval target missing opinions and provide near-complete entity coverage.
  • Deployment: All variants add under 330 ms reranking latency at p99, while pool density determines whether dense, sparse, or fallback methods are used.The variants trade calibration quality and speed across deployment conditions.

2 Related Work

Prior retrieval methods diversify or calibrate evidence, but they generally lack an ordinal opinion target or operate during training. WARP adapts Wasserstein optimal transport into a training-free, query-time objective for opinion-distribution matching.

  • Retrieval diversity: Diversity methods such as MMR and DPP spread retrieved items, but their signal runs out once sentiment bins have representatives.After that point, these methods can revert toward relevance ordering.
  • Optimal transport: Wasserstein methods have appeared in document similarity, listwise ranking, extractive summarization, and data summarization, but not as runtime opinion-conditioned selection.Earlier applications function at training time or over generic content.
  • Reported comparison: 43% Seller Forums, 79% Yelp, and 69% OpinRank are the reported distributional-error reductions for W1 re-rankers without re-retrieval.The curves converge at k∗=3–5 documents in the cited figure.
  • Opinion-aware retrieval: Opinion-aware work distinguishes aleatoric disagreement and proposes coverage, fidelity, and fairness objectives, but reports limited domains or standard semantic retrieval.WARP instantiates the coverage idea with W1 and practical retrieval strategies across three domains.
  • Calibrated selection: KL and JS calibration address target distributions but treat labels categorically, while fairness-aware ranking does not model ordinal opinion distance.Categorical penalties cannot exploit severity differences between opinion bins.
  • WARP: WARP contributes an ordinal W1 ground cost for query-time opinion matching in RAG, combining calibrated selection and optimal-transport ranking without training.The method is evaluated across three review domains.

3 Methodology

WARP labels documents on a seven-bin sentiment-intensity axis, constructs a population target, and minimizes Wasserstein distance when selecting k documents. Its variants address dense and sparse candidate pools, with evaluation using distributional fidelity, entity relevance, and latency metrics.

  • Datasets: The evaluation uses roughly 35K documents from Seller Forums, Yelp, and OpinRank, with 156 queries and six questions per entity.The corpora contain approximately 8K, 14K, and 13K documents respectively.
  • Problem formulation: Documents receive sentiment-intensity labels mapped to seven ordered bins, and Ppop is derived from corpus fractions or supplied external targets.External targets can include star ratings or survey-calibrated priors.
  • Evaluation: The benchmark compares Top-k, MMR, DPP, OpinionMMR, and KL/JS Minimizer using W1, Entity Match, latency, and generation-agreement metrics.Top-k is cosine retrieval, while KL/JS uses an order-agnostic divergence.
  • Pool expansion: Entity-gated expansion retrieves entity-matched tail documents by opinion extremity, while adaptive expansion re-retrieves only deficit bins and otherwise adds zero overhead.Both strategies address sparse or imbalanced candidate pools.
  • Problem formulation: Given N candidates, WARP selects k documents whose empirical distribution minimizes transport distance to Ppop.The objective operates along an ordinal sentiment-intensity axis.
  • Wasserstein objective: W1 computes distributional distance from cumulative-distribution differences in O(m) per candidate, assigning larger costs to farther-apart opinion bins.With seven bins spaced by 10, +30 versus −30 costs 60 while adjacent-bin errors cost 10.
  • Variants: The W1 Minimizer filters to entity matches for dense pools, whereas W1-MMR uses all candidates and blends relevance with calibration gain for sparse pools.The relevance term is controlled by λ.
  • Variants: WassRank OT globally assigns k slots proportionally to Ppop and finds a minimum-cost candidate-to-slot matching in O(k^2 · N).It is repurposed as a training-free inference-time re-ranker without learned parameters.

4 Results

WARP consistently reduces distributional error relative to Top-k across domains, with gains propagating to generated answers and variant choice determined by entity-pool density. The method remains latency-compliant and robust across labeling and target-distribution checks.

  • Main Results: Entity-Gated re-retrieval plus W1 Minimizer raises Yelp EM% from 40.1% to 99.2%.The paired expansion and reranking configuration improves entity coverage alongside distributional fidelity.
  • Metric Choice: W1 improves over JS when the same greedy loop and entity filtering are held constant, cutting error by 8.7% on Yelp and 9.7% on OpinRank.Entity filtering alone explains 43.4% of the W1 reduction on Yelp, while W1 Minimizer adds 62.6% beyond that baseline.
  • Generation Evaluation: At k=5, calibration methods win 70–89% of decided comparisons against Top-k, while larger k lets language models average out imbalance.Under proportional-accuracy judgments, W1 Minimizer leads DPP by 8 percentage points and comes within 1 point of an oracle stratified sampler.
  • Generation Evaluation: Distributional methods retain their advantage as k grows, whereas MMR and OpinionMMR Fair% falls to 29–33% at k=20.WassRank OT rises from 70% to 73% to 92% across k values of 5, 10, and 20.
  • Deployment Characterization: Entity Match percentage determines deployment choice: W1 Minimizer suits dense pools, while W1-MMR is preferred for sparse and variable pools.Re-ranking latency remains under 330 ms; dense pools need N=100, while W1-MMR benefits from larger candidate pools in sparse settings.
  • Robustness: WARP’s gains remain favorable against independent star-rating targets, with a worst-case reduction of at least 44.4% versus Top-k.The check recomputes Yelp population targets from native 1–5 star ratings rather than using the LLM-derived labels.

5 Conclusion

WARP is a training-free post-retrieval approach that improves proportional opinion representation within production-oriented latency constraints. Its evidence remains bounded by corpus bias, offline evaluation, labeling requirements, scale, and single-axis review-domain assumptions.

  • Conclusion: WARP reduces W1 distance to the population target by at least 43% within a 330 ms latency envelope.It requires no model fine-tuning, retriever changes, or additional inference calls.
  • Limitations: All reported evaluations are offline, so effects on live user trust, engagement, and decision quality remain untested.The authors identify a controlled online A/B test as the natural next step.
  • Limitations: WARP corrects retrieval-induced bias but does not remove bias in the self-selected review corpus.Matching Ppop can therefore reflect writers’ distribution rather than the underlying population’s; external targets can be plugged in directly.
  • Limitations: Sentiment-intensity labels must be pre-computed at indexing time, while labeling cost–quality tradeoffs are not evaluated.A single LLM pass or VADER is reported as sufficient, with 50–100 labels per entity enough.
  • Limitations: Scaling beyond 14K documents and temporal drift are not addressed.The evaluation covers 156 queries across 26 entities, and the authors report no test beyond 14K documents.
  • Limitations: WARP models one ordinal sentiment-intensity axis, so multi-issue opinions would require multi-marginal optimal transport.The paper does not evaluate such multi-dimensional opinion spaces.
  • Limitations: The evaluation covers product and service reviews, and reported generation gaps depend on the judging criterion.Seller Forums differentiates weakly because of sparse pools, while Yelp and OpinRank provide most decided comparisons.

A.1 Dataset Details

The evaluation uses three review domains, entity selection filters, ordinal sentiment labels, and templated questions to assess proportional opinion summarization. A blind pairwise judge evaluates fairness, informativeness, and groundedness.

  • Opinion labels: Sentiment-Intensity labels use a seven-bin ordinal scale, while mixed labels collapse to the neutral bin.
  • Entity selection: Entity selection applies LLM extraction, diversity filtering, and entropy-ranked top-n selection.
  • Dataset: Three evaluation domains are summarized in the dataset, with entity and sentiment-intensity labels extracted from review documents.
  • Evaluation: Blind pairwise judges compare answers on opinion fairness, informativeness, and groundedness.
  • Queries: Each selected entity receives six questions spanning breadth, polar, and segment templates.

B.5 End-to-End Latency Breakdown

WARP’s reranking is lightweight and backend-independent, while retrieval and re-retrieval determine most managed-endpoint latency. With local FAISS, the complete pipeline remains within a sub-second service target.

  • Latency components: Re-ranking is dominated by CDF differencing over m=7 bins and remains backend-independent.
  • Reranking latency: All WARP variants stay under 310 ms at p99 reranking latency.
  • Deployment boundary: End-to-end latency with local FAISS retrieval fits within 365 ms, whereas managed vector stores add approximately 1.5 s of retrieval latency.

C.1 JS vs. W1: Metric Comparison and Entity-Match Confound

Controlled comparisons favor W1 over KL/JS when entity-match rates are equal, while uncontrolled hybrid results are confounded by different candidate pools. Calibration gains diminish as output size grows and vary by entity density and skew.

  • Entity-match confound: Uncontrolled hybrid differences reflect entity-match gaps, with KL(JS)-MMR retrieving more entity-matched documents than W1-MMR.
  • Metric comparison: W1 wins controlled Minimizer comparisons on Yelp and OpinRank: 2.76 vs. 3.00 and 4.14 vs. 4.54, respectively.
  • Convergence: W1 Minimizer reaches diminishing returns at k∗=3 for Seller Forums, k∗=4 for Yelp, and k∗=5 for OpinRank.
  • Metric robustness: W1 and W2 preserve the same method ranking on Yelp, with a linear relationship having slope = 1.19 and R2 > 0.99.
  • Per-entity variation: On Yelp, W1 Minimizer achieves below 2.0 on 7 of 10 entities, but Business Center and Shuttle Service remain substantially harder.
  • Per-entity variation: Hard cases in OpinRank include Vehicle Size with 42 mentions and Dealer Experience with a baseline W1 of 23.07.

D.2 Entity-Clustered Bootstrap

Entity-clustered bootstrap resampling addresses dependence among queries sharing an entity-level opinion distribution. The resulting uncertainty intervals support consistent positive W1 effects across domains, with wider uncertainty for the smallest entity set.

  • Bootstrap procedure: 10,000 entity-level bootstrap iterations produce 95% confidence intervals excluding zero for every W1 method on every domain.
  • Uncertainty: Seller Forums has a wider W1 Minimizer interval, [0.67, 10.38], because it contains only N=4 entities.
  • Cross-domain coverage: The aggregate signal spans 10 Yelp, 9 OpinRank, and 4 Seller Forums entities.

E Independent-Target Validation

Independent star-rating targets validate that WARP’s gains are not solely artifacts of its LLM-derived labels, while stress tests show robustness to target and labeling noise.

  • Independent-target validation: WARP remains better aligned than Top-k against an independent star-derived population target, with a worst-case reduction of at least 44.4%.The bound is W1(WARP, star-Ppop) ≤5.64 versus W1(Top-k, star-Ppop) ≥10.15.
  • Independent-target validation: Mean W1 divergence between star-derived and LLM-derived population distributions is 2.88 across ten Yelp entities.
  • Target perturbation: At ε=0.2 Dirichlet perturbation on Yelp, W1 Minimizer remains 72% ahead of Top-k despite degrading by 31%.W1-MMR degrades by less than 1% under the same perturbation.
  • Label noise: W1 Minimizer never crosses the Top-k baseline on Yelp even when candidate-document labels and Ppop are simultaneously corrupted at 50% error.For sparse pools, the paper recommends W1-MMR regardless of label quality.
  • Noise control: Random 5% Gaussian score perturbation changes W1 by only 2–3%, versus 14–79% for WARP methods.The random change is non-significant, supporting distributional optimization rather than arbitrary score reshuffling.

F.4 Population Estimation and Cold-Start Robustness

Population estimates converge with modest entity-matched data, and smoothed domain-prior interpolation provides a safer cold-start fallback when entity-level evidence is scarce.

  • Population estimation: At n=50 entity-matched documents, mean W1 estimation error falls below 3.0; at n=100, it falls below 2.0.Bootstrap 95% confidence intervals narrow monotonically with sample size.
  • Cold-start robustness: Even at n=10 labeled documents, all population-estimation strategies outperform uncalibrated Top-k.Smoothed interpolation with a domain prior performs best at every count.
  • Cold-start robustness: Smoothed interpolation achieves 61% W1 reduction at n=10 and is recommended when entity-level evidence is insufficient.The production rule uses entity-level Ppop at n ≥25 with at least two sufficiently populated sentiment bins; otherwise it falls back to the smoothed prior.
  • Sensitivity and deployment: N=100 candidates suffice for near-optimal dense-pool performance, while sparse Seller Forums require larger pools for W1-MMR to locate distributional gaps.
  • Judge robustness: When judges commit to a preference, directional agreement reaches 85–100%, with all five judges agreeing at κd=0.61–1.00.

G.1.3 Majority Vote Results (3/5 Judges Must Agree)

Majority-vote generation results depend on evidence budget and pool density: distributional calibration becomes increasingly advantageous at larger k, while sparse pools constrain all methods.

  • Cross-budget results: At k=20, WassRank OT reaches 92% Fair%, rising from 70% at k=5 and 73% at k=10.MMR and OpinionMMR fall to 29% and 33% at k=20 as semantic diversity saturates.
  • Cross-budget results: At k=5, MMR and OpinionMMR win 83% and 82% of decided comparisons, when only 2–3 sentiment bins are active.
  • Domain density: On dense Yelp at k=10, calibration methods achieve 86–97% fairness decided rate, with W1 Minimizer reaching 97%.Dense OpinRank similarly gives W1 Minimizer 94%, while sparse Seller Forums is limited to 54–55% for W1 methods.
  • Prompt ablation: Under proportional accuracy, W1 Minimizer leads DPP by 8 percentage points at k=5 and 6 points at k=10.It comes within 1 percentage point of stratified oracle sampling at k=5.
  • Prompt ablation: Under viewpoint coverage, W1’s gap narrows to 3 points at k=5 and 2 points at k=10 because DPP already covers most sentiment bins.
  • Metric isolation: W1 leads KL(JS) by +7 pp Fair% cross-domain at k=5, although OpinRank ties on Fair% and favors KL(JS) on Win Rate.
Loading 2608.22859v1…