PaperlayerView exact revision on arXiv

Source-linked AI summary

Preference Is Not Intervention: The Structure and Stability Boundaries of Reader-Specific Evidence Utility

Shi Zhou

arXiv:2608.17781v1cs.CL

TL;DR

The paper asks whether reader-specific evidence utility reflects stable reader properties or query-local interactions. Using controlled RAG interventions and split-half geometry analyses across four settings, it finds stable relative preferences but task-bounded signed help/harm directions that do not support intervention transfer.

  • Problem

    The paper asks whether model-specific evidence-utility differences contain reusable reader structure rather than query-local interactions, which matters for reader-conditioned retrieval and personalization.

  • Method

    The study measures evidence activity, ordinal preference, and conditional signed direction using controlled RAG interventions and split-half geometry reliability across four independent settings.

  • Results

    Ordinal reader geometry is stable across four settings (split-half ρ=0.60–0.83), whereas signed geometry is weaker in open-ended QA, stronger in binary fact-checking, and does not predict intervention transfer.

  • Takeaways & Limitations

    Stable reader preferences support ranking and composition personalization, but stable ordinal similarity does not license transferable help/harm intervention decisions.

  • Takeaways & Limitations

    The task boundary’s causal axis is not identified, and the two study arms use different reader panels and nonidentical measurements.

Abstract

from arXiv · show

ML systems increasingly condition decisions on downstream model identity, but this is useful only if model-specific differences form reusable structure rather than input-local interactions. We test this in retrieval-augmented generation (RAG), where evidence utility can be measured under controlled interventions. Holding query, evidence, task, scoring, and intervention fixed, nine readers disagree on effect sign in 33\% of jointly affected cells; reader$\times$query interaction explains 29.8\% of utility variance versus an 8.4\% permutation null; and self-selected evidence improves F1 by $+0.031$ ($t=3.39$). We then ask the sharper question: \emph{which components of this heterogeneity are stable reader properties across queries?} Separating three measurable objects---evidence \emph{activity}, \emph{ordinal preference}, and \emph{conditional signed direction}---we find ordinal reader geometry stable across four independent settings (split-half $ρ=0.60$--$0.83$): leave-one-out interventions, PRISM preferences, RAMDocs, and RAGuard. Signed geometry is task-bounded: weak in open-ended QA (0.14, 0.35), especially for misleading and irrelevant evidence, but strong in binary fact-checking (0.75) with no significant ordinal gap, though still below its sparsity-matched ceiling. Sparsity, decoding noise, and metric artifacts do not explain the main ordinal--signed gap. Finally, stable ordinal similarity fails to predict cross-reader intervention transfer (oracle-distance $ρ=-0.27$; regret reliability $-0.28$). Reader-specific utility exists, but preference is not intervention: stable ranking similarity does not license transfer of help/harm decisions.

1 INTRODUCTION · 2 RELATED WORK

Reader-specific evidence utility is substantial but dominated by reader×query interactions, motivating a separation between reusable reader structure and situation-local effects. The paper isolates reader identity under controlled conditions and distinguishes stable ordinal preferences from unstable help/harm direction, positioning this analysis against confounded personalization and conflict-robustness work.

  • 1 INTRODUCTION: Reader identity is a load-bearing axis when model-conditioned retrieval, query routing, and ensemble weighting rely on differences that generalize beyond situation-local interactions.The central motivation is whether model-specific differences contain reusable structure rather than effects that evaporate on new inputs.
  • 1 INTRODUCTION: 29.8% of utility variance is explained by reader×query interaction, versus 8.4% for the null median, while the reader main effect is only 0.4%.Reader-involving terms sum to 68%, but an unreplicated three-way component accounts for 37.1% and absorbs residual variation.
  • 1 INTRODUCTION: Controlled interventions hold query, evidence, task, scoring, and intervention fixed while changing reader identity to characterize reader-specific passage utility.The design separates the existence of reader heterogeneity from the stability of its components.
  • 2 RELATED WORK: Existing consumer-dependent retrieval studies confound reader identity with task, dataset, backbone, and strategy, whereas this work isolates reader identity with all else fixed.Prior work includes personalized retrieval for 18 RAG agents and retriever routing per generator.
  • 2 RELATED WORK: Concurrent work reports limited transfer of utilitarian passages across generators, while this paper measures reader-conditioned utility differences under controlled interventions and tests their stable components.The question differs by targeting which heterogeneity components persist across queries.
  • 2 RELATED WORK: Figure 1 shows that ordinal agreement between readers can coexist with opposite help/harm signs, separating relative-preference supervision from inclusion decisions.The same ranking can therefore produce different zero-crossings for intervention utility.
  • 2 RELATED WORK: PRISM provides an independent replication resource for ordinal structure, while generator-agnostic rerankers align with utility being predictable mainly from query/evidence activity and ordinal structure.Rank4Gen learns generator-conditioned ranking from PRISM; related transfer work is consistent with this decomposition.
  • 2 RELATED WORK: RAMDocs and RAGuard provide typed supporting, misleading, and noise evidence, while this study asks whether effect direction is a stable reader attribute and finds instability concentrated in adversarial types.The protocol builds on classical split-half reliability with stratified splits and sparsity controls.

3 THREE STABILITIES OF READER-SPECIFIC UTILITY

Reader-specific utility is defined through controlled utility operators and separated into activity, ordinal preference, and conditional signed direction. These geometries distinguish stable ranking structure from query-local help/harm patterns, which would be required for intervention transfer.

  • Utility operators: Utility measures a reader’s score change from removing one document or comparing single-document evidence with the closed-book baseline.The leave-one-out operator uses top-k context D, while the single-document operator compares {d} against ∅; scores are deterministic task metrics.
  • Three geometries: The three reader geometries are activity, ordinal preference, and conditional signed direction, respectively capturing movement, relative ranking, and help-versus-harm agreement.Ordinal preference uses Spearman correlation over utility vectors; conditional signed direction is restricted to dual-nonzero cells.
  • Three geometries: Spearman = 1 can coexist with disagreement on one-third of documents, showing that identical rankings do not imply identical intervention directions.The example motivates separating ordinal preference from conditional signed direction when constructing reader-pair distance matrices.
  • Stability: Cross-query stability is measured by split-half Spearman ρ between reader-pair geometry distance vectors, with stable reader properties expected to yield DA ≈ DB.Queries are split into stratified halves, and reliability is summarized across 1,000 random splits using the median and 2.5/97.5 percentiles.
  • Stability: Sparse signed utility requires stable-world permutation calibration because dual-nonzero cells are rare and sparsity alone can reduce split-half reliability.The calibration preserves observed sparse support and relevant conflict-count marginals while imposing stable reader-pair conflict propensities across queries.

4 EXPERIMENTAL SETUP

The experiments compare reader-specific evidence utility across controlled leave-one-out interventions and external preference and evaluation datasets. They use fixed query/document protocols, multiple reader panels, deterministic scoring, and preregistered reproducibility controls.

  • External reader panel: The external single-document arm expands the panel to 13 readers, adding four models to the nine-reader core roster.Decoding is deterministic wherever specified, with full rosters and configurations provided in Appendix A.
  • Internal LOO arm: 9 core readers evaluate 100 NQ and HotpotQA queries with eight ranked candidate documents and eight leave-one-out contexts, producing a 9 × 100 × 8 utility tensor.The readers include five API endpoints and four local 8–9B GGUF endpoints; token-level F1 scores answers against gold responses.
  • External evaluation datasets: RAMDocs contributes 149 open-ended disambiguation queries with supporting, misleading, and noise documents evaluated under five single-document conditions.Scoring uses the dataset’s strict protocol, adapted as normalized any-gold matching for single-document contexts.
  • External evaluation datasets: RAGuard contributes 212 fact-verification claims with misleading Reddit retrievals, using deterministic binary verdict-match scoring and separate misleading and noise analyses.Query splits are stratified by dataset and, for RAGuard, gold verdict; support positions are pooled in per-position signed analyses.
  • Preference data: PRISM/Rank4Gen-DPO supplies 58,404 preference rows mapped to 29,197 query-generator keys across 7,791 queries, five sources, and seven downstream generators.Chosen ordered document sets represent preferred contexts, with candidate indices translated into canonical document IDs using normalized-text digests.
  • Reproducibility: Eligibility rules, decision rules, split seeds, parsing tiers, transfer plans, checksums, and analysis artifacts were frozen before results analysis.All analyses run on frozen artifacts with fixed seeds.

5 RESULTS

Reader-specific evidence utility is substantial and actionable, but it is driven mainly by reader×query interactions rather than uniform reader effects. Ordinal reader preferences are stable across settings, whereas signed help/harm directions are task-bounded and do not reliably transfer across readers.

  • Existence: 33.3% of dual-nonzero cells show opposite reader effect signs, while 72.3% of pair-cell cases affect exactly one reader.Per-reader nonzero rates span 13.4–36.6%, and all readers gain +0.169 to +0.386 F1 on average.
  • Existence: +0.031 F1 favors each reader’s own measured preference set over the average of other readers’ sets.The result has query-clustered t = 3.39, df = 49, supporting in-sample actionability.
  • Existence: 29.8% of utility variance comes from reader×query interaction versus an 8.4% permutation-null median, while the reader main effect is 0.4%.The decomposition is interaction-led rather than a uniform reader-level shift.
  • Stability: Ordinal split-half reliability spans 0.599–0.833 across LOO, PRISM, RAMDocs, and RAGuard, covering distinct operators, answer formats, generators, and query sources.The PRISM result also survives identity-shuffle artifact controls.
  • Stability: Open-ended signed geometry is weak at 0.14 and 0.35, whereas binary fact-checking reaches 0.75 but remains below its sparsity-matched ceiling.In RAMDocs, supporting evidence is partially stable at 0.330, while misleading and noise evidence are near query-local at 0.104 and 0.093; RAGuard values are 0.658 / 0.713 / 0.377.
  • Transfer: ρ = −0.271 links oracle utility distance to transfer regret, and regret split-half reliability is −0.281, so stable ordinal similarity does not support intervention transfer.Self-selected evidence still retains the in-sample +0.031 F1 advantage.

6 ALTERNATIVE EXPLANATIONS

Control analyses show that decoding stochasticity is genuine but does not account for the main ordinal–signed gap, whose corrected stability ceiling remains substantially below ordinal stability across both evaluation arms.

  • Table 2 summarizes the alternative-explanation controls.
  • Decoding stochasticity is real but does not explain the main ordinal–signed gap.
  • The attenuation-corrected ceiling remains far below ordinal stability on both the internal and RAMDocs arms.

7 DISCUSSION

The discussion distinguishes stable reader-specific ordinal preferences from query-local intervention direction. This boundary supports reader-conditioned ranking and composition, but not unvalidated transfer of help/harm decisions.

  • What kind of reader specificity exists?: Readers differ on matched query–evidence conditions, while ordinal reader structure reproduces across disjoint query sets in all four settings.The discussion separates reader differences, stable ordinal preference, and less-uniform signed direction rather than treating them as one property.
  • Reconciling generator-conditioned and generator-agnostic methods: Generator-conditioned ranking can exploit cross-query-stable ordinal preference geometry, while generator-agnostic rerankers can use predictable query/evidence-side utility and activity structure.Both approaches remain compatible with signed direction being query-local.
  • For personalization systems: Personalization should condition ranking and composition on the reader, but help/harm direction should be treated as query-local unless the task regime stabilizes it.Stable ordinal similarity alone does not imply transferable intervention decisions or a reliably structured transfer outcome.
  • For evaluation practice: Preference-pair and listwise evaluations measure ordinal structure, whereas deployment inclusion decisions depend on intervention direction.The discussion recommends cross-query signed-utility reliability with sparsity-matched calibration before claiming stable reader preference.

8 LIMITATIONS … AI USAGE DISCLOSURE

The paper characterizes reader-specific evidence utility while delimiting its causal and cross-setting scope: stable relative preferences do not imply stable help-versus-harm directions or intervention transfer. Public-data use, preregistered analysis controls, artifact release, and disclosed AI assistance support reproducibility and accountability.

  • 8 LIMITATIONS: The forced-choice perturbation provides partial causal evidence, but the causal axis remains unidentified because of label-stratum degeneracy.The authors do not claim priority for model-dependent passage utility and frame their contribution as controlled characterization and stability decomposition.
  • 8 LIMITATIONS: Cross-setting comparisons assess stability patterns rather than identical measurements because the two arms use different reader panels and PRISM lacks a signed operator.This limits direct equivalence across the reported settings.
  • 9 CONCLUSION: Reader-specific evidence utility is consequential: holding other factors fixed, changing the reader changes which documents help, harm, or do nothing.Relative evidence preferences remain cross-query stable across readers, datasets, and preference constructions.
  • 9 CONCLUSION: Help-versus-harm direction stability is task-bounded, weaker in open-ended QA and concentrated on misleading and irrelevant evidence, while binary fact-checking shows no gap.The conclusion distinguishes stable preference from stable intervention direction.
  • ETHICS AND REPRODUCIBILITY: All datasets are public, used within their licenses, and contain no human subjects or personal data.RAGuard documents originate from Reddit and are used only as retrieved evidence under the dataset’s terms.
  • ETHICS AND REPRODUCIBILITY: Experimental decisions were frozen before results analysis, with artifacts, frozen plans, and analysis scripts slated for release.The paper also states that misleading-evidence findings describe model behavior under misinformation and do not endorse the misinformation.
  • AI USAGE DISCLOSURE: AI assistants supported literature cross-checking, framing and experimental-design feedback, result interpretation, and manuscript editing.All decision rules were frozen beforehand, analyses ran programmatically on frozen artifacts, every reported number was recomputed, and the authors retain full responsibility.

A READER ROSTER AND DEPLOYMENT CONFIGURATIONS · B PROMPTS AND SCORING CONTRACTS

The study defines readers as model endpoints under fixed deployment configurations, while standardizing QA prompts, evidence presentation, and scoring across evaluation arms. It separately specifies closed-book, fact-checking, and RAMDocs forced-choice conditions to control answer formats and gold-label construction.

  • A READER ROSTER AND DEPLOYMENT CONFIGURATIONS: 13 readers are evaluated as deployment configurations, with K3 treated separately because its endpoint enforces temperature 1.0 and always-on reasoning.All other endpoints use deterministic decoding with temperature 0, disabled reasoning, and a 128-token cap; K3 uses minimal reasoning and a 2,048-token budget.
  • A READER ROSTER AND DEPLOYMENT CONFIGURATIONS: The roster distinguishes the 9-reader internal leave-one-out arm from 13-reader external single-document arms.Table 3 labels these arms LOO and SD, respectively.
  • A READER ROSTER AND DEPLOYMENT CONFIGURATIONS: No reader triggered the preregistered exclusion rule in any final arm.The rule excluded readers with fallback or parse failure above 25% of conditions, or a nonzero utility rate below 5%; the highest fallback rate was 21.6%.
  • B PROMPTS AND SCORING CONTRACTS: All QA-style conditions share a prompt contract requiring factual answers in an exact output format.This contract applies to every LOO condition, RAMDocs and RAGuard single-document conditions, and all transfer evaluations.
  • B PROMPTS AND SCORING CONTRACTS: The QA prompt presents a question and indexed evidence documents, then requires only a short answer in the format “Answer: ⟨short answer⟩”.The evidence block contains documents numbered [1] through [k].
  • B PROMPTS AND SCORING CONTRACTS: Closed-book conditions omit evidence, while RAGuard reformulates questions as True/False claim verification.RAGuard uses gold answers True/False and supplies each document’s title plus its first 4,000 characters; 97.2% of selected documents are covered in full.
  • B PROMPTS AND SCORING CONTRACTS: RAMDocs forced choice uses a deterministic query-hash rule to assign supporting and misinformation options, with gold labels following support1 in 44 of 149 rows containing divergent valid interpretations.Readers respond with the option letter only; option A is supporting and option B misinformation when the query-id md5 hash is even.

C PER-READER UTILITY STATISTICS (LOO ARM) · D FULL SPLIT DISTRIBUTIONS

The LOO arm shows substantial reader disagreement and asymmetric utility activity, while split-half analyses reveal much stronger ordinal than signed stability. The distributions also expose support limitations for position-specific signed estimates.

  • C PER-READER UTILITY STATISTICS (LOO ARM): Table 4 summarizes per-reader utility over 800 cells each, spanning 100 queries and 8 documents.Nonzero rate is the fraction of cells with U̸ = 0; gain is F1(full context) − F1(closed-book), averaged over queries.
  • C PER-READER UTILITY STATISTICS (LOO ARM): 33.3% of dual-nonzero cells show reader-pair sign conflicts, based on 1,067 of 3,206 cells.The query-cluster bootstrap 95% CI is [0.283, 0.377].
  • C PER-READER UTILITY STATISTICS (LOO ARM): 72.3% of pair–cell cases exhibit activity asymmetry, covering 8,348 of 11,554 cases.The dual-nonzero support is 3,206 cells, serving as the sparsity-calibration checksum.
  • D FULL SPLIT DISTRIBUTIONS: 80.6% of internal-arm ordinal splits exceed 0.5, whereas 92.5% of signed splits fall below 0.3.The signed distribution includes 11.5% negative splits.
  • D FULL SPLIT DISTRIBUTIONS: Table 5 reports split-half reliability percentiles over 1,000 splits, with 999 splits for forced ordinal analyses.NN measures the fraction of readers whose nearest neighbor is preserved across the two halves; paired ∆rows compare ordinal with signed reliability.
  • D FULL SPLIT DISTRIBUTIONS: Table 6 reports per-position conditional signed stability using dual-nonzero cells pooled over splits.Forced mislead1 is undefined on 50 splits because one half has zero support.

E STABLE-WORLD NULL CALIBRATIONS · F TEST–RETEST AND DECODING-NOISE BOUNDS · G FORCED-CHOICE LABEL STRATA

Stable-world calibrations show that ordinal reliability is measurable at current sample sizes while signed reliability remains substantially lower. Test–retest results bound signed stability, and forced-choice analyses show that aggregate signed stability is robust to label-stratum balancing but depends on the informative stratum.

  • E STABLE-WORLD NULL CALIBRATIONS: The internal sparsity calibration preserves the real mask and pairwise conflict counts while permuting conflict indicators within reader pairs across 5,000 simulations.The observed value is 0.138 on 3,206 dual-nonzero cells.
  • E STABLE-WORLD NULL CALIBRATIONS: At 100 queries, signed split-half ρ is 0.138 versus a stable-world expectation of 0.373, while ordinal ρ is 0.599.Expected split-half ρ rises from 0.104 at 20 queries to 0.548 at 200 queries under the stable null.
  • F TEST–RETEST AND DECODING-NOISE BOUNDS: In exact duplicate runs, informative-cell sign changes occur at f = 0.123, while both-nonzero signs agree in 107/109 cases.Answer agreement is 0.960 and utility-magnitude Pearson correlation is 0.977.
  • F TEST–RETEST AND DECODING-NOISE BOUNDS: Under independent per-cell flips, decoding noise attenuates signed reliability to 0.569× true and caps true signed reliability at 0.242.The bound follows from observed signed reliability 0.138 divided by 0.569.
  • G FORCED-CHOICE LABEL STRATA: Balancing forced-choice label strata changes overall signed stability only from 0.479 to 0.482, showing the aggregate elevation is not a label-mixing artifact.The balanced estimate is defined as the mean of the two stratum distances per split.
  • G FORCED-CHOICE LABEL STRATA: The mislead1 geometry is measurable only when gold = A; when gold = B, readers choose the misleading letter nearly uniformly.With constant conflict rates across pairs, the gold = B stratum contributes no measurable geometry, despite uniform flipping producing maximal signed agreement.

H TRANSFER EXPERIMENT DETAILS · I BEHAVIORAL PROBE BANK (UNINFORMATIVE) · J ARTIFACT MAP

The transfer experiment evaluates reader-specific evidence selection across 9 readers and 50 queries, while a fictional behavioral probe bank fails to provide an informative geometry predictor. The artifact map records the paper’s primary claims and frozen artifacts.

  • H TRANSFER EXPERIMENT DETAILS: 47.7% of source-reader pairs select identical document sets per query, with mean Jaccard 0.421 among differing sets and 3.74 distinct sets per query.The design includes 187 unique transfer cells per target and 4,050 source × target × query cells overall.
  • H TRANSFER EXPERIMENT DETAILS: Transfer regret is defined as the target’s own-selection F1 minus transferred-selection F1, averaged across 50 queries.Negative regret means the transferred set outperformed the target’s own selection, which may itself be sparse and backfilled.
  • H TRANSFER EXPERIMENT DETAILS: Nearest-source transfer is compared with random-source transfer using behavior-profile distance from the saturated probe bank.The comparison is reported in Table 10 as F1-based transfer regret.
  • I BEHAVIORAL PROBE BANK (UNINFORMATIVE): 7 of 9 readers answer all 48 fictional behavior-bank pairs at ceiling, leaving nonzero behavioral variation for only two local readers.The bank contains six dimensions and was intended to predict utility geometry from behavioral profiles.
  • I BEHAVIORAL PROBE BANK (UNINFORMATIVE): The behavior–geometry association is ρ = 0.265 with permutation p = 0.424, but saturation makes it uninformative and not estimable on the finalized roster.The two newest API readers have EM = 1.0, contributing no variation to the probe-bank instrument.
  • J ARTIFACT MAP: Table 11 maps the paper’s primary claims to their frozen artifacts.The supplied passage identifies the table as the artifact map but does not enumerate its individual claims or artifacts.
Loading 2608.17781v1…