PaperlayerView exact revision on arXiv

Source-linked AI summary

Preference Is Not Intervention: The Structure and Stability Boundaries of Reader-Specific Evidence Utility

Shi Zhou

arXiv:2608.17781v1cs.CL

TL;DR

Whether reader-specific evidence utility reflects reusable structure or input-local interactions remains an open question. In RAG, the paper tests the cross-query stability of different forms of reader geometry, finding ordinal preferences reproducible across settings while signed geometry is task-bounded.

  • Problem

    It remains unclear whether model-specific evidence utility reflects stable, reusable reader properties across inputs or situation-local interactions.

    Source: §1 · p. 1

  • Method

    The paper measures evidence utility in RAG and tests the split-half stability of evidence activity, ordinal preference, and conditional signed geometry.

    Source: §1 · p. 2, §3.1 · p. 3, §3.3 · p. 4

  • Results

    Ordinal preferences reproduce across four settings, while signed geometry is task-bounded: observed values are 0.14 and 0.35 in open-ended QA, but 0.748 in binary fact-checking.

    Source: §5.3 · p. 6, §7 · p. 8

  • Takeaways & Limitations

    Stable ordinal similarity does not license cross-reader transfer of intervention decisions; treat help/harm direction as query-local unless the task regime is known to stabilize it.

    Source: §7 · p. 9

    The task boundary is established, but its causal axis is not identified because the forced-choice perturbation provides only partial evidence with label-stratum degeneracy.

    Source: §8 · p. 9

Abstract

from arXiv · show

ML systems increasingly condition decisions on downstream model identity, but this is useful only if model-specific differences form reusable structure rather than input-local interactions. We test this in retrieval-augmented generation (RAG), where evidence utility can be measured under controlled interventions. Holding query, evidence, task, scoring, and intervention fixed, nine readers disagree on effect sign in 33\% of jointly affected cells; reader$\times$query interaction explains 29.8\% of utility variance versus an 8.4\% permutation null; and self-selected evidence improves F1 by $+0.031$ ($t=3.39$). We then ask the sharper question: \emph{which components of this heterogeneity are stable reader properties across queries?} Separating three measurable objects---evidence \emph{activity}, \emph{ordinal preference}, and \emph{conditional signed direction}---we find ordinal reader geometry stable across four independent settings (split-half $ρ=0.60$--$0.83$): leave-one-out interventions, PRISM preferences, RAMDocs, and RAGuard. Signed geometry is task-bounded: weak in open-ended QA (0.14, 0.35), especially for misleading and irrelevant evidence, but strong in binary fact-checking (0.75) with no significant ordinal gap, though still below its sparsity-matched ceiling. Sparsity, decoding noise, and metric artifacts do not explain the main ordinal--signed gap. Finally, stable ordinal similarity fails to predict cross-reader intervention transfer (oracle-distance $ρ=-0.27$; regret reliability $-0.28$). Reader-specific utility exists, but preference is not intervention: stable ranking similarity does not license transfer of help/harm decisions.

1 INTRODUCTION · 2 RELATED WORK

Controlled RAG interventions show that reader identity produces substantial differences in evidence utility, but stable ordinal preferences do not establish stable help-or-harm decisions or enable reliable cross-reader transfer. The paper distinguishes evidence activity, ordinal preference, and conditional signed direction, and tests where each is stable.

1 INTRODUCTION

  • 33% of jointly affected cells showed reader disagreement on help versus harm; reader×query interaction explained 29.8% of utility variance versus an 8.4% null, and self-selected evidence improved F1 by +0.031.The self-selection advantage had query-clustered t = 3.39; the variance interaction was significant at p < 10^-4.Source: §1 · pp. 1–2
  • Activity and ordinal preference are cross-query stable, with ordinal split-half ρ=0.60/0.79/0.83/0.69 across LOO, PRISM [3], RAMDocs, and RAGuard.The paper separates activity, ordinal preference, and conditional signed direction because ranking supervision measures ordinal preference while inclusion decisions depend on signed direction.Source: §1 · p. 2
  • Conditional signed geometry is task-bounded: stability is 0.14 in internal open-ended QA and 0.35 on RAMDocs, versus 0.75 in binary fact-checking on RAGuard.In open-ended QA, signed stability is localized to evidence type: misleading and irrelevant evidence score 0.10/0.09, while supporting evidence scores 0.33; RAGuard has no significant ordinal gap but remains below its sparsity-matched ceiling.Source: §1 · p. 2
  • Cross-reader transfer regret was predicted neither by behavioral profiles (ρ = 0.05) nor in-sample oracle utility distance (ρ = −0.27, n.s.), and regret split-half reliability was −0.28.The experiment covered 9×9 source–target pairs, 50 pre-registered queries, and 4,050 cells; ordinal agreement can coexist with opposite help/harm signs.Source: §1 · p. 2, Figure 1 · p. 3
  • Sparsity, decoding, and metric artifacts do not explain the main ordinal–signed gap; a matched forced-choice perturbation offers suggestive evidence that constraining answers stabilizes responses to misleading evidence.The forced-choice result is described as suggestive, not conclusive.Source: §1 · p. 2

2 RELATED WORK

  • Prior consumer-dependent retrieval work reports retriever–LLM preference divergence [4], personalized rankings for 18 RAG agents [8] [7], and retriever routing per generator, but confounds reader identity with task and other factors.This paper isolates reader identity while holding query, evidence, task, and intervention fixed.Source: §2 · p. 2
  • Concurrent work proposes LLM-specific passage utility and reports limited transfer across generators [14]; Rank4Gen [3] provides PRISM for ordinal replication, while generator-agnostic rerankers [11] [1] align with the activity/ordinal decomposition.The paper independently measures reader-conditioned utility under controlled interventions and focuses on which heterogeneity components remain stable across queries.Source: §2 · pp. 2–3
  • RAMDocs [9] and RAGuard [13] provide typed evidence, while MAGIC [6] studies conflict resolution; this work instead tests whether document-effect direction is a stable reader attribute.The protocol uses stratified split-half reliability and sparsity-matched permutation calibration.Source: §2 · p. 3

3 THREE STABILITIES OF READER-SPECIFIC UTILITY

The section operationalizes a reader as a fixed deployment endpoint and separates reader-specific utility into activity, ordinal preference, and conditional signed direction. It defines cross-query stability for each geometry and explains what such stability would—and would not—license.

3.1 READERS AND UTILITY

  • A reader is a model endpoint with its model, decoding policy, and serving stack fixed; utility is measured relative to a baseline using leave-one-out or single-document interventions.Scores use deterministic task metrics, such as token-F1 or exact/binary match, under each dataset’s protocol.Source: §3.1 · p. 3

3.2 THREE MEASURABLE OBJECTS

  • The three reader-pair objects are activity (including zero patterns), ordinal preference (Spearman correlation), and conditional signed direction on documents nonzero for both readers.Readers can rank documents identically while disagreeing on the sign of a third of them, motivating separate ordinal and signed geometries.Source: §3.2 · p. 3

3.3 STABILITY AS SPLIT-HALF RELIABILITY OF THE GEOMETRY

  • Cross-query stability is split-half Spearman reliability between reader-pair distance vectors, computed over stratified query halves across 1,000 random splits.The median reliability and 2.5th/97.5th percentiles are reported; stable geometry should agree across halves, unlike query-local interaction.Source: §3.3 · p. 4

3.4 PERMUTATION CALIBRATION AGAINST SPARSITY

  • Because signed utility is sparse, signed estimates are calibrated against stable-world permutations that preserve sparse support and relevant conflict-count marginals.Observed reliability far below the null median indicates that sparsity alone does not explain weak stability; parallel calibration addresses ordinal stability, and repeated decoding bounds stochasticity.Source: §3.4 · p. 4

3.5 WHAT STABILITY WOULD LICENSE

  • Stable conditional signed geometry would support transferring help/harm patterns across queries, whereas stable ordinal geometry supports ranking-style conditioning but does not by itself license intervention-choice transfer.The paper tests this transfer consequence directly in Section 5.6.Source: §3.5 · p. 4

4 EXPERIMENTAL SETUP

The experiments compare reader-specific evidence utility across controlled leave-one-out interventions, external single-document tasks, and a large ordinal-preference dataset. The setup distinguishes ordinal preferences from signed intervention effects and compares relative reader geometry rather than absolute capability tiers.

4.1 READERS

  • The internal LOO arm uses 9 API and local readers, while the external single-document arm adds 4 API readers for 13 total.Local endpoints are quantized and served on one RTX-4060 host; comparisons target relative reader geometry, not absolute capability tiers.Source: §4.1 · p. 4

4.2 INTERNAL ARM: LOO UTILITY ON NQ/HOTPOTQA

  • The internal arm tests 9 readers on 100 NQ and HotpotQA [12] queries, each with 8 BGE-M3-ranked documents and 10 context conditions.Token-level F1 yields a 9 × 100 × 8 utility tensor of 7,200 cells.Source: §4.2 · p. 4

4.3 EXTERNAL ARMS: SINGLE-DOCUMENT INTERVENTIONS

  • RAMDocs [9] covers 149 eligible disambiguation queries, while RAGuard [13] covers 212 eligible fact-checking claims with binary verdict-match scoring.RAMDocs uses closed-book and four single-document conditions with normalized any-gold match; signed analyses pool support positions and report misleading and noise positions separately.Source: §4.3 · pp. 4–5
  • 33.3% of jointly affected reader–evidence pairs reverse help/harm sign, while 72.3% move only one reader.Across 9 readers and 100 fixed NQ/HotpotQA queries, evidence activity ranges from 13.4% to 36.6%, and mean retrieval gain is positive for every reader.Source: §4.2 · p. 4, Figure 2 · p. 5

4.4 EXTERNAL ORDINAL ARM: PRISM

  • PRISM/Rank4Gen-DPO [3] provides 29,197 unique query–generator keys across 7,791 queries and 7 generators to calibrate ordinal preference geometry only.Cross-generator comparisons use canonicalized documents with 99.98% pool-Jaccard; ordinal similarity is measured by per-query RBO (p = 0.9) across 21 generator pairs.Source: §4.4 · p. 5

5 RESULTS

Reader-specific evidence utility is measurable, but its stable structure depends on what is measured and the task: ordinal preference geometry reproduces across settings, while signed direction is weaker in open-ended QA than in binary fact-checking. Stable ordinal similarity does not reliably predict cross-reader intervention transfer.

5.1 RQ1 — READER-SPECIFIC UTILITY IS REAL (CONTROLLED HETEROGENEITY)

  • 33.3% of jointly affected reader-pair cells show opposite effect signs, while reader×query interaction explains 29.8% of utility variance versus an 8.4% permutation-null median.All readers gain from retrieval on average (+0.169 to +0.386 F1), but activity is asymmetric: 72.3% of pair-cell cases move exactly one reader. A reader’s own preference set beats other readers’ average by +0.031 F1 (query-clustered t = 3.39).Source: §5.1 · pp. 5–6

5.2 RQ2 — ORDINAL PREFERENCE GEOMETRY IS CONSISTENTLY STABLE

  • 0.599, 0.786, 0.833, and 0.685 split-half reliability show ordinal reader geometry reproduces across the internal LOO arm, PRISM, RAMDocs, and RAGuard, respectively.On PRISM, the observed 0.786 exceeds the identity-shuffle null of 0.314; order-shuffle gives 0.703, exactly size-matched comparison gives 0.862, and composition-only Jaccard geometry gives 0.742.Source: Figure 3 · p. 6, §5.2 · p. 6

5.3 RQ3 — CONDITIONAL SIGNED GEOMETRY IS WEAKER, WITH A TASK BOUNDARY

  • 0.14 and 0.35 signed stability in open-ended QA contrast with 0.748 in binary fact-checking, where signed and ordinal geometry are statistically indistinguishable but remain below the 0.814 stable-world ceiling.The open-ended QA gap is substantial on RAMDocs (paired per-split Δ = 0.487 [0.226, 0.721]); the authors identify a task boundary, not universal instability. The external arms use a different intervention operator than the internal arm, so the weakness is not an artifact of leave-one-out redundancy.Source: Table 1 · p. 6, §5.3 · p. 6

5.4 RQ4 — IN OPEN-ENDED QA, INSTABILITY LOCALIZES TO MISLEADING AND NOISE EVIDENCE

  • 0.330 supporting, 0.104 misleading, and 0.093 noise evidence signed stability in RAMDocs contrasts with 0.658, 0.713, and 0.377 in RAGuard, respectively.In open-ended QA, supporting-evidence directions are partially stable, while misleading and noise directions are essentially query-local; binary verification shows stable direction even for misleading evidence.Source: Figure 4 · p. 7, §5.4 · p. 7

5.6 PRACTICAL CONSEQUENCE — STABLE SIMILARITY DOES NOT LICENSE TRANSFER

  • ρ = −0.271 (p = 0.264) for oracle utility distance versus transfer regret, and regret split-half reliability is −0.281 [−0.530, 0.001], so stable similarity does not predict intervention transfer.Self-selected evidence still beats other readers’ sets by +0.031 F1 (t = 3.39); mean regret is 0.035 F1 with mean standard error 0.037.Source: Figure 6 · p. 8, §5.6 · p. 8

6 ALTERNATIVE EXPLANATIONS

Test–retest controls show that decoding stochasticity does not explain the ordinal–signed gap: attenuation-corrected ceilings remain below ordinal stability on both the internal and RAMDocs arms.

6 ALTERNATIVE EXPLANATIONS

  • Attenuation-corrected ceilings remain far below ordinal stability on both the internal and RAMDocs arms, although decoding stochasticity is real.Table 2 summarizes the controls.Source: §6 · p. 8

7 DISCUSSION

Reader-specific utility separates into consequential per-query differences, stable cross-query ordinal preferences, and signed help/harm direction whose stability depends on the task. Stable ranking similarity does not establish transferable intervention decisions.

DISCUSSION

  • Readers differ consequentially on matched query–evidence conditions, while ordinal preferences are cross-query stable and signed direction varies across open-ended QA and binary verification.In open-ended QA, signed stability is weak and concentrated on adversarial evidence; it is strong in binary verification.Source: §7 · p. 8
  • Generator-conditioned ranking can exploit stable ordinal and compositional preference geometry, while generator-agnostic utility rerankers can generalize from query/evidence-side utility and activity structure without stable signed direction.Source: §7 · pp. 8–9
  • For intervention-level personalization, stable ordinal similarity implies neither transferable intervention decisions nor reliably structured transfer outcomes, so treat help/harm direction as query-local unless the task regime stabilizes it.Source: §7 · p. 9
  • Preference-pair and listwise supervision measure ordinal structure, but deployment depends on intervention direction, so ranking-style stability may not establish stability for inclusion decisions.Cross-query signed-utility reliability with sparsity-matched calibration should accompany claims of stable reader preference; the proposed task-boundary hypothesis and causal axis remain for future work.Source: §7 · p. 9

8 LIMITATIONS … AI USAGE DISCLOSURE

The paper concludes that relative evidence preferences are cross-query stable, but stable ordinal similarity neither guarantees stable help-versus-harm directions nor licenses cross-reader intervention transfer. It also limits the causal and cross-setting scope of its measurements and documents ethics, reproducibility, and AI use.

8 LIMITATIONS

  • The forced-choice perturbation provides only partial evidence, with label-stratum degeneracy; the study does not identify the causal axis of reader-specific utility.The authors claim controlled characterization and stability decomposition, not priority for model-dependent passage utility; a concurrent preprint proposes that notion [14].Source: §8 · p. 9
  • Cross-setting comparisons concern stability patterns rather than identical measurements, because reader panels differ and PRISM lacks a signed operator.Transfer estimates use 50 queries per cell, and utility is measured with task-specific metrics.Source: §8 · p. 9

9 CONCLUSION

  • Relative evidence preferences are consistently cross-query stable, but help-versus-harm stability is task-bounded and ordinal similarity licenses neither transfer nor stable directional claims.Directional stability is weaker in open-ended QA, concentrated on misleading and irrelevant evidence, and absent as a gap in binary fact-checking.Source: §9 · p. 9

ETHICS AND REPRODUCIBILITY

  • All datasets are public and used within their licenses; no human subjects or personal data are involved, and Reddit-origin RAGuard documents serve only as retrieved evidence.The authors state that misleading-evidence findings describe model behavior under misinformation and do not endorse the content.Source: ETHICS AND REPRODUCIBILITY · p. 10
  • An earlier candidate reader was excluded before final analysis after failing response validity, and primary conclusions were recomputed on the finalized roster.Artifacts, frozen plans, and analysis scripts will be released; experimental decisions were frozen before results analysis.Source: ETHICS AND REPRODUCIBILITY · p. 10

AI USAGE DISCLOSURE

  • AI assistants supported literature cross-checking, framing and design feedback, result interpretation, and manuscript editing; authors retain full responsibility for the content.Decision rules were frozen before analysis, analyses ran programmatically on frozen artifacts, and reported numbers were recomputed from those artifacts.Source: AI USAGE DISCLOSURE · p. 10, ETHICS AND REPRODUCIBILITY · p. 10

A READER ROSTER AND DEPLOYMENT CONFIGURATIONS · B PROMPTS AND SCORING CONTRACTS

The sections define the reader deployment configurations and the shared prompt, scoring, and utility contracts used to compare them. They specify the distinct handling of K3, QA-style and RAGuard prompts, and RAMDocs forced-choice conditions.

A READER ROSTER AND DEPLOYMENT CONFIGURATIONS

  • The roster lists 13 readers; all except K3 use temperature 0, disabled reasoning, and a 128-token cap, while K3 is analyzed as its own deployment configuration.K3 forces temperature 1.0 and always-on reasoning, run at minimal reasoning effort with a 2,048-token budget.Source: A READER ROSTER AND DEPLOYMENT… · p. 10, Table 3 · p. 11
  • The roster distinguishes 9 readers in the internal leave-one-out arm from 13 readers in the external single-document arms.K3 permits neither lower temperatures nor disabling reasoning.Source: A READER ROSTER AND DEPLOYMENT… · p. 10, Table 3 · p. 11
  • No reader triggered the preregistered per-arm exclusion rule; the highest fallback rate was 21.6% for Ministral-8B on RAMDocs.The rule excluded readers with fallback or parse failure above 25% of conditions, or a nonzero utility rate below 5%; scale strata were also defined for descriptive cuts.Source: A READER ROSTER AND DEPLOYMENT… · p. 11

B PROMPTS AND SCORING CONTRACTS

  • All QA-style LOO, RAMDocs, RAGuard, and transfer conditions share a prompt requiring a short answer in a fixed format, with closed-book conditions omitting evidence.RAGuard wraps questions as true/false claim judgments and includes each document’s title plus its first 4,000 characters; 97.2% of selected documents were covered in full.Source: B PROMPTS AND SCORING CONTRACTS · p. 11
  • LOO utility is full-context F1 minus F1 without a document; single-document utility is document-only score minus closed-book score, using F1 or accuracy by task.Answers are normalized and scored against gold aliases; LOO uses a full eight-document context.Source: B PROMPTS AND SCORING CONTRACTS · p. 11
  • RAMDocs forced choice asks for an option letter; supporting-answer placement is determined by query-ID hash, and the gold follows support1 when documents assert different valid interpretations.A tiered parser accepts exact letters, matching letter-and-text responses, or unambiguous option-text mentions; tier-0 responses are scored incorrect.Source: B PROMPTS AND SCORING CONTRACTS · p. 11

C PER-READER UTILITY STATISTICS (LOO ARM) · D FULL SPLIT DISTRIBUTIONS

In the LOO arm, reader-specific utility is dominated by reader-by-query variation and disagreement in signed effects. Internal split halves show ordinal reliability more often than signed reliability.

C PER-READER UTILITY STATISTICS (LOO ARM)

  • Reader×query accounts for 29.8% of utility variance, versus an 8.4% permutation-null median, making it the clearest reader-structure result.The 68.0% total across reader-involving terms is descriptive and includes a three-way residual that cannot be separated from measurement error or other residual variation.Source: C PER-READER UTILITY STATISTICS (LOO… · p. 12
  • Dual-nonzero sign conflicts occur in 1,067 of 3,206 cells (33.3%; query-cluster bootstrap 95% CI [0.283, 0.377]).The 3,206 dual-nonzero support is identified as the checksum of the sparsity calibration.Source: C PER-READER UTILITY STATISTICS (LOO… · p. 12
  • Activity asymmetry occurs in 8,348 of 11,554 pair–cell cases (72.3%).Source: C PER-READER UTILITY STATISTICS (LOO… · p. 12
  • Reader main effect explains 0.4% of variance and reader×position 0.8%; the reader-involving total is 68.0%, including an inseparable residual.The three-way term cannot be separated from measurement error or other residual variation because each reader×query×position cell is observed once.Source: C PER-READER UTILITY STATISTICS (LOO… · p. 12
  • Permutation tests reject the null for reader main effect (0.4% vs. null median 0.08%, p < 10−4) and reader×query (29.8% vs. 8.4%, p < 10−4), but not reader×position (p = 0.08).Source: C PER-READER UTILITY STATISTICS (LOO… · p. 12

D FULL SPLIT DISTRIBUTIONS

  • On the internal arm, 80.6% of ordinal splits exceed 0.5, while 92.5% of signed splits fall below 0.3 (11.5% negative).Estimates use the frozen stratified split-half protocol with 1,000 splits; Table 5 reports the full split distributions.Source: Table 5 · p. 12, D FULL SPLIT DISTRIBUTIONS · p. 12

E STABLE-WORLD NULL CALIBRATIONS · F TEST–RETEST AND DECODING-NOISE BOUNDS · G FORCED-CHOICE LABEL STRATA

The calibrations place internal signed stability below its stable-world expectation, while decoding noise does not explain the ordinal–signed gap in RAMDocs; forced-choice signed behavior is strong but varies by label stratum. The section also reports PRISM artifact controls and limits on what the test–retest evidence can establish.

E STABLE-WORLD NULL CALIBRATIONS

  • At 100 queries, internal signed reliability is 0.138 versus a stable-world expectation of 0.373, while ordinal reliability is 0.599.The stable-world calibration preserves each reader pair’s conflict count and sparsity mask; the table reports a pooled conflict rate of 0.333 and stable between-pair SD 0.059.Source: E STABLE-WORLD NULL CALIBRATIONS · p. 13, Table 7 · p. 13
  • PRISM’s observed RBO of 0.786 exceeds the identity-null RBO of 0.314 [0.021, 0.545], while its order-null RBO is 0.703 [0.611, 0.771].The identity null resamples document identities while preserving chosen-list lengths; the order null preserves chosen sets and shuffles order. Mean-aggregated size-matched geometry is 0.862.Source: E STABLE-WORLD NULL CALIBRATIONS · p. 13
  • External signed nulls preserve dual-nonzero support and conflict counts within reader-pair-by-evidence-position strata before permuting conflicts across queries.The pair-only variant drops position stratification; the ordinal control swaps support positions within reader–query cells. With K=4, type-shuffle tests only instance-level support ordering.Source: E STABLE-WORLD NULL CALIBRATIONS · p. 13, Table 8 · p. 13

F TEST–RETEST AND DECODING-NOISE BOUNDS

  • In one local reader’s duplicate run, informative-cell sign changes occurred at f = 0.123, and signs agreed in 107/109 cells nonzero in both runs.Answer agreement was 0.960 and utility-magnitude Pearson correlation 0.977; the noise-based cap on true signed reliability is 0.242, below the stable null (0.363) and ordinal reliability (0.599). This is one reader only; API readers were not rerun.Source: F TEST–RETEST AND DECODING-NOISE BOUNDS · p. 14
  • In RAMDocs, decoding noise cannot bridge the ordinal–signed gap: ordinal reliability is 0.833 versus signed 0.345, with a maximum compatible true signed value of 0.550.In RAGuard, pooled f = 0.216 gives attenuation 0.323 and an uninformative ceiling above 1; decoding noise attenuates, rather than creates, the reported stable signed geometry.Source: F TEST–RETEST AND DECODING-NOISE BOUNDS · p. 14

G FORCED-CHOICE LABEL STRATA

  • Balancing forced-choice label strata changes signed stability only from 0.479 to 0.482, while measurable mislead1 geometry appears only when gold = A.When gold = B, readers flip to the misleading letter nearly uniformly, yielding no measurable geometry; both strata show more consistent signed behavior than open-ended 0.104. Closed-book forced accuracy spans 0.61–0.87 across readers, limiting support-side utility headroom.Source: G FORCED-CHOICE LABEL STRATA · p. 14, Table 9 · p. 14

H TRANSFER EXPERIMENT DETAILS · I BEHAVIORAL PROBE BANK (UNINFORMATIVE) · J ARTIFACT MAP

In the transfer experiment, reader-profile similarity did not reliably predict cross-reader intervention regret, and the behavioral probe bank saturated on the finalized reader roster.

H TRANSFER EXPERIMENT DETAILS

  • ρ = 0.047 for behavior-profile distance versus symmetrized regret (permutation p = 0.901), while regret-matrix split-half reliability was −0.281 [−0.530, 0.001].In-sample oracle utility distance versus regret was ρ = −0.271 (p = 0.264).Source: H TRANSFER EXPERIMENT DETAILS · p. 14
  • 4,050 source × target × query cells covered 9 target readers and 50 pre-registered queries, with 187 unique transfer cells per target.Each source reader’s tie-aware preference set was evaluated on the target reader.Source: H TRANSFER EXPERIMENT DETAILS · p. 14
  • 47.7% of source pairs selected identical document sets; 54.4% of cells had zero cross-source spread, and selections averaged 1.08 positive-utility documents versus 2.77 backfilled.Mean absolute regret was 0.035 F1 against a mean per-cell standard error of 0.037.Source: H TRANSFER EXPERIMENT DETAILS · p. 14
  • −0.017 (HotpotQA) and −0.008 (NQ) nearest-source minus random-source regret; the frozen transfer decision rule was not met.Negative regret means the transferred set beat the target’s own selection, which was itself sparse and backfilled.Source: H TRANSFER EXPERIMENT DETAILS · p. 14, Table 10 · p. 15

I BEHAVIORAL PROBE BANK (UNINFORMATIVE)

  • 7 of 9 readers answered all 48 fictional behavior-bank pairs at ceiling, leaving variation for only two local readers and making the behavior–geometry association unestimable.The reported association was ρ = 0.265 (permutation p = 0.424); the authors treat this as a measurement-ceiling observation, not evidence about behavior–utility relationships.Source: I BEHAVIORAL PROBE BANK (UNINFORMATIVE) · p. 15

J ARTIFACT MAP

Related papers

Loading 2608.17781v1…