PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Dense Retrieval Need Asymmetric Geometry? A Bias-Variance Theory of Shared and Dual Projections

Maojun Sun, Yancheng Yuan, Jian Huang, Ruijian Han

arXiv:2609.32488v1cs.IRcs.LG

TL;DR

The paper addresses the unresolved criterion for choosing shared or dual query–document projections under finite-sample estimation noise. It derives an approximation and risk framework and selection rules, then finds a signal–sample-size crossover in a Gaussian retrieval experiment.

  • Problem

    A theoretical criterion remains unresolved for when dual projections’ flexibility justifies their additional estimation error.

    Source: §1 · p. 2

  • Method

    The paper characterizes shared and dual operator families, derives their approximation gap and a local Gaussian risk boundary, and develops risk-based selection rules.

    Source: §1 · p. 2, §2 · p. 2

  • Results

    At rank 8, the first positive grid mean occurred at n = 256 for 0.8-radian rotation and n = 64 for 1.2-radian rotation.

    Source: §5.1 · p. 6

  • Takeaways & Limitations

    In the local Gaussian model, Dual has lower risk precisely when squared directional signal exceeds the variance cost of its additional degrees of freedom.

    Source: §1 · p. 2

    The exact risk boundary assumes local, isotropic Gaussian operator noise.

    Source: §3 · p. 3

Abstract

from arXiv · show

Dense retrieval powers retrieval-augmented generation, semantic search, and question answering, yet the theoretical basis for choosing between shared and dual query-document projections remains unclear. We introduce a bias-variance theory for low-rank bilinear scoring. Shared projections induce positive-semidefinite operators, whereas dual projections realize arbitrary low-rank operators. We derive their exact approximation gap and prove a local Gaussian boundary: dual has lower risk exactly when squared directional signal exceeds the estimation cost of its additional degrees of freedom. This boundary motivates the Cross-fitted Asymmetry Risk Selector (CARS), which estimates reproducible directional signal from training pairs; its Gaussian counterpart admits exact selection-power and regret formulas. Guided by the theory, we run retrieval experiments across multiple datasets and embedding models. The mean Dual-minus-Shared NDCG@10 advantage more than doubles as query rotation increases from 0 degrees to 90 degrees. In the rank-sample-size grids, Shared wins 13 of 16 cells at n=32, whereas Dual wins all 32 cells at n=1024 and n=2048. Consistent with this shift, all 168 comparable operator-risk curves move toward Dual as training data grow. Compared to the two fixed-geometry baselines, CARS reduces held-out regret by 49-96% and achieves 90.1% mean geometry-selection accuracy.

1 INTRODUCTION

The paper asks when dual query–document projections’ extra directional flexibility is worth their estimation cost. It contrasts shared PSD operators with general low-rank dual operators and develops a bias–variance criterion for choosing between them.

1 INTRODUCTION

  • Shared projections induce positive-semidefinite scoring operators, while dual projections can represent arbitrary rank-at-most-r operators that need not be symmetric.The paper frames the choice as a tradeoff between directional flexibility and estimation noise.Source: §1 · pp. 1–2
  • The paper derives the exact approximation loss imposed by shared projections and a local Gaussian boundary comparing directional signal with dual’s additional estimation cost.It also develops risk-based selection rules and tests predicted signal and sample-size effects in retrieval and operator-risk experiments.Source: §1 · p. 2

2 OPERATOR GEOMETRY AND APPROXIMATION

Shared projections restrict the scoring operator to a rank-constrained PSD family, while dual projections realize the broader rank-constrained family. The section derives the exact approximation gap and relates it to retrieval separation and estimation complexity.

2 OPERATOR GEOMETRY AND APPROXIMATION

  • Shared operators form a strict subset of dual operators: shared maps yield PSD matrices, whereas dual maps can factor any matrix of rank at most r.The scoring margin is the Frobenius inner product of the operator with the query and positive–negative document outer product.Source: §2 · p. 2
  • The Frobenius-nearest shared approximation retains the largest at most r positive eigenvalues of the symmetric part of the target.Its exact excess squared error over the rank-r dual approximation is the shared minimum error minus the discarded singular-value tail; it is nonnegative because the shared family is nested inside the dual family.Source: §2 · p. 2
  • For whitened query and document views, the shared-minus-dual approximation gap equals the gap in optimal rank-r positive–negative separation under a unit negative-score second-moment constraint.When the whitened relevance moment is symmetric PSD, shared geometry is sufficient.Source: §2 · p. 3
  • The global Rademacher bound preserves the ordering R_n,s ≤ R_n,d but gives both classes the same ceiling, B R/√n.Thus this bound does not quantify the extra estimation cost of dual’s additional directions; the local noise model addresses that difference.Source: §2 · p. 3

3 THE BIAS–VARIANCE BOUNDARY

The section compares shared and dual projection risk in a local Gaussian model, where directional signal and the estimation cost of dual-only dimensions can be evaluated on a common scale. It then develops a noisy-data selection rule and a cross-fitted selector for real embeddings.

3 THE BIAS–VARIANCE BOUNDARY

  • The local Gaussian experiment places isotropic operator noise and directional departures from a rank-r shared target on the same n^-1/2 scale.This makes approximation gain and estimation cost directly comparable.Source: §3 · pp. 3–4
  • The dual-only tangent-space dimension is k = r(r − 1)/2 + r(p − r), covering within-subspace skew perturbations and differing left–right cross-subspace perturbations.The shared tangent space is contained in the dual tangent space.Source: §3 · p. 4
  • Dual has lower local asymptotic risk if and only if squared directional signal exceeds the variance cost of its k additional dimensions.Sharing removes noise in those directions but also discards signal there.Source: §3 · p. 4

3.1 A SELECTION RULE FROM NOISY DATA

  • SURE estimates each tangent-space projection’s risk and selects the family with the lower estimate; its risk difference is δ² − σ²k.Its dual-selection threshold is 2σ²k, accounting for the expected noise energy in the additional directions and the risk boundary.Source: §3.1 · pp. 4–5

3.2 CARS: CROSS-FITTED ASYMMETRY RISK SELECTOR

  • CARS selects Dual when cross-half fit agreement exceeds its sampling-variation penalty, using labeled training triples rather than a specified noise variance.Under the stated independent-half model, its expected score matches the local signal-minus-variance comparison, (δ² − σ²k)/n.Source: §3.2 · p. 5

4 EXPERIMENTAL SETTING

The experimental program tests the theory’s signal–sample-size prediction, whether directional signal can be estimated, how mismatch and data availability affect retrieval, and whether training-data-based geometry selection improves held-out operator risk.

4 EXPERIMENTAL SETTING

  • The retrieval comparisons use paired training subsets, frozen embeddings, relevance labels, and test queries across four base encoders; quality is measured by mean test-query NDCG@10.The encoders are E5-base-v2, BGE-base, GTE-base, and Contriever-MSMARCO.Source: §4 · p. 5
  • The controlled Gaussian retrieval experiment varies document-loading rotation, rank, and training-pair count, ranking each positive against 299 independent negatives.Five paired seeds are used at each mismatch–sample-size setting.Source: §4 · p. 6
  • The directional-signal study compares raw and uncertainty-corrected asymmetry against held-out operator-loss advantage in simulation and across 840 real-embedding operator fits.Disjoint held-out relevance moments define the Shared–Dual operator-loss advantage.Source: §4 · p. 6
  • The retrieval study rotates query vectors in eight relevance-informed planes across seven angles, then varies adapter rank and training size across five datasets and two encoders.Document embeddings, evaluation corpora, and relevance labels remain unchanged in the rotation study; complete collections are ranked by exact inner-product search.Source: §4 · p. 6
  • Held-out operator-risk curves use NQ and SciFact, while CARS selection is evaluated on five other named task folds with all four encoders.Selection regret compares the chosen geometry’s loss with the lower of Shared and Dual losses; accuracy counts cells choosing the lower-loss geometry, with ties assigned to Shared.Source: §4 · p. 6

5 RESULTS

The results show that the relative value of Shared and Dual projections varies with directional mismatch and training size. Cross-fitted selection performs better than fixed geometry choices, while corrected asymmetry scores better predict held-out operator fit.

5.1 RQ1: DOES THE PREDICTED BOUNDARY APPEAR?

  • At rank eight, the first positive grid mean occurs at n=256 for 0.8-radian rotation but at n=64 for 1.2-radian rotation.The local-Gaussian calibration matches risk limits within 0.25% and SURE selection probabilities within 0.001.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 RQ2: CAN DIRECTIONAL SIGNAL BE ESTIMATED?

5.3 RQ3: HOW DO MISMATCH, RANK, AND TRAINING SIZE AFFECT RETRIEVAL?

  • Across five datasets, the Dual–Shared NDCG@10 gap rises from about .0032 at 0° to .0146 at 90°, a 4.6-fold increase.All five dataset means increase between these endpoints.Source: §5.3 · p. 7, Figure 5 · p. 8
  • Shared wins 13 of 16 displayed cells at n=32, whereas Dual wins all 32 cells at n=1024 and n=2048.Within each grid, rank and rotation are fixed along the sample-size axis, and initially Shared-favored conditions switch to Dual as training data increase.Source: §5.3 · p. 7, Figure 6 · p. 8

5.4 RQ4: DOES OPERATOR RISK SHIFT WITH MORE DATA, AND CAN CARS SELECT THE BETTER GEOMETRY?

  • All 168 comparable sample-size slopes are positive across four datasets, and three of four NQ curves cross toward Dual-favored held-out fit.The results indicate that more data make directional structure easier to estimate.Source: §5.4 · p. 8, Figure 7 · p. 8
  • CARS reaches 90.1% mean selection accuracy and reduces mean regret by 49–96% versus the better fixed choice across four encoders.It wins 85/100 encoder–fold comparisons; when the preferred family varies across training sizes, ranks, and datasets, cross-fitted signal minus disagreement is more reliable than committing to one geometry.Source: §5.4 · pp. 8–9, Table 1 · p. 9

6 DISCUSSION AND FUTURE DIRECTIONS

The discussion argues that geometry choice should reflect both directional mismatch and available training data. It also identifies assumptions and extensions of the exact risk boundary.

6 DISCUSSION AND FUTURE DIRECTIONS

  • Geometry choice should account for both directional mismatch and available training data.The rotation experiment shows a larger Dual advantage as mismatch increases, while rank–sample-size grids show that more training data can reverse a low-sample preference for Shared.Source: §6 · p. 9
  • The exact risk boundary assumes local, isotropic Gaussian operator noise.Extending risk estimation to anisotropic noise and directly to ranking metrics would better align geometry selection with retrieval performance.Source: §6 · p. 9
  • Partial sharing is another proposed direction, with the number of separately parameterized directions selected alongside rank.This is presented as a future research direction.Source: §6 · p. 9

7 CONCLUSION

The paper characterizes approximation and estimation costs for Shared and Dual projections and reports retrieval and operator-risk evidence on their tradeoff. CARS reduces held-out regret relative to the better fixed choice across five datasets.

7 CONCLUSION

  • In the local Gaussian model, Dual projections have lower risk when squared directional signal exceeds σ^2r(2p − r − 1)/2.The paper characterizes the approximation and estimation costs of Shared and Dual projections for dense retrieval.Source: §7 · p. 9
  • CARS reduces held-out regret by 49–96% relative to the better fixed choice across five datasets.The framework may guide data-efficient retrieval-head design and other two-view representation problems in which inputs play different roles.Source: §7 · p. 9
  • Retrieval and held-out operator experiments show how mismatch, rank, and sample size affect the Shared–Dual tradeoff.The conclusion presents the framework as potentially useful for data-efficient retrieval-head design and other two-view representation problems.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

The reproducibility statement identifies the location of derivations, proofs, and experimental protocols, and describes seed and averaging practices.

REPRODUCIBILITY STATEMENT

  • Detailed derivations, proofs, and experimental protocols are provided in the appendix.The statement also notes that results are averaged across multiple seeds where applicable.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Randomized experiments use predefined seeds for data splits, training, and cross-fitting.Results are averaged across multiple seeds where applicable.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

The authors report AI assistance with language polishing, proof checking, brainstorming, and experimental-code review/debugging, while taking responsibility for the final work. The appendix provides proofs, experiment protocols, and held-out operator-risk trajectories cited in the main text.

AI USE STATEMENT

  • The authors used generative AI for language polishing, proof checking, brainstorming, and reviewing or debugging experimental code, and verified modified content.They state that they take full responsibility for the final proofs, code, analyses, and claims.Source: AI USE STATEMENT · p. 10

APPENDIX

  • The appendix provides proofs, protocols for reported experiments, and held-out operator-risk trajectories cited in the main text.Source: APPENDIX · p. 13

A COMPLETE PROOF OF THE APPROXIMATION THEOREM

The proof establishes the strict nesting of shared and dual operator classes and an exact shared-approximation error. It then develops tangent-space and local-projection arguments for comparing their estimation risks.

A COMPLETE PROOF OF THE APPROXIMATION THEOREM

  • The shared class contains only symmetric positive-semidefinite operators of rank at most r, and is strictly contained in the dual class.Source: A COMPLETE PROOF OF THE APPROXIMATION… · p. 13
  • The nearest shared approximation retains the largest at most r positive eigenvalues of the symmetric target component; its error includes unavoidable skew and negative-eigenvalue contributions.The proof compares this error with the rank-r SVD optimum in the dual class, yielding a nonnegative exact approximation gap.Source: A COMPLETE PROOF OF THE APPROXIMATION… · p. 13

B.1 TANGENT SPACES AND DIMENSIONS

  • The dual tangent space has dimension 2pr − r^2, with k = r(2p − r − 1)/2 dual-only directions.Source: §B.1 · pp. 13–14

B.2 DERIVATIVE OF NEAREST-POINT PROJECTION

  • Locally, the nearest-point projection changes to first order by projecting the perturbation onto the model's tangent space.For the local alternatives considered, the first-order risk boundary holds away from equality; rank-changing points and distant targets require different geometric treatment.Source: §B.2 · pp. 14–15

C PROOF OF THE SURE SELECTOR

The proofs derive the Gaussian SURE selection threshold and its exact selection and regret consequences, then characterize the shared-versus-dual approximation gap in two-view retrieval. They also show what distribution-free complexity bounds establish—and why they cannot expose the strict estimation penalty.

C PROOF OF THE SURE SELECTOR

  • SURE selects dual exactly when the squared observation in the dual-only directions exceeds 2σ² times their dimension gap; its selection power and regret follow from exact noncentral-χ² tail probabilities.The proof represents the observation in an orthonormal basis of the dual-only space and obtains the exact risk comparison.Source: C PROOF OF THE SURE SELECTOR · p. 15

C.1 EXPECTATION OF THE CROSS-FITTED CARS SCORE

  • For cross-fitted CARS, the expected score is unchanged by averaging repeated half-splits, but this identity does not give an exact finite-sample selection probability.With isotropic noise spanning the dual-only directions, its noise term matches the unscaled first-order boundary.Source: §C.1 · p. 15

D OPTIMAL RANK-r SEPARATION IN TWO-VIEW RETRIEVAL

  • For rank-r retrieval, the unrestricted optimum uses the top singular values of the whitened relevance moment, while the shared optimum uses positive eigenvalues of its symmetric part.The optima coincide when the whitened moment is symmetric PSD, including the stated matched-latent-view setting.Source: D OPTIMAL RANK-r SEPARATION IN… · p. 16
  • In the planar rotation example, the rank-one shared optimum is ρ cos θ, while the dual optimum remains ρ across the stated angle range.At zero rotation the optima match; the shared optimum decreases as the angle increases to π/2.Source: D OPTIMAL RANK-r SEPARATION IN… · p. 17

E A DISTRIBUTION-FREE BOUND AND ITS LIMITATION

  • For bounded Lipschitz margin loss, both families receive O(LBR/√n) deviation bounds, but the argument cannot establish a strict dual complexity penalty.The bound discards the leading-r spectrum, and boundedness permits degenerate or arbitrarily small designs; the local theorem adds regular-design and Gaussian-localization assumptions.Source: E A DISTRIBUTION-FREE BOUND AND ITS… · p. 17

F EXPERIMENTAL PROTOCOLS AND SUPPORTING EVIDENCE

The protocols test the theory through synthetic calibration, real-embedding retrieval, operator-risk trajectories, and geometry selection across varied datasets, encoders, ranks, and training sizes. The section also situates the methods among related work and states limits on what operator-level results imply for projected cosine.

F EXPERIMENTAL PROTOCOLS AND SUPPORTING EVIDENCE

  • Synthetic retrieval varies query–document rotation, training size, and fitted rank, while local-risk calibration compares simulated risks and SURE selection probabilities.A separate training-score diagnostic compares raw and uncertainty-corrected scores with held-out operator-loss advantage across 840 fits.Source: F EXPERIMENTAL PROTOCOLS AND… · p. 18

F.1 REAL-EMBEDDING RETRIEVAL AND OPERATOR-RISK PROTOCOLS

  • The full-corpus rotation experiment varies query rotation from 0° to 90° while keeping documents, relevance labels, and query splits fixed.It evaluates five datasets with BGE-base and GTE-base using rank-16 adapters trained on the same 1,024 queries.Source: §F.1 · p. 18
  • Rank-by-data grids cross seven sample sizes with four ranks, and held-out operator-risk trajectories compare normalized Shared-minus-Dual loss across nested training sizes.The trajectory figures cover 168 comparable curves; five-fold geometry selection uses held-out query folds and averages regret across sample-size–rank cells.Source: §F.1 · pp. 18–19, Figure 8 · p. 20, Figure 9 · p. 21

G ADDITIONAL RELATED WORK

  • The related-work discussion connects the analysis to metric and bilinear learning, dense-retrieval adaptation, CCA, low-rank projection, and risk estimation.It distinguishes pairwise-ranking theory and NDCG as addressing an order-based objective.Source: G ADDITIONAL RELATED WORK · p. 19

H IDENTIFIABILITY AND BOUNDARY CASES

  • Marginal covariance mismatch is neither necessary nor sufficient to identify relevance mismatch, because paired structure determines relevance in the examples.Proportional views can preserve cosine rankings despite different marginals, while a nonsymmetric orthogonal rotation can leave marginals equal.Source: H IDENTIFIABILITY AND BOUNDARY CASES · p. 19
  • The same bilinear operator can yield different projected-cosine rankings and arbitrarily different factor distances, so operator-level results require bilinear scoring or additional norm control.The passage illustrates this with two document rankings under factorizations whose product is the identity.Source: H IDENTIFIABILITY AND BOUNDARY CASES · p. 19
Loading 2609.32488v1…