PaperlayerView exact revision on arXiv

Source-linked AI summary

OBLIQ-IR: Training a Dense Retriever for Oblique Queries

Mahmoud Abdalla, Abdelrahman Abdallah, Shaimaa Sedek, Adam Jatowt

arXiv:2609.29649v1cs.IR

TL;DR

Oblique queries hinge on latent properties with little surface expression, and current pipelines face a first-stage retrieval bottleneck despite reliable candidate verification. OBLIQ-IR combines mechanism-matched synthetic queries with kNN-graph distillation from a frozen authorship encoder; it achieves new best non-oracle NDCG@10 on each of four reported tasks.

  • Problem

    Oblique retrieval depends on latent document properties with little surface expression, while existing pipelines show a first-stage retrieval gap despite reliable candidate verification.

    Source: §1 · pp. 1–2

  • Method

    OBLIQ-IR combines per-mechanism synthetic queries with writing-task kNN-graph distillation from a frozen authorship encoder.

    Source: §1 · p. 2

  • Results

    OBLIQ-IR achieves new best non-oracle NDCG@10 on every reported task, reaching 0.211 Writing-Style, 0.171 Math, 0.177 Twitter, and 0.281 Congress.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    The authors position OBLIQ-IR as a concrete answer to the call for retrieval architectures that make latent document attributes available at search time.

    Source: §8 · p. 9

    WildChat-Errors remains unsupported: tested variants fell below the dense baseline, and 40 queries with Recall@100 0.040 limit statistical power.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Oblique retrieval, as exemplified by OBLIQ-Bench, asks a retriever to find documents whose relevance is determined by a latent attribute (an implicit stance, an analogous reasoning technique, an authorial fingerprint, or a vague tip-of-the-tongue recollection) that has little or no surface expression in the document. State-of-the-art dense encoders and agentic search pipelines built around frontier language models exhibit a large first-stage bottleneck on these tasks, while the same language models reliably verify relevance when shown candidates. We address this with OBLIQ-IR, a single-vector dense retriever whose training mixture combines per-mechanism synthetic queries with a new form of cross-model supervision: kNN-graph distillation from a frozen authorship encoder, which transfers a style-versus-topic inductive bias into the student. A 3B retriever fine-tuned reaches 0.211 NDCG@10 on Writing-Style, 0.171 on Math, 0.177 on Twitter, and 0.281 on Congress, improving over the GPT-5.2 Multi-Hop Agent by \xr{0.010 to 0.150} NDCG@10 and over Gemini-2-Embedding by 0.027 to 0.222 NDCG@10 on every reported task. The code, data and checkpoints are available https://github.com/DataScienceUIBK/obliq-ir

1 Introduction

Oblique queries depend on latent attributes with little or no surface expression, and existing pipelines have a first-stage retrieval bottleneck even when language models can verify presented candidates. OBLIQ-IR combines mechanism-matched synthetic queries with authorship-encoder graph distillation and improves on the strongest published non-oracle pipelines across the four reported tasks.

1 Introduction

  • The relevance of an oblique query depends on a latent property—such as stance, reasoning technique, style, or rhetorical dynamic—with little surface expression in the relevant document.Topical-match retrieval does not capture this relevance criterion.Source: §1 · p. 1
  • Frontier-language-model pipelines score close to zero NDCG@10 on most OBLIQ-Bench tasks, although the same models reliably verify relevance when shown candidates.The paper identifies first-stage search as the bottleneck: downstream reasoning cannot recover a relevant document missing from the candidate pool.Source: §1 · p. 2
  • OBLIQ-IR trains a bi-encoder with mechanism-matched synthetic queries and, for writing, kNN-graph distillation from a frozen authorship encoder.The distillation transfers graph structure rather than the teacher’s similarity scores.Source: §1 · p. 2
  • OBLIQ-IR achieves new best non-oracle NDCG@10 on every reported task: 0.211 Writing-Style, 0.171 Math, 0.177 Twitter, and 0.281 Congress.The reported gains are 0.010 to 0.150 NDCG@10 over the GPT-5.2 Multi-Hop Agent and 0.027 to 0.222 over Gemini-2-Embedding.Source: §1 · p. 2
  • The results are supervised, and the authors make no claim of zero-shot transfer to unseen oblique tasks.Mechanism lenses, instruction prefixes, and reranker policy are chosen with knowledge of the benchmark’s mechanism taxonomy.Source: §1 · p. 2

2 Problem Formulation

The paper formalizes oblique retrieval as finding documents relevant through latent attributes, rather than surface overlap. It evaluates ranked retrieval with NDCG and Recall and describes a single-vector encoder trained with a contrastive objective.

2 Problem Formulation

  • Given a corpus, query, and cutoff k, a retriever returns an ordered list of k documents, evaluated with NDCG@k and Recall@k under Gold and Pooled judgments.Pooled judgments are used where available.Source: §2 · p. 2
  • An oblique query is relevant to a document through a latent property that has little surface expression in the document.The four covered properties are implicit stance, abstract math reasoning technique, authorial style, and the rhetorical dynamic of Congressional exchanges.Source: §2 · p. 2
  • OBLIQ-Bench groups its tasks by query mechanism: descriptive for Twitter, analogue for Math and Writing, and tip-of-the-tongue for Congress.The task properties are respectively stance, proof strategy, authorial style, and Congressional rhetorical dynamics.Source: §2 · p. 2
  • The single-vector retriever encodes queries and documents with a shared encoder and ranks documents by cosine similarity.Its training objective is a contrastive log-likelihood using a mined positive and mined or in-batch negatives.Source: §2 · p. 2
  • The training pipeline identifies each task’s latent attribute, synthesizes mechanism-lensed queries, adds authorship-graph pairs for Writing, and trains a bi-encoder.The figure’s inset reports Writing NDCG@10 rising from 0.096 with no distillation to 0.211 at the OBLIQ-IR dose; larger doses overfit on Writing while degrading other tasks.Source: Figure 2 · p. 3

3 Method

OBLIQ-IR combines synthetic queries tailored to each oblique-query mechanism with writing-only structural supervision from a frozen authorship encoder. The combined training mixture is used with a contrastive objective, followed by mechanism-selective reranking.

3.3 Training objective

  • The method applies a contrastive objective to the combined training mixture, using the same query-and-document encoder and parameters across tasks.A per-task instruction prefix supplies task identity at training and inference time.Source: §3 · p. 3, §3.3 · p. 4

3.1 Per-mechanism synthetic queries

  • Stage A describes a document’s latent attribute, and Stage B generates a query through a mechanism-matched lens while avoiding named entities and rare document vocabulary.The descriptive lens targets Twitter stance, the analogue lens targets Math proof strategy and Writing style, and the tip-of-the-tongue lens targets Congressional rhetorical dynamics.Source: §3.1 · p. 3

3.2 Stylometric knowledge distillation

  • For Writing, a frozen authorship encoder defines a style-space kNN graph whose directed edges become anchor-positive training pairs for the student.The encoder distinguishes same-author passages across topics from same-topic passages by different authors; the graph structure, not teacher scores, supplies the supervision.Source: §3.2 · pp. 3–4

3.4 Mechanism-selective reranking

  • At inference, the dense retriever returns 100 candidates for a task-specific reranker selected on held-out development NDCG@10.The available policies are identity, tournament-style reranking, and task-aware listwise reranking; Writing uses identity, and the reported results confirm that generic LLM reranking helps topical tasks but hurts Writing.Source: §3.4 · pp. 4–5

4 Experiment Setup

The experiments evaluate four OBLIQ-Bench tasks using official query splits and relevance judgments. The setup fine-tunes a 3B bi-encoder on synthesized task queries and writing-style distillation pairs, comparing it with published baselines and a no-distillation variant.

4 Experiment Setup

  • Evaluation covers Twitter-Conflict, Math Meta-Program, Writing-Style, and Congress Hearings using official query splits and qrels.The reported metric panel includes NDCG@10/50 and Recall@10/50/100, with Gold and available Pooled judgments.Source: §4 · p. 5
  • The model starts from a public 3B NV-Embed bi-encoder and is fine-tuned with LoRA adapters on attention and MLP projections.Document representations are mean-pooled token embeddings followed by L2 normalization, yielding a 3072-dimensional unit vector.Source: §4 · p. 5
  • Each task uses one synthesized query per sampled document with four BM25 hard negatives, while Writing also uses frozen-authorship-encoder distillation pairs.The authorship graph uses three neighbors per anchor.Source: §4 · p. 5
  • The baselines include BM25, dense embedding models, GPT-5.2 query rewriting and multi-hop search, and an oracle tournament; OBLIQ-IR is tested with and without distillation.The oracle is described as a verification-only ceiling.Source: §4 · p. 5

5 Experiments

OBLIQ-IR (full) wins NDCG@10 on all four evaluated tasks against the best published non-oracle baseline. The experiments also show that distillation and reranking have task-dependent effects.

5 Experiments

  • 0.010 NDCG@10 on Math against the GPT-5.2 Multi-Hop Agent and 0.036 on Twitter against the same baseline are the reported gains.Source: §5 · p. 5
  • 0.047 NDCG@10 on Writing against Gemini-2-Embedding and 0.098 on Congress against the GPT-5.2 Multi-Hop Agent are the reported gains.Source: §5 · p. 5
  • The recipe contribution is significant on Writing (p < 0.001, ∆= +0.116) and Congress (p < 0.001, ∆= +0.085), but marginal on Math (p = 0.08) and Twitter (p = 0.06).The reranker contribution is significant on Math (p = 0.007), Twitter (p = 0.014), and Congress (p < 0.001).Source: §5 · pp. 5–6
  • Removing authorship-derived training pairs drops Writing NDCG@10 from 0.211 to 0.096, while costing at most 0.021 on each other task; the chosen dose is the only one where every task improves or holds.At 31k pairs, Writing reaches 0.374 while the average of the other three tasks falls to 0.117.Source: §5 · p. 6
  • TourRank improves NDCG@10 over dense-only on Math (+0.031), Twitter (+0.027), and Congress (+0.094), but lowers Writing from 0.211 to 0.080; task-aware reranking raises Writing to 0.175.The selected policy uses identity on Writing and TourRank on the other three tasks.Source: §5 · pp. 6–7
  • OBLIQ-IR (full) wins NDCG@10 on all four tasks against the best published non-oracle baseline.The results are reported for Twitter-Conflict, Math Meta-Program, Writing-Style, and Congress Hearings.Source: §5 · p. 5, Table 2 · p. 6, Table 1 · p. 6

6 Analysis

The analysis finds that first-stage recall limits what reranking can recover, while reranking’s effects vary by task and can harm Writing. For authorship, graph distillation is a targeted signal; a neighbour-purity diagnostic is proposed to assess whether a specialist may help.

6.1 Recall is the bottleneck on first-stage retrieval

  • On Math, 87% of gold documents appear in the dense top-1000 versus 41% in the top-100; across all four tasks, top-1000 recall is roughly twice top-100 recall.The authors identify adapting later stages to a larger candidate pool as a tractable direction.Source: §6.1 · p. 7, Table 6 · p. 7

6.2 Per-query failure analysis

  • On Twitter, reranking hurts 76 queries and helps 93, while 178 of 254 Congress queries score zero under both pipelines; 137 of 512 Writing queries also score zero.Writing’s reranker is identity by policy, so its zero changes do not imply that all Writing queries succeed.Source: §6.2 · p. 7, Table 7 · p. 8

6.3 Why does reranking hurt Writing?

  • 0.211 to 0.080 NDCG@10: generic TourRank degrades Writing, while a style-aware prompt recovers it to 0.175 but remains below dense-only retrieval.The authors attribute TourRank’s drift toward topic matching to its lack of information about authorship features; Figure 3 shows gains concentrated near the top of the ranking.Source: §6.3 · pp. 7–8, Figure 3 · p. 9

6.4 Cost of the full pipeline

  • The dense pipeline takes roughly 48 ms per query; the task-aware reranker adds about 0.95 s, and TourRank at Y=5 adds 8.2 s.The full pipeline is reported as one to two orders of magnitude cheaper than the GPT-5.2 Multi-Hop Agent while exceeding its NDCG@10 on Twitter, Writing, and Congress, and matching it on Math.Source: §6.4 · p. 8, Table 8 · p. 8

6.5 When does kNN-graph distillation apply?

  • kNN-graph distillation targets authorship, where the latent attribute has no topical footprint; per-mechanism queries alone exceed the strongest published non-oracle baselines on Twitter and Congress.A one-pass, no-training purity diagnostic measures whether a specialist’s neighbours share the target attribute; replacing teacher neighbours with random documents collapses Writing from 0.223 to 0.001, and the effect persists across neighbourhood sizes kϕ ∈ {1, 3, 5, 10}. The authors generalize the recipe to frozen models whose neighbour structure aligns with the attribute, while finding a specialist for WildChat-Errors remains open.Source: §6.5 · pp. 8–9

7 Related Work

Related work includes dense bi-encoders that perform strongly on topical benchmarks but underperform on reasoning- or stylometry-oriented benchmarks. Other lines condition retrievers on task instructions or synthesize training queries, while the paper points to additional retrieval and authorship work in its appendix.

7 Related Work

  • Dense bi-encoders such as NV-Embed, GTE, E5, and Qwen3-Embedding reach state of the art on MTEB and BEIR but underperform on BRIGHT, Tempo!, and OBLIQ-Bench.The cited families convert decoder-only language models into retrievers.Source: §7 · p. 9
  • Other approaches condition a single retriever on natural-language task instructions or synthesize training queries with an instruction-tuned language model; ReasonIR combines both ideas for BRIGHT.Source: §7 · p. 9
  • The related-work discussion also covers late-interaction retrieval, listwise LLM reranking, and authorship modelling in Appendix A.Source: §7 · p. 9

8 Conclusion

The paper introduces OBLIQ-IR, a 3B single-vector dense retriever trained with per-mechanism synthetic queries and kNN-graph distillation from a frozen authorship encoder. It outperforms the strongest published non-oracle pipelines on four evaluated OBLIQ-Bench tasks, while first-stage recall remains a constraint.

8 Conclusion

  • OBLIQ-IR combines per-mechanism synthetic queries with kNN-graph distillation from a frozen authorship encoder, outperforming the strongest published non-oracle pipelines on the four evaluated tasks.Neither stage uses OBLIQ qrels, and no frontier-scale model is called at training or retrieval time.Source: §8 · p. 9
  • The method offers a concrete response to the call for retrieval architectures that make latent document attributes available at search time.The conclusion frames this as a response to Tchuindjo et al. (2026).Source: §8 · p. 9
  • The first-stage bottleneck is narrowed rather than closed, with recall remaining the binding constraint on tasks where OBLIQ-IR gains least.Source: §8 · p. 9

Limitations

OBLIQ-IR leaves several boundaries unresolved: WildChat-Errors remains unclaimed, transfer beyond the benchmark is unmeasured, and the training signals and reranking pipeline have practical and auditing limits.

Limitations

  • WildChat-Errors remains unclaimed after training variants fell below the dense baseline; Recall@100 was 0.040 over 507,729 documents, and the split has only 40 queries.The authors describe the task as recall-bound and say these conditions leave little statistical power for differences of this magnitude.Source: Limitations · p. 10
  • The recipe is supervised on OBLIQ-Bench’s mechanism taxonomy, and transfer to unseen tasks or mechanisms remains unevaluated.The reranker policy is also selected on development NDCG; a held-out-mechanism or leave-one-task-out evaluation is left open.Source: Limitations · p. 10
  • Synthetic queries come from one 27B instruction model, whose biases may bound query diversity and shape which latent attributes the retriever learns.The authors also did not measure the effect of changing generators or isolate mechanism-matched lenses from a generic hard-query prompt.Source: Limitations · p. 10
  • The Reddit-trained authorship teacher may encode demographic and dialectal correlates, but its graph was not audited for them; authorship retrieval can also enable deanonymization.The §6.5 diagnostic measures target-attribute alignment, not the absence of unwanted correlates; deployment over people’s writing should be audited on both counts.Source: Limitations · p. 10
  • The gains narrow rather than close the first-stage bottleneck, and TourRank dominates latency at roughly 8.2 seconds per query versus 48 ms for dense retrieval.The authors call the dense-only configuration the deployable core of the system.Source: Limitations · p. 10

Appendices

The appendices collect supplementary prompts, diagnostics, comparisons, and sweeps. The related work situates OBLIQ-IR’s synthetic-query and graph-distillation approach, while the methods materials describe query generation and task-specific instructions.

Appendices

  • The appendices include a kNN-graph purity diagnostic, synthetic-query overlap and leakage analysis, a direct teacher-retrieval comparison, WildChat-Errors results, and additional sanity checks and sweeps.They also provide confidence intervals, hyperparameters, prompt listings, and an analysis of the remaining gap to the Oracle GPT-5.2 Tournament ceiling.Source: Appendices · p. 12

A Related Work

  • Prior dense-retrieval work motivates the 3B NV-Embed initialization, while OBLIQ-IR extends ReasonIR-style synthetic-query training with mechanism-specific lenses and stylometric graph distillation.The paper distinguishes its distillation from score transfer: the teacher supplies kNN-graph topology, defining positives while the student controls its metric.Source: A Related Work · pp. 13–14
  • The retriever stays single-vector because the corpora permit fast flat cosine search, and its reported scores meet or exceed the published late-interaction baseline on each reported task.The authorship graph signal also transfers at the level of whole-document representations; generic TourRank falls below the dense baseline on Writing-Style.Source: A Related Work · pp. 13–14

B Synthetic data generation prompts

  • Synthetic-query generation uses Stage A to describe a document’s latent attribute and Stage B to draft queries answerable by other documents sharing that attribute while avoiding lexical overlap.Qwen3.6-27B is the generator throughout.Source: B Synthetic data generation prompts · p. 14

B.3 Per-task instruction prefix

  • Task-specific instruction prefixes specify latent-attribute retrieval for Twitter, Math, and Writing, and retrieval of a Congressional passage from a lossy recollection.The format follows ReasonIR’s two-element query format.Source: §B.3 · p. 14

D Bootstrap confidence intervals

  • Paired bootstrap 95% confidence intervals are computed over per-query NDCG@10 with 10,000 resamples for recipe and reranker contrasts.Recipe contribution compares the full pipeline with the no-distillation ablation; reranker contribution compares against dense-only.Source: D Bootstrap confidence intervals · p. 15, Table D.1 · p. 15

E LLM reranking is task-dependent and non-monotonic

Reranking compute has task-dependent, sometimes non-monotonic effects: TourRank depth peaks differ across tasks, while the best listwise candidate depth also varies by task. These sweeps show why one uniform reranking budget need not be optimal.

F Listwise reranker depth sweep

  • The task-aware listwise reranker’s best input depth is K=20 for Math and Twitter, K=10 for Writing, and K=50 for Congress.At K=100, the language model saturates on every task except Twitter.Source: Table F.1 · p. 16

E LLM reranking is task-dependent and non-monotonic

  • Math peaks at TourRank Y=1 (NDCG@10 0.176 versus 0.171 at paper-final Y=5), Twitter peaks at Y=3 (0.183 versus 0.163 at Y=10), Congress rises through Y=5, and Writing declines with added rounds.The paper notes that a task-uniform Y leaves performance on the table except on Congress; per-task Y selection on development data is described as essentially free compute optimization.Source: E LLM reranking is task-dependent and… · pp. 15–16, Table G.1 · p. 16

G TourRank tournament-depth sweep

  • TourRank’s depth sweep is non-monotonic on three tasks, with task-specific peaks; the main results use Y=5 for comparability with the original TourRank paper.Congress increases through Y=5, while Writing declines as additional rounds move rankings away from authorship and toward topic.Source: Table G.1 · p. 16

H Teacher-graph ablations

The ablations show that the frozen authorship encoder’s neighborhood structure, rather than simply adding writing-domain examples, supplies the useful distillation signal. The appendices assess when that signal transfers, how synthetic queries behave, and where the system remains limited.

H Teacher-graph ablations

  • Replacing teacher neighbors with random writing documents collapses Writing from .223 to .001 and Congress from .187 to .000, despite identical anchors, negatives, and dose.The collapsed model sends 366 of 512 writing queries to one top result, with only 20 distinct top results overall.Source: H Teacher-graph ablations · p. 17, Table H.1 · p. 18
  • Writing NDCG@10 is .203 / .223 / .234 / .249 for kϕ = 1/3/5/10, and every setting exceeds the .096 no-distillation floor.A second authorship-encoder checkpoint reaches Writing .239; the authors report that its inductive bias survives transfer from Reddit authorship data to long-form prose.Source: H Teacher-graph ablations · pp. 17–18

I Choosing a specialist: kNN-graph purity

  • A low-cost specialist screen measures same-attribute nearest-neighbor lift over the random-pair rate, then distills only when lift is large.The reported 1.8%-purity graph collapses Writing to .001, whereas the 45.2%-purity graph reaches .223.Source: I Choosing a specialist: kNN-graph… · p. 18

J Synthetic query quality

  • Synthetic-query entity leakage is 1.7–6.4%, and lexical overlap is comparable to official queries on three tasks, but Congress coverage is .359 versus .102.Evaluation uses official vocabulary-disjoint Congress queries; the authors identify the synthetic tip-of-the-tongue lens as the weakest generation component.Source: J Synthetic query quality · p. 19

K The frozen teacher as a direct retriever

  • The frozen teacher reaches .206 Writing NDCG@10, while the student reaches .211 and exceeds it by +.024 NDCG@50 and +.061 Recall@100.The teacher is a Writing-only baseline; the student uses just 5,000 of its 31,167 graph edges and shares one index across the four tasks.Source: K The frozen teacher as a direct… · pp. 19–20

L Full benchmark results, including WildChat-Errors

M Random-shuffle sanity check

N What we tried that did not work

Loading 2609.29649v1…