PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieved but not ranked: surface-form bias in structural retrieval, from mathematics to agent trajectories

Nabira Rashid, Manolis Kellis

arXiv:2609.01556v1cs.LGcs.AIcs.IR

TL;DR

Embedding retrieval is poorly understood when surface wording and underlying structure diverge, so this paper evaluates that separation across mathematics and agent trajectories under one protocol. It finds that retrieval follows literal surface content, while lexical and LLM reranking behave differently across benchmark regimes and judges. The results support reading structural-retrieval scores together with benchmark surface variation and reranker provenance.

  • Problem

    Evidence is limited on whether embedding retrieval follows surface wording or underlying structure when the two are deliberately separated across different domains.

  • Method

    The paper evaluates mathematics and agent-trajectory retrieval under one shared protocol with structural relevance definitions, lexical controls, and three LLM rerankers.

  • Results

    Across both domains, retrieval anchors on literal surface content; lexical reranking harms mathematics but helps trajectories, while LLM reranking recovers gap in both with judge- and domain-dependent magnitudes.

  • Takeaways & Limitations

    Structural-retrieval evaluations should be interpreted alongside benchmark surface-variation regimes and the provenance and judge dependence of reranking results.

  • Takeaways & Limitations

    The trajectory evaluation uses one dataset family, 336 corpus items, and 118 queries, while three judges provide a small and variable sample of judge behavior.

Abstract

from arXiv · show

We evaluate embedding retrieval where surface form and meaning are pulled apart on purpose: retrieving items that share underlying structure but not wording, in two unrelated domains under one protocol, competition mathematics (MathNet-Retrieve; 500 queries, 117,088-item corpus) and embodied-agent trajectories (ALFWorld-derived; 118 queries, 336 trajectories). In mathematics the failure is complete: strict Hit@1 at the heaviest disguise tier is 0.0% for both production embedders (bootstrap 95% CI [0.0, 0.0]) while the correct item sits in the top 10 nearly always, and in 95.2 to 99.8% of misses the winner is more lexically similar to the query than the correct answer. In trajectories, where surface variation is incidental, the same models land at or near hypergeometric chance when gold must involve a different object, and below chance for all three embedders once gold must differ in object and receptacle: retrieval anchors on literal tokens, not task structure. A lexical reranker control hurts in mathematics and helps in trajectories (closing 26 to 36% of the gap, CIs excluding zero); its sign reveals whether a benchmark's surface variation is adversarial or incidental. An LLM reranker recovers 5 to 63% of the gap in mathematics and 43 to 76% in trajectories; direction replicates across three judges (all 21 cells positive), but effect sizes, tier profiles, and the outlier judge change with domain (paired differences excluding zero everywhere). Mathematics gains concentrate on well-known competitions (+19.8 points, CI [+6.7, +33.2], one of six cells), so part of the recovery is memorization. In a paired downstream experiment (210 queries, graders at 96 to 99% agreement), oracle retrieval was indistinguishable from adversarially bad retrieval (McNemar p = 0.678); the solver's 69.5% zero-shot accuracy is largely a truncation proxy (97 to 100% on finished answers), leaving no headroom.

1 Introduction

The paper evaluates structural retrieval across mathematics and agent trajectories using one shared protocol, separating surface wording from underlying meaning. It finds that retrieval follows literal surface content, while reranking and benchmark construction determine how much structure can be recovered.

  • Motivation: Structural retrieval deliberately separates surface wording from underlying structure across mathematics problems and agent trajectories.The shared protocol tests whether retrieval follows wording or task structure when they diverge.
  • Cross-domain synthesis: The lexical reranker harms mathematics but helps trajectories, because lexical overlap is adversarial in MathNet and incidental in ALFWorld.Its sign is proposed as a cheap diagnostic of a benchmark’s surface-variation regime.
  • Cross-domain synthesis: LLM reranking recovers gap in both domains, but effect sizes, tier profiles, and outlier judges vary by domain and judge.The recovery direction replicates across judges, whereas magnitudes vary by more than eightfold and memorization contributes where provenance varies.
  • Evaluation design: The evaluation spans 500 MathNet-Retrieve queries over 117,088 items and 118 ALFWorld-derived trajectory queries over 336 trajectories.Both domains use the same retrieval, reranking, and uncertainty-reporting framework.

4 Domain 1: mathematical problems

In mathematical structural retrieval, the correct answer is usually retrieved but ranked behind lexically similar near-misses; reranking helps, but its gains vary by judge and may partly reflect memorization.

  • Baseline retrieval: 0.0% strict Hit@1 at hard tier contrasts with 99–100% Hit@10, showing retrieval succeeds while ranking fails.The hard-tier zero has bootstrap 95% CI [0.0, 0.0] for both production embedders.
  • Failure structure: 95.2–99.8% of misses select a false positive that is lexically closer to the query than gold.The planted near-miss accounts for 84–98% of misses, depending on embedder and tier.
  • Reranking: Lexical reranking loses 9.1% and 4.8% of the recoverable gap at easy tier for Gemini-emb and Qwen-emb, respectively.The control is flat at hard tier because surface similarity is anti-correlated with correctness in this construction.
  • LLM reranking: All twelve LLM-judge-by-configuration cells are positive, but judge gains range from 5.4% to 63.3% of the recoverable gap across tiers and configurations.Haiku-j is strongest at easy tier but weakest at hard tier, while Gemini-j and GLM-j show different tier profiles.
  • Contamination: One of six hard-tier contamination cells is confidently non-null, with a +19.8-point gain for well-known competitions over the pooled remainder.The other five cells point in the same direction but are individually indistinguishable from zero; the effect is not established as judge-independent.
  • Deployment divergence: Two deployments of identical Qwen3-Embedding-8B weights disagree significantly on hard-tier Hit@10 despite mean pairwise cosine 0.9947.There were 17 discordant queries in one direction versus 1 in the other, with McNemar exact p = 0.00014.

5 Domain 2: agent trajectories

On ALFWorld-derived trajectories, retrieval generally follows literal task tokens rather than structural similarity: stricter cross-object and cross-receptacle gold definitions expose below-chance ranking. A lexical control helps in this domain, while LLM reranking improves retrieval but varies by judge and query subset.

  • Setup: The corpus contains 336 AgentInstruct trajectories and 118 queries, including 78 added from ALFWorld’s public valid_unseen split.The planned 150 added queries were unreachable without a heavy simulation dependency, so the shortfall was reported.
  • Evaluation: Strict gold requires the same task type with a different target object, while definition (ii) additionally requires a different receptacle; lenient gold accepts any same-task-type trajectory.Chance is computed per query using the exact hypergeometric baseline because gold-set sizes vary.
  • Baseline retrieval: Under definition (ii), all three embedders fall below exact hypergeometric chance, showing that rankings are actively steered away from cross-object, cross-receptacle structural matches.For definition (i), pooled strict Hit@1 is 17.8% for Qwen-emb, 15.3% for Gemini-emb, and 9.3% for MiniLM against 15.3% chance.
  • Limitations: The two trajectory query subsets behave differently: strict Hit@1 is lower on the original 40 than on the raw validation-split 78 for both production embedders.This difference is confounded with phrasing style and is identified as a benchmark-construction effect.
  • Lexical control: +25.9%, +36.4%, and +32.1% of the strict gap were closed by the lexical control for Qwen-emb, Gemini-emb, and MiniLM, respectively.All confidence intervals excluded zero, and a verb-only variant reproduced most of the effect on the original-40 subset.
  • Lexical control: The lexical control’s sign depends on benchmark construction: incidental surface variation makes overlap useful in trajectories, unlike adversarial paraphrasing in mathematics.The control is inexpensive and its sign can identify the benchmark regime before more expensive evaluations.
  • LLM reranking: LLM reranking closed substantial strict gaps for every embedder, but judge behavior reversed relative to mathematics and differed across query subsets.Haiku-j and Gemini-j converged on 43.9% to 48.2% gap closure on this domain, while GLM-j was the upward outlier; Gemini-j gains were lower on templated new-78 queries than on the original 40.

6 Downstream: does retrieval quality reach the solver?

A paired downstream experiment found no measurable accuracy advantage for oracle retrieval over adversarially bad retrieval. The solver’s apparent zero-shot performance was largely tied to whether derivations finished within the token budget, leaving little headroom for retrieval context.

  • Setup: The downstream study tested no retrieval, adversarially reranked top-1 retrieval, and the gold equivalent with its solution on 210 mathematics queries.Two independent graders achieved 96.2% to 98.6% per-condition binary agreement.
  • Downstream accuracy: Accuracy remained 67% to 70% in every condition, and McNemar’s test comparing no retrieval with gold gave p = 0.678.The paired counts were 13 queries where context hurt versus 10 where it helped; both graders concurred on the null pattern.
  • Downstream accuracy: On the 64 queries that failed zero-shot, gold context recovered 10 but disrupted 13 already solved, producing a net change of minus 3, identical to deliberately bad retrieval.For this solver, oracle retrieval was therefore indistinguishable from adversarial retrieval.
  • Mechanism: Among 127 queries whose answers finished within budget in all three conditions, accuracy was 97.6% to 100% in every condition.This complete-answers-only check identifies budget completion as the binding constraint rather than a conjectured moderator.
  • Mechanism: The headline 69.5% zero-shot accuracy is closer to a derivation-fit-to-budget proxy than to a measure of solving ability.The null follows from essentially no headroom on problems the solver could finish.

7 Cross-domain synthesis

Across mathematics and trajectories, embedding retrieval follows literal surface content when surface and structure diverge. Lexical and LLM reranking reveal benchmark- and judge-dependent recovery, while downstream gains remain constrained.

  • Cross-domain finding: Below-chance trajectory retrieval appears when the gold differs from both the query’s literal object and receptacle.The cross-domain mechanism is literal surface ranking rather than task-structure ranking.
  • Control: The lexical control’s sign diagnoses benchmark construction: it is harmful under adversarial variation and informative under incidental variation.The control is cheap to run and distinguishes the two regimes.
  • Reranking: LLM reranking recovers substantial gap in both regimes, but effect sizes vary more than eightfold across judges and tier profiles disagree.Each domain has a different outlier judge, and one judge’s gains fall by more than a third across query styles.
  • Pipeline errors: Judge misses in trajectories are disguise failures over the correct structure, with siblings comprising 72 to 88% of misses in every cell.Embedder errors instead involve structural confusions, so the two pipeline stages fail differently.
  • Downstream link: Under a paired downstream design, one competent solver extracted nothing from oracle context because complete-answer analysis indicated near-zero effective headroom.This cautions against reading zero-shot accuracy directly under token budgets.

9 Limitations

The study’s conclusions are bounded by narrow sampling, judge variability, provenance confounds, and uncertainty that is mostly within the observed samples. Its broadest claim is that surface-over-structure retrieval persists across two unlike domains, but reranking magnitudes do not transfer cleanly.

  • Sampling: Mathematics uses one fixed seed, while the trajectory domain has 118 queries, one dataset family, and one 336-item corpus.Bootstrap confidence intervals therefore quantify within-sample uncertainty only for mathematics.
  • Judge variability: Judge variability is undersampled: three judges differ by more than eightfold, have domain-specific outliers, and disagree in tier profiles.One additional judge was structurally infeasible, further limiting the judge comparison.
  • Provenance: Provenance and phrasing style are confounded, and contamination attribution is correlational rather than causal.Family-level affinity to Gemini-generated paraphrases may contribute to Gemini-j’s mathematics advantage.
  • Scope: Across two unlike domains, embedding retrieval tracks literal surface content over underlying structure when the two are separable.The conclusion concerns retrieval behavior under the benchmark’s structural-relevance conditions.
  • Reranking limits: Reranking recovers much of what ranking loses, but its magnitude does not transfer across judges, prompts, domains, or query styles.Part of the recovery is attributable to memorization where provenance varies.
  • Evaluation implication: The lexical reranker’s sign reveals whether benchmark surface variation is adversarial or incidental, motivating routine reporting of this control and exact chance baselines.This is presented as the study’s most useful instrument.

A Verbatim judge prompts

The appendix specifies terse and chain-of-thought judge prompts for mathematics, all asking judges to identify shared underlying technique while ignoring surface similarity. The chain-of-thought variants differ mainly in reasoning and answer-format instructions.

  • A Verbatim judge prompts: All three judges use identical prompt wording within each domain, with one terse prompt per domain.The same template object is imported across judge scripts.
  • A.1 Mathematics domain, terse prompt: The mathematics prompt presents one anchor and 10 candidates, exactly one of which uses the same underlying technique despite possible surface differences.The operational criterion is the core idea actually used to solve the problems.
  • A.1 Mathematics domain, terse prompt: Judges are instructed to ignore shared variables, wording, language, story framing, and numbers when selecting the matching technique.Surface similarity can accompany a different technique, while dissimilar wording can accompany the same one.
  • A.1 Mathematics domain, terse prompt: The terse prompt directs judges to focus on the mathematical concept, theorem, or method needed to solve each problem.It asks which candidate needs the same one as the anchor.
  • A.1 Mathematics domain, terse prompt: The terse variant requires only the candidate number from 1 to 10, with no explanation or other text.This defines a minimal structured output.
  • A.2 Mathematics domain, chain-of-thought prompt: The chain-of-thought mathematics prompt preserves the same task framing but adds step-by-step reasoning and a structured final-answer marker.The final answer must appear on its own line as FINAL ANSWER: <candidate number>.
  • A.2 Mathematics domain, chain-of-thought prompt: The chain-of-thought prompt repeats the same candidate setup, surface-similarity exclusions, and technique-focused matching instructions.It then asks judges to state the anchor technique, assess each candidate, and conclude.

A.3 Mathematics domain, concise chain-of-thought variant (diagnostic only)

The concise chain-of-thought variant was a diagnostic prompt, not a headline configuration. It preserved technique-based matching while explicitly limiting reasoning length and output format.

  • A.3 Mathematics domain, concise chain-of-thought variant: 63.3% of GLM-j’s full chain-of-thought run truncated mid-reasoning, so the concise variant was tested only diagnostically.It used a 30-query sample and never replaced the full-run prompt.
  • A.3 Mathematics domain, concise chain-of-thought variant: The diagnostic prompt retained the requirement that exactly one candidate shares the anchor’s underlying mathematical technique despite surface differences.The criterion remains the core idea actually used to solve the problem.
  • A.3 Mathematics domain, concise chain-of-thought variant: It again instructs judges to ignore shared wording, language, framing, numbers, and other surface-level similarities.A nearly identical candidate may still use a different technique, and vice versa.
  • A.3 Mathematics domain, concise chain-of-thought variant: The prompt focuses judgments on the mathematical concept, theorem, or method needed to solve each problem.It asks which candidate needs the same one as the anchor.
  • A.3 Mathematics domain, concise chain-of-thought variant: The variant requests concise reasoning: one sentence for the anchor technique and one short phrase per candidate with match or no match.It explicitly rejects full solution write-ups and asks judges to choose directly when torn.
  • A.3 Mathematics domain, concise chain-of-thought variant: The response format still requires a final answer on its own line as FINAL ANSWER: <candidate number>.This marker structures extraction of the selected candidate.

A.4 Trajectory domain, terse prompt (Gemini-j, GLM-j, Haiku-j)

The trajectory task asks retrieval systems to identify the candidate sharing an anchor’s underlying procedure, while ignoring object names, receptacles, and surface phrasing.

  • Evaluation variant: No chain-of-thought variant is used for this domain; chain-of-thought was run only in mathematics.
  • Task definition: Each query presents an agent instruction and 10 candidate trajectories, with exactly one candidate following the same underlying procedure.Procedures include transformations such as placement, cleaning then placement, heating then placement, cooling then placement, examination with light, and two-object placement.
  • Structural criterion: Matching object names, receptacles, or similar wording does not establish procedural equivalence.A same-object candidate may use a different procedure, whereas a different-object candidate may use the same procedure.
  • Response protocol: The decision should depend only on the action sequence or transformation type required by each trajectory.The intended output is only the selected candidate number from 1 through 10.

B Worked examples

A worked mathematics example contrasts a paraphrased inequality query with its equivalent reformulation, showing that embedding retrieval ranks a lexically matching near-miss above the gold source.

  • Mathematics query: The representative query asks for positive integers n such that an inequality holds for every real x ≥ n.
  • Gold reformulation: The easy-tier gold reformulation instead asks for every natural number m ≥ 1 satisfying the inequality for any real x ≥ m.
  • Retrieval outcome: The rank-1 candidate is character-identical to the query with the inequality reversed, while the gold item appears at rank four.

B.2 Trajectories: a sibling miss

A trajectory example shows a reranker selecting a same-object, different-receptacle sibling instead of the structurally correct trajectory involving a different object.

  • Query: The query asks to place a remote control on the sofa in a simple pick-and-place task.
  • Sibling miss: GLM-j selects a trajectory placing a remote control in an armchair, creating a same-object, different-receptacle sibling miss.
  • Structural match: The strict-gold trajectory places newspaper on the sofa, preserving task type and receptacle while changing the object.
  • Miss taxonomy: Table C.1 counts rank-1 mathematics misses by category, while Table C.2 breaks down sibling dominance across judge-by-embedder cells.

C.2 Trajectories: LLM-reranker miss taxonomy, all nine cells

The trajectory miss taxonomy reports sibling failures across reranker cells, while the grading protocol evaluates candidate mathematics solutions against references on a 0–7 correctness scale.

  • Trajectory miss taxonomy: Table C.2 reports percentages of strict misses within each judge-by-embedder cell for the sibling-dominance analysis.
  • Grading setup: A single grading prompt is instantiated per query and condition with the problem, reference solution, and candidate solution substituted into it.
  • Scoring rubric: Graders assess candidate mathematical correctness from 0 to 7 by comparing reasoning and final conclusions with the reference, independent of writing style or language.
  • Output format: The grading response is restricted to the integer score alone, without explanation or other text.
Loading 2609.01556v1…