LLM-guided Hierarchical Search for End-to-end Reasoning Intensive Retrieval

1 Introduction

• On reasoning-intensive workloads, embedding models can miss relevant documents from the top-k, leaving an LLM verifier unable to recover them.This challenges the standard retrieve-then-rerank recipe, which works well when the embedding model puts the right documents in its shortlist.Source: §1 · p. 2
• Query rewriting and agentic retrieval keep the LLM upstream of the retriever, so documents are still selected by the cheap retriever.The paper notes that rewrites may reflect the LLM’s parametric beliefs, while richer queries can still be mishandled by the embedder.Source: §1 · p. 2
• LATTICE lets an LLM search directly through a semantic tree whose leaves are documents and whose internal nodes summarize the documents below them.At query time, the LLM scores children at selected nodes and descends into promising subtrees; no embedding model is used at search time.Source: §1 · pp. 2–3
• LATTICE builds its hierarchy top-down using LLM judgments over multi-level summaries, then calibrates traversal with cross-branch references and path-aggregated scores.These mechanisms address noisy, slate-dependent scores that can cause greedy traversal to abandon a branch that later proves correct.Source: §1 · p. 3
• 46.7 nDCG@10 overall lets base LATTICE exceed BGE-Reasoner-0928 (46.4) by 0.3 and trail DIVER v3 (46.8) by 0.1; LATTICE++ reaches 49.1 nDCG@10.Base LATTICE uses a single zero-shot LLM; LATTICE++ fuses LATTICE with BM25 and a dense retriever.Source: §1 · p. 3
• Because each search step is an LLM call, LATTICE is best suited to applications that can afford tens of seconds of latency per query.The paper names deep research, legal QA, and technical question answering as examples.Source: §1 · p. 3

2.1 Offline Hierarchical Search Index Construction

• Embedding-based bottom-up indices perform poorly at the studied corpus sizes, especially for passages from independent sources; LATTICE instead uses LLMs to partition clusters.The authors attribute the bottom-up weakness to noisy higher-level embedding clusters that can lock conceptual conflations into summaries.Source: §2.1 · p. 4
• LATTICE recursively partitions oversized nodes with LLMs, assigning each new topic an LLM-written description and its mapped leaves.It selects the most specific deduplicated summary level that fits the clustering model’s context budget, then repeats until nodes have at most M children.Source: §2.1 · pp. 4–5
• For corpora with natural parent–child structure or too many leaves for direct top-down clustering, LATTICE first forms clusters or pseudo-leaves and then applies top-down partitioning.This reduces the leaf set for LLM clustering; natural parent structure is also preserved for later passage evaluation.Source: §2.1 · p. 6

2.2 Online LLM-guided Hierarchical Search

• At each expansion, the search LLM scores a node’s children alongside recurring cross-branch reference candidates, and a per-slate-bias model estimates slate-independent latent scores.Repeated candidates constrain their latent scores across slates, distinguishing the estimator from averaging raw scores that confound relevance with candidate strength.Source: §2.2 · pp. 7–8
• LATTICE accumulates latent scores along root-to-node paths to prioritize the frontier, then returns the top-scoring leaves after the configured search iterations.With the default α = 0.5, deeper splits receive more weight because they use finer-grained summaries; the traversal addresses local scores’ limited reach across the tree.Source: Figure 3 · p. 6, §2.2 · pp. 7–8

3.2 Main Results on BRIGHT

• 49.1 nDCG@10 for LATTICE++ exceeds DIVER v3 (46.8) and BGE-Reasoner-0928 (46.4); base LATTICE reaches 46.7, near those fine-tuned ensembles.LATTICE++ also sets a new best on 8 of 16 Table 1 columns.Source: Table 1 · p. 9, §3.2 · p. 9
• After a moderate token budget, LATTICE overtakes sliding-window reranking on Biology (+5.3 nDCG@10), Robotics (+5.0), and Earth Science (+3.8), and matches it on Economics (+0.7).Reranking starts ahead at low budgets by inheriting the shortlist’s base nDCG@10, then plateaus at its retrieval ceiling; LATTICE’s crossover holds for both Gemini models.Source: §3.2 · p. 10, Figure 4 · p. 10
• Gemini-3.1-Flash-Lite reaches 56.2 nDCG@10 at $0.06/query, while Qwen3.5-27B reaches 56.7 at $0.11/query and Gemini-3-Flash reaches 58.8 at $0.38/query.Across Qwen3.5 sizes, quality rises steeply from 10.7 with 2B to 43.6 with 4B, then saturates near 27B.Source: §3.2 · pp. 10–11, Table 2 · p. 10
• On the three dynamic-corpus BRIGHT subsets, LATTICE trails the best ensembles by 9–13 points because per-query exclusions make its once-built tree summaries stale.LATTICE++ partly closes this gap; search-guided insertion improves over a static tree by +2.3 nDCG@10 on TheoremQA-T.Source: §3.2 · p. 11

3.3 Generalisation Beyond Reasoning-Intensive Retrieval

• On SciFact, SciDocs, and NQ, LATTICE is competitive, including 66.8 nDCG@10 on 2.68M-document NQ, but its extra per-query cost is not justified on these workloads.The paper identifies reasoning-intensive retrieval as the regime where the added compute is worthwhile.Source: §3.3 · p. 11, Table 3 · p. 11

3.4 Analysis and Ablations

• Top-down LLM-guided construction beats bottom-up RAPTOR-style embedding clustering by 2 points on Biology and 12 points on TheoremQA-Theorems.The authors attribute the larger gap to embedding clusters that group mathematically unrelated theorems sharing surface symbols.Source: §3.4 · p. 11, Table 4 · p. 11
• Calibration lifts Biology from 46.9 nDCG@10 without reference nodes to 69.5 at ℓ=10, while under B×N=40, B=2,N=20 reaches 69.5 and B=8,N=5 reaches 64.2.Calibration saturates around ℓ=5–10; depth beats breadth, with the same ordering on Gemini-2.5-Flash.Source: §3.4 · p. 12, Figure 5 · p. 12
• On StackExchange average, path-relevance aggregation contributes +3.4 nDCG@10, explicit reasoning +2.7, and score calibration +2.4 against their respective ablated variants.The ablation uses Gemini-2.5-Flash because Gemini-3-Flash does not expose a way to disable internal reasoning.Source: §3.4 · p. 12

LLMs in Retrieve-then-Rerank Pipelines

• LLM rerankers reorder a cheap retriever’s shortlist, while performance remains bottlenecked by the initial retrieval stage.Alternative generative and long-context approaches respectively face scaling and updating difficulties or computational infeasibility at moderate scale.Source: §4 · pp. 12–13

Hierarchical Indices

• RAPTOR and EHI use semantic hierarchies but still traverse by embedding similarity; LATTICE uses an LLM to read node content and reason at each step.Earlier hierarchical search structures such as HNSW instead use geometric hierarchies and vector dot-product traversal.Source: §4 · p. 13

Adaptive Corpus-Graph Retrieval

• Graph retrieval methods expand from an initial pool over precomputed corpus or ANN graphs, whereas LATTICE uses an LLM-built semantic hierarchy.The compared graph structures are embedding-built proximity graphs optimized for nearest-vector search.Source: §4 · p. 13

LLM-based Clustering

• LLMs have been used for top-down topic discovery and more reliable clustering than embedding k-means; LATTICE extends this work recursively into a hierarchy.Query expansion and agentic IR keep the LLM upstream of retrieval, while Graph-RAG uses LLMs over pre-structured knowledge graphs.Source: §4 · p. 13

5 Conclusion

• LATTICE places the LLM inside retrieval, traversing a semantic hierarchy directly with no embedding model in the search-time loop.The hierarchy scaffolds the search agent’s action space while its reasoning supports traversal.Source: §5 · pp. 13–14
• LATTICE matches the best four-component fine-tuned ensembles on BRIGHT, and LATTICE++ reaches 49.1 nDCG@10.The paper reports that LATTICE++ is a three-source ensemble.Source: §5 · p. 14
• LATTICE is LLM-agnostic, runs on open-weight backbones, and has about four times lower per-query cost.The paper identifies deep research, legal QA, and technical QA as quality-over-latency applications for the current method.Source: §5 · p. 14

A Limitations and Future Work

• The current semantic tree is static; adapting it to a changing corpus would require efficient localized updates to summaries rather than full reconstruction.The authors identify such update methods as future work.Source: A Limitations and Future Work · p. 19
• For extremely large corpora, offline tree construction can be computationally intensive because clustering and summarization repeatedly use LLMs.The authors suggest combining traditional clustering at lower levels with LLM summarization only at the most abstract levels.Source: A Limitations and Future Work · p. 19
• Greedy best-first traversal is effective in a zero-shot setting; the authors suggest exploring reinforcement learning to optimize search policies for retrieval rewards.They frame this as a future research direction, not an evaluated method.Source: A Limitations and Future Work · p. 19

B.1.1 Offline Tree Construction

• Offline construction uses a maximum branching factor of 10–20; bottom-up clustering uses Gecko embeddings and iterative spectral clustering, while top-down construction generates five summary levels.Gemini-2.5-flash summarizes internal nodes and multi-level document summaries.Source: §B.1.1 · p. 20
• For StackExchange passages, initial clusters group passages by source article and subdivide oversized groups by their proximity within that article.This produces more coherent initial clusters than embedding-based grouping alone, according to the passage.Source: §B.1.1 · p. 20

B.1.2 Online Traversal

• Online traversal uses Gemini-2.5-flash, 20 iterations, beam size 2, path-relevance momentum α=0.5, and 10 cross-branch calibration leaves.The default reasoning budget is −1, leaving the model to decide how long to think.Source: §B.1.2 · p. 20
• The latent-score solver minimizes MSE with Adam at learning rate 10^-2 for 100 steps after each batch of slate evaluations.The manuscript reports using LLMs as a collaborative writing assistant; the authors conducted the core ideas, experiments, and analyses.Source: §B.1.2 · p. 20

B.2 Dataset Details

• BRIGHT’s StackExchange subsets contain passages from longer source documents, while some Coding and Theorem-based datasets require excluding query-specific documents from search.The latter setting poses a challenge for static semantic-tree indices.Source: §B.2 · p. 21

B.3.1 Bottom-up

• Bottom-up construction clusters embeddings, summarizes each current layer, then repeats embedding-based clustering until at most M nodes remain.The algorithm can start from optional initial clusters; its root connects to the final layer.Source: §B.3.2 · p. 22

B.3.2 Top-down

• Top-down construction generates five summary levels per document, then clusters unique summaries and selects a level with enough distinct summaries under the LLM token limit.The chosen summary level is the first meeting the heuristic’s clustering-count and context-length conditions.Source: §B.3.2 · pp. 21, 23

B.4 Tree Traversal

• The appendix points to Algorithm 4 for the full pseudocode of the online LATTICE tree search.Source: §B.4 · p. 23

• The ensemble normalizes each system’s per-document scores to [0,1] and combines them with weights 0.6 for LATTICE and 0.2 each for BM25 and BGE-Reasoner-Embed.Source: §B.5 · p. 23
• 46.5 nDCG@10 for LATTICE++ closes a further 3.5-point gap over LATTICE alone (43.0), as Table 7 decomposes the ensemble components.LATTICE averages 43.0 nDCG@10, versus 40.3 for BGE-Reasoner-Embed and 30.5 for BM25 on the reasoning-rewritten query.Source: Table 7 · p. 24, §B.5 · p. 24
• Gains over LATTICE are concentrated on dynamic-corpus and topic-match subsets, while Economics, Robotics, and Sustainable Living show little additional gain from ensembling.The cited dynamic-corpus examples include LeetCode, AoPS, TheoremQA-Q, and TheoremQA-T; Biology and Earth Science illustrate topic-match gains.Source: §B.5 · pp. 24–25

B.6 Hyperparameter Ablations with Gemini-2.5-Flash

• Calibration remains essential and depth beats breadth under a fixed budget with Gemini-2.5-Flash, whose absolute scores are 4–10 points lower than Gemini-3-Flash.The relative ordering of configurations is identical.Source: §B.6 · p. 25, Figure 7 · p. 25

B.7 Cost Per Query Breakdown for Table 2

• Cost per query is computed from average input and output tokens multiplied by provider token rates; Gemini-2.5-Flash costs $0.30/$2.50 per million input/output tokens.Measured Gemini-2.5-Flash usage averages 168K input and 155K output tokens per query.Source: §B.7 · p. 25

B.8 Retrieval Coverage on StackExchange

• 74.8% average Recall@100 on seven StackExchange subsets exceeds BM25 by 9.5 percentage points and ReasonIR-8B by 4.0 percentage points.The largest coverage gains occur on multi-step-reasoning subsets.Source: §B.8 · p. 25, Figure 8 · p. 26

B.9 Search-and-Insert Index Updates

• On TheoremQA-Theorems, search-guided insertion reaches 49.7 nDCG@10 and 81.1 Recall@100, above the static tree’s 47.4 and 73.9, respectively.The authors caution that this is a single-dataset result and do not claim it as a general phenomenon.Source: Table 8 · p. 26, §B.9 · p. 26, §B.8 · p. 26
• The update procedure searches the existing tree using the new document as the query, attaches it as a leaf under the highest-scoring returned leaf cluster, and leaves that cluster’s summary unchanged.Source: §B.9 · p. 26

C.1 Sample scoring response from LLM

• A yaw-rotation query reaches a relevant document deep in the hierarchy, with the callout showing the LLM comparing child topics against the query’s quaternion-related needs.The example describes the user’s quaternion_from_euler task and a child discussing conversions among matrices, quaternions, and Euler angles.Source: §C.1 · pp. 26–27

C.2 Search failure on dynamic corpus

• On a dynamic AoPS corpus, stale summaries based on pruned children mislead traversal after two correct levels and lead to retrieval failure.The figure marks dynamically excluded leaves in red and the ground-truth path in gold.Source: §C.2 · p. 27, Figure 9 · p. 28

D Prompts

• The appendix includes prompts for node scoring, multi-level keyword generation, keyword clustering, and bottom-up group summaries.The clustering prompt groups keywords into [Mmin,Mmax] clusters.Source: Figure 10 · p. 29, Figure 11 · p. 29, Figure 12 · p. 30