PaperlayerView exact revision on arXiv

Source-linked AI summary

From Chunks to Functional Evidence: Function-Aware Retrieval for EDA Documentation QA

Xiaotian Qiu, Kairui Liu, Shi Chenyi, Jinyuan Deng, Qi Sun, Cheng Zhuo

arXiv:2610.09361v1cs.IRcs.AI

TL;DR

EDA documentation QA is difficult because required evidence is distributed across heterogeneous, functionally coupled artifacts, and retrieval units can fail to preserve those roles. The paper builds typed, provenance-linked functional units and combines query–unit alignment with unit and chunk retrieval. It reports improved EDADocEval-QA ROUGE-L over Chunk RAG and transfer to ORD-MMBench, within the evaluated EDA settings.

  • Problem

    EDA documentation scatters evidence needed for an answer across heterogeneous, functionally coupled artifacts, while chunk-based retrieval can separate evidence that belongs together.

    Source: §1 · p. 1

  • Method

    The framework groups typed artifacts into source-linked functional units, aligns queries with units, and combines unit retrieval with direct chunk retrieval before reranking source evidence.

    Source: §1 · p. 2

  • Results

    EDADocEval-QA ROUGE-L improves 37% over Chunk RAG, and the framework transfers to ORD-MMBench.

    Source: §5 · p. 7

  • Takeaways & Limitations

    The results support function-aware evidence organization in the evaluated EDA documentation settings.

    Source: §5 · p. 7

    The study is limited to technical EDA documentation and does not measure the semantic accuracy of intermediate extraction.

    Source: Discussion · p. 7

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) is widely used to ground answers in documents. For complex technical documentation, however, the primary bottleneck is often not model reasoning but a mismatch between a query and the way knowledge is organized for retrieval. This mismatch is pronounced in Electronic Design Automation (EDA) documentation, where the information needed for an answer is scattered across heterogeneous yet tightly coupled artifacts. We therefore redesign the basic retrieval unit of RAG. Instead of operating on isolated chunks or binary relations, we collect typed artifacts into EDA functional units. Each unit is recorded as a hyperedge with links to its source chunks. We then train an encoder to align queries with functional units and combine unit retrieval with direct chunk retrieval. After mapping the selected units back to their sources, a unified reranker chooses the evidence given to the generator. On the newly constructed EDADocEval-QA dataset, our method improves ROUGE-L by 37.1% over Chunk RAG and 55.6% over the strongest graph baseline. On the public ORD-MMBench benchmark, it improves ROUGE-L by 30.0% over the strongest baseline. These results support function-aware evidence organization in the evaluated EDA documentation settings.

1 Introduction

EDA documentation scatters functionally coupled evidence across heterogeneous artifacts, while conventional retrieval units can separate or flatten those relationships. The paper organizes typed artifacts into provenance-preserving functional units and combines unit and chunk retrieval with query–unit alignment.

  • EDA answers may require artifacts from different sections, yet conventional chunking separates functionally coupled evidence and binary relations can flatten it.Source: §1 · pp. 1–2
  • The framework groups source-localized artifact occurrences around functional anchors, preserving their types and provenance in hyperedges.These units support conservative connections across documents.Source: §1 · p. 2
  • Query–unit contrastive alignment distinguishes semantically similar units by whether their available artifact types can support the query.Source: §1 · p. 2
  • The hybrid retriever combines functional-unit and direct-chunk retrieval to return a unified evidence context to the generator.Source: §1 · p. 2
  • EDADocEval-QA yields 0.473 ROUGE-L and 93.6 LLM-Score; ORD-MMBench provides complementary generation and document-retrieval evaluation.Source: §1 · p. 2

2 Related Work

Prior EDA RAG systems face challenges from specialized documentation and from evidence spread across tightly coupled artifact types. The related work contrasts chunk- and graph-based approaches with higher-order evidence organization, and positions this framework as complementary to reasoning-enhanced methods.

LLMs and RAG in EDA

  • EDA documentation challenges general LLMs with proprietary protocols, complex parameter constraints, and specialized terminology; most EDA RAG still uses independent chunks or generic relations.Source: LLMs and RAG in EDA · p. 2

Evidence Organization in RAG

  • Chunk retrieval can separate cross-entity dependencies, while graph retrieval may split higher-order usage events or omit source context from procedural and code evidence.Source: Evidence Organization in RAG · p. 2

Structured Evidence Organization

  • Related systems organize evidence hierarchically or preserve technical-document relations; this framework instead groups source-localized EDA occurrences by tool function and artifact roles.Source: Structured Evidence Organization · p. 2

Reasoning-Augmented RAG

  • Reasoning-enhanced methods depend on retrieved evidence; the framework complements them by encoding typed documentation dependencies with intact source context.Figure 2 depicts unit construction, hybrid retrieval and reranking, and query–unit contrastive training.Source: Reasoning-Augmented RAG · p. 2, Figure 2 · p. 3

3 Method

The method organizes typed EDA artifacts into functional units, aligns queries with those units, and combines unit retrieval with direct chunk retrieval. Selected units are mapped back to source chunks, which are reranked for the generation context.

Problem Formulation and Overview

  • The framework organizes EDA functional units, aligns queries to them, searches unit and chunk indexes in parallel, then reranks source chunks for generation.The task is framed as selecting a source-chunk context that covers the facts and artifact types needed to answer a query.Source: Problem Formulation and Overview · p. 3

EDA Functional Unit Organization

  • Construction parses document structure, extracts typed artifacts with deterministic rules or an LLM, validates provenance, and reprocesses only changed documents during updates.Structural validation checks unit contents, source references, type labels, and type diversity; it does not establish semantic accuracy of LLM extraction.Source: EDA Functional Unit Organization · pp. 3–4
  • A functional occurrence groups typed artifacts around a command or file anchor within one source region while retaining its source chunks and document-line provenance.Artifact type denotes evidence form, whereas functional role describes an artifact's participation in a usage event; training uses observable types, not latent role names.Source: EDA Functional Unit Organization · pp. 3–4
  • Cross-document units require at least two occurrences from different documents, at least two artifact types, and an admissible canonical anchor.The linker filters generic placeholders and language type aliases and caps occurrences per anchor to limit irrelevant hubs.Source: EDA Functional Unit Organization · p. 4
  • The define_pdn_grid example links its default parameter, syntax, and strap-orientation constraints as a table, code block, and text paragraph, preserving all three source chunks.The canonical command anchor connects the artifacts while their source IDs remain intact.Source: EDA Functional Unit Organization · p. 4

Query–Unit Contrastive Alignment

  • Contrastive alignment favors units with query-required artifact types over semantically plausible negatives that lack a critical type, without explicitly predicting a type set at inference.The training queries share their target type combination with reference units by construction, and the objective combines contrastive pairs with artifact-coverage ranking.Source: Query–Unit Contrastive Alignment · pp. 4–5

Function-Aware Evidence Retrieval

  • Parallel unit and chunk indexes provide complementary evidence: selected units expand only to their own source chunks, while direct retrieval can recover precise local details.The unit path locates a function; the chunk path can recover a command option, table row, or warning that a broad unit summary may underrepresent.Source: Function-Aware Evidence Retrieval · p. 5
  • A unified reranker fuses function-aware and base-semantic scores to select source chunks for generation, so functional units organize selection but are not themselves the generator's evidence format.Both retrieval paths terminate at source chunks, and the described fusion sets α = 0.4.Source: Function-Aware Evidence Retrieval · p. 5

4 Experiments

Across two EDA documentation QA settings, the method leads the reported generation results; ablations, construction tests, and retrieval measurements characterize its components and costs.

Experimental Setup

  • The evaluation uses EDADocEval-QA, with 100 QA pairs across four categories and 33 OpenROAD documents, plus public ORD-MMBench text and screenshot queries.The categories cover parameter lookup, syntax retrieval, procedure synthesis, and debug and constraint questions.Source: Experimental Setup · p. 5

Construction and Retrieval Cost

  • 16.6% fewer generation-context tokens (1,697→1,415) came with retrieval latency rising from 20.69 to 27.67 ms.Indexing processed 404 chunks and 3,650 typed nodes; incremental updates reused 402 units and matched all seven outputs from a full rebuild, but extraction accuracy was not manually labeled.Source: Construction and Retrieval Cost · p. 6

Discussion

  • The evaluation measures end-to-end performance, not the isolated effect of hypergraph structure, and does not establish generality beyond technical EDA documentation.Hybrid retrieval reduces generator context but adds latency; the paper reports neither multi-seed averages nor statistical significance.Source: Discussion · pp. 6–7

Main Results

  • 0.473 ROUGE-L, 0.189 BLEU, and 93.6 LLM-Score place the method first on EDADocEval-QA and 0.128 ROUGE-L above Chunk RAG.ReAct reaches 0.374 ROUGE-L; LightRAG's 92.8 LLM-Score coexists with 0.219 ROUGE-L and 0.024 BLEU, illustrating metric divergence.Source: Main Results · p. 6, Table 1 · p. 7

Ablation Study

  • 0.081 ROUGE-L (0.345→0.426) is the largest ablation gain; hybrid retrieval adds 0.035 to 0.461, and alignment adds 0.012 to 0.473.The authors report no confidence interval and do not interpret the final increment as a statistically established standalone gain.Source: Ablation Study · p. 6, Table 2 · p. 7

ORD-MMBench Results

  • 0.095 ROUGE-L gain (0.317→0.412) over VISTA-RAG accompanies +0.060 BLEU (0.180→0.240) and +10.0 LLM-Score (69.6→79.6).Document Recall@1/5 is 0.700/0.917 versus MM-GRADE's 0.679/0.896; the retrieval advantage ranges from 0.015 to 0.022 across cutoffs.Source: ORD-MMBench Results · p. 6, Table 3 · p. 7, Table 4 · p. 7

5 Conclusion

The framework organizes typed EDA artifacts into functional units and uses query–unit alignment with hybrid retrieval to select evidence. It improves EDADocEval-QA ROUGE-L over Chunk RAG and transfers to ORD-MMBench, within a study limited to two EDA benchmarks.

5 Conclusion

  • 37% higher EDADocEval-QA ROUGE-L over Chunk RAG is reported for the framework.The study also reports transfer to ORD-MMBench.Source: §5 · p. 7
  • The framework combines typed EDA functional units, query–unit alignment, and hybrid retrieval to select evidence from units and source chunks.Source: §5 · p. 7
  • Evaluation is limited to one domain-specific dataset and one public EDA benchmark; construction requires offline LLM extraction, and hybrid retrieval trades reduced generation context for latency.The authors identify extraction accuracy and additional EDA tool ecosystems as future evaluation needs.Source: §5 · p. 7

A Indexing and Retrieval Procedure

The procedure first indexes typed artifacts and their source chunks into functional units, then retrieves candidates through both unit and direct-chunk paths. It reranks source chunks using a fused similarity score to form the generation context.

A Indexing and Retrieval Procedure

Incremental Updates

The update test starts from a 32-document index and adds a document, recomputing only the addition and affected anchors. Its table compares records and single-run times across the 32-to-33-document update.

Incremental Updates

  • Starting from 32 documents, the update inserted 27Flute3.md and recomputed only the added document and affected anchors.Source: Incremental Updates · p. 10
  • The 32-to-33-document comparison covers seven matched index-record files, including hyperedges, nodes, provenance, and chunk content.Source: Table A1 · p. 10
  • The table reports single-run times for the seven matched record files.Source: Table A1 · p. 10

Markdown Normalization Invariance

The invariance test indexes four equivalent Markdown variants. All preserve the semantic graph, although repeated blank lines change the serialized-output hash.

Markdown Normalization Invariance

C Qualitative Example

The qualitative example shows that the method retrieves a more specific explanation of sink clustering in clock-tree synthesis: it recovers the pre-H-tree subtree and buffer-endpoint relationship.

C Qualitative Example

  • The method’s answer specifies that pre-clustering creates one subtree level before the H-tree, with each cluster buffer becoming an H-tree endpoint.Source: C Qualitative Example · p. 10
  • Chunk RAG describes grouping sinks into clusters, while the method’s answer also identifies the subtree and buffer-endpoint structure.Source: C Qualitative Example · p. 10
  • One functional-unit-derived chunk and four directly retrieved chunks recover the pre-H-tree subtree and buffer-endpoint relation.Source: C Qualitative Example · p. 10
Loading 2610.09361v1…