PASemiQA: Plan-Assisted Agent for Question Answering on Semi-Structured Data with Text and Relational Information

1 Introduction

• Semi-structured data combines graph relations with node-level documents, supporting flexible information storage across domains such as biomedicine and e-commerce.Biomedical nodes can describe objects, while e-commerce product nodes can include descriptions and reviews alongside co-purchasing links.Source: §1 · pp. 1–2
• Text-retrieval methods can miss complex cross-node reasoning, while existing semi-structured-data baselines may neglect node connections and perform poorly on relational questions.Text-only storage also limits exact step-by-step reasoning, whereas graph edges may not express complex relations spanning multiple entities.Source: §1 · pp. 1–2
• PASemiQA plans which nodes and edges are relevant, then uses an LLM agent to traverse the data and extract information for answering questions.The paper presents the planning module and agent framework as contributions for using relational and text information together.Source: §1 · p. 2
• The paper reports empirical effectiveness across semi-structured datasets from different domains.The stated evaluation spans different semi-structured datasets and domains.Source: §1 · p. 2

2.1 Retrieval-Augmented Generation (RAG)

• RAG commonly retrieves text by embedding similarity; later methods add iterative retrieval, retrieval-model tuning, retrieved-content fine-tuning, or decomposition with chain-of-thought [27] [15] [2] [13].GraphRAG connects text chunks by semantic similarity, while G-retriever uses text-attributed graphs but is described as unsuitable for nodes with large amounts of text; GraphRAG lacks external relational information [9] [7].Source: §2.1 · p. 2

2.2 Knowledge Graph Question Answering (KGQA)

• KGQA includes semantic parsing into executable graph queries and retrieval-augmented approaches that explore or generate relation paths [14] [16] [25].RoG generates relation paths as plans, ToG explores paths interactively, and GoG allows more flexible traversal [14] [16] [25].Source: §2.2 · p. 2
• Existing KGQA methods do not use node text, which can limit answers and make text important when semi-structured-data nodes lack clear unique names.The figure is identified as comparing data structures, but the supplied caption does not specify its individual encodings.Source: §2.2 · p. 2, Figure 1 · p. 3

3 Problem Definition

• Semi-structured data is a text-attributed heterogeneous graph whose nodes map to text descriptions and whose edges map to relation types.The definition names the node set V, edge set E, node-text mapping fT, and edge-type mapping fR.Source: §3 · p. 3
• QA on semi-structured data aims to find answer nodes a in the graph for a given question q.The task is defined over the graph introduced in the preceding definition.Source: §3 · p. 3
• The task requires jointly using relations and node text: standard RAG uses text but lacks multi-step relational reasoning, while KGQA neglects node descriptions.The paper states that these limitations motivate methods that use both information types.Source: §3 · pp. 3–4

4.1 Planning Generation

• Planning generation maps a question to relevant nodes and relation-path sequences, including a text type for questions answerable from node documents alone.Source: §4.1 · p. 4

4.1.1 Text information.

• Node matching first uses keyword matches on node names; if none are found, it selects top-k nodes by question-to-document embedding similarity.Source: §4.1.1 · p. 4

4.1.2 Relational information.

• For each relation path, candidate answers are nodes reachable from the planned nodes, with a uniform conditional answer distribution over those nodes.Source: §4.1.2 · p. 4
• The relation-path generator is instruction-fine-tuned to favor shortest paths linking planned nodes to gold answers, maximizing next-relation likelihood.Source: §4.1.2 · pp. 4–5

4.2 Graph Traversing Agent

• The LLM agent iterates among Search over graph neighbors, Query over node text by embedding similarity, and Finish to return an answer.Source: §4.2 · pp. 5–6

4.3 Complete Algorithm

• Algorithm 1 initializes matched nodes, generates relation paths, repeats thought/action generation for up to T rounds, and outputs the last Finish answer.Source: §4.3 · p. 6

5 Experiments

• Experiments use Amazon, MAG, and PrimeKG, report Hit@1, and compare against RAG baselines and recent KGQA methods using GPT-4 agents.The path generation model is LLaMa2-7B unless otherwise specified.Source: §5 · p. 6

5.1 Comparison on Accuracy and Latency

• PASemiQA achieves the best overall answering performance; RAG baselines do better on text-heavy Amazon questions, while KGQA methods outperform them on relationally complex PrimeKG questions.The comparison uses Hit@1 scores across semi-structured data sets.Source: Table 2 · p. 6, §5.1 · p. 6
• PASemiQA has comparable time cost to VSS with reranker while achieving better performance; VSS and RoG are faster because they do not iteratively select relational information.ToG is also faster on PrimeKG, where its performance is worse than other KGQA methods.Source: Table 3 · p. 6, §5.1 · p. 7

5.2.1 Using Different LLM Models as the Agent.

• GPT-4 generally performs best as the agent, while Qwen2 approaches it on Amazon and LLaMa3 is almost comparable on MAG.GPT-3.5 performs worse than GPT-4 on all three data sets.Source: Table 4 · p. 7, §5.2.1 · p. 7

5.2.2 Implementation of Planning Modules.

• Hybrid node matching achieves the best overall performance across data sets; exact node matching works well on PrimeKG, while Amazon benefits from using node text.Performance across node-matching methods is generally stable on MAG.Source: Table 5 · p. 7, §5.2.2 · p. 7
• Without path generation, performance remains better than other baselines on PrimeKG and comparable on MAG, but falls below the simple VSS baseline on Amazon.The path-generation module matters more on Amazon and PrimeKG than on MAG; the authors attribute PrimeKG's sensitivity to complex biomedical edge types that may be unfamiliar to general LLMs.Source: Table 6 · p. 7, §5.2.2 · p. 8

5.3 Hyper-parameter Sensitivity

• Performance is robust to K and T: values that are too small or too large lead to sub-optimal performance, but differences across values are not significant.K controls the number of related nodes extracted per step; T controls the total number of steps.Source: Figure 3 · p. 7, Figure 4 · p. 7, §5.3 · p. 8

5.4 Case Studies

• In the MAG example, the plan identifies author Th. Friedrich and the author-writes-paper relation, after which the agent uses the paper title to select the answer.Source: Table 7 · p. 8, §5.4 · p. 9

• PASemiQA jointly uses text and relational information to answer questions over semi-structured data.Source: §6 · p. 9
• PASemiQA first generates a plan to identify useful text and relational information for a question.Source: §6 · p. 9
• An LLM agent then follows the plan to traverse the data and extract relevant information; experiments across different domains demonstrate effectiveness.Source: §6 · p. 9

• The agent follows interleaved Thought, Action, and Observation steps to solve a question.Source: A Prompts · p. 10
• Search returns one-hop subgraphs for exact graph nodes, while Query finds the node most related to the question by text-embedding similarity.Source: A Prompts · p. 10
• Finish returns complete answer-node names or unknown when the answer is not known.Source: A Prompts · p. 10

• Amazon is a product-recommendation data set with single-hop relations among brands, categories, and products in complementary or substitute relationships.Source: B Experiment Details · p. 10
• MAG centers on paper nodes in an academic network, with citation and authorship relations and queries that combine relational paths with textual properties such as topic and methodology.Source: B Experiment Details · p. 10
• PrimeKG is a biomedical data set covering diseases, drugs, genes, and proteins.Source: B Experiment Details · p. 10

C More Experiment Results

• PASemiQA outperforms other KGQA baselines on macro F1, with Table 9 presenting the method comparison.Macro F1 is not applicable to the RAG baselines because they return ranked node lists without specifying final answers.Source: C More Experiment Results · p. 10, Table 9 · p. 10
• PASemiQA generally achieves the best MRR across the semi-structured datasets, as reported in Table 10.Source: C More Experiment Results · p. 10, Table 10 · p. 10
• Because KGQA methods return only several likely answer nodes, arbitrary Hit@k or Recall@k comparisons may be unfair.Source: C More Experiment Results · p. 10