PaperlayerView exact revision on arXiv

Source-linked AI summary

PrivGemo: Privacy-Preserving Dual-Tower Graph Retrieval for Empowering LLM Reasoning with Memory Augmentation

Xingyu Tan, Xiaoyang Wang, Qing Liu, Xiwei Xu, Xin Yuan, Liming Zhu, Wenjie Zhang

arXiv:2601.08739v1cs.CL

TL;DR

Private KGs create leakage risks when remote LLMs receive graph evidence or exploration traces, while name masking alone leaves structural and reasoning limitations. PrivGemo keeps raw knowledge local, exposes an anonymized and structurally sanitized view, and coordinates remote reasoning with local verification and memory. Across six benchmarks, it reports state-of-the-art performance, including gains of up to 17.1% and smaller-model reasoning comparable to GPT-4-Turbo.

  • Problem

    Private KG-based RAG must provide grounded answers while restricting semantic and structural information observable to remote models.

  • Method

    PrivGemo uses a dual-tower architecture with anonymized long-hop retrieval, local grounding and verification, hierarchical control, and privacy-aware experience memory.

  • Results

    PrivGemo achieves state-of-the-art results across tested benchmarks, surpasses a strong baseline by up to 17.1%, and enables smaller models to approach GPT-4-Turbo reasoning performance.

  • Takeaways & Limitations

    The framework supports KG-grounded reasoning while controlling semantic and structural exposure and reducing unnecessary exploration and remote interactions.

  • Takeaways & Limitations

    PrivGemo considers only KG-based evidence and does not incorporate complementary modalities such as images, audio, or video.

Abstract

from arXiv · show

Knowledge graphs (KGs) provide structured evidence that can ground large language model (LLM) reasoning for knowledge-intensive question answering. However, many practical KGs are private, and sending retrieved triples or exploration traces to closed-source LLM APIs introduces leakage risk. Existing privacy treatments focus on masking entity names, but they still face four limitations: structural leakage under semantic masking, uncontrollable remote interaction, fragile multi-hop and multi-entity reasoning, and limited experience reuse for stability and efficiency. To address these issues, we propose PrivGemo, a privacy-preserving retrieval-augmented framework for KG-grounded reasoning with memory-guided exposure control. PrivGemo uses a dual-tower design to keep raw KG knowledge local while enabling remote reasoning over an anonymized view that goes beyond name masking to limit both semantic and structural exposure. PrivGemo supports multi-hop, multi-entity reasoning by retrieving anonymized long-hop paths that connect all topic entities, while keeping grounding and verification on the local KG. A hierarchical controller and a privacy-aware experience memory further reduce unnecessary exploration and remote interactions. Comprehensive experiments on six benchmarks show that PrivGemo achieves overall state-of-the-art results, outperforming the strongest baseline by up to 17.1%. Furthermore, PrivGemo enables smaller models (e.g., Qwen3-4B) to achieve reasoning performance comparable to that of GPT-4-Turbo.

1 Introduction

Private knowledge graphs can ground LLM reasoning, but exposing their content or exploration traces to remote models creates leakage risks. PrivGemo addresses these risks with anonymized dual-tower retrieval, long-hop multi-entity reasoning, hierarchical control, and privacy-aware memory.

  • Motivation: Private KGs provide structured evidence for grounded multi-hop reasoning, but sending their content to closed-source LLMs risks leakage.The risk is especially relevant in sensitive domains such as enterprise, healthcare, and internal security graphs.
  • Limitations of existing methods: Name masking alone remains vulnerable to structural leakage and does not fully address remote interaction, multi-hop reasoning, or experience reuse.Existing retrieve-and-select approaches can be effective for shallow queries but face four stated limitations.
  • PrivGemo: PrivGemo keeps raw KG evidence local while exposing a structurally sanitized anonymized view for remote analysis and selection.The framework combines session-specific anonymization with structural de-uniqueness to reduce semantic and structural exposure.
  • PrivGemo: PrivGemo retrieves long-hop paths covering all topic entities, avoiding greedy chains and inconsistent evidence merging.The method explicitly requires joint coverage of topic entities within one verified reasoning path.
  • PrivGemo: A hierarchical controller and privacy-aware experience memory reduce unnecessary exploration and remote calls while supporting continual adaptation.The memory stores successful reasoning records and the controller manages mode and depth transitions.
  • Results: PrivGemo achieves state-of-the-art results across tested datasets, surpasses a strong baseline by up to 17.1%, and enables smaller models to approach GPT-4-Turbo reasoning performance.These results are reported as core contributions of the framework.

2 Related Work

Related work frames KG-based RAG as inference-time graph reasoning and connects hierarchical controllers with retrievable memory for guiding exploration and reusing verified patterns.

  • KG-based RAG: KG-based RAG systems use LLMs as inference-time reasoners over retrieved graph evidence, including hop-by-hop walks and repeated read–reason cycles.ToG prompts the LLM to choose neighbors, while StructGPT organizes repeated retrieval and reasoning.
  • LLM memory: Long-term memory approaches retrieve prior interaction histories or selectively control access through controllers.These methods motivate combining retrieval with controlled exploration.
  • Research direction: PrivGemo builds on this literature with a hierarchical controller and retrievable experience pool for steering exploration and reusing verified reasoning patterns.The related-work discussion identifies this combination as the motivating design direction.

3 Preliminaries

The preliminaries formalize KGs, reasoning paths, entity paths, and privacy-preserving KGQA. Privacy is defined through restricting both semantic and structural information observable to remote models.

  • Knowledge graphs: A knowledge graph G(E, R) represents factual knowledge as triples connecting entities through relations.Entities belong to E and relations belong to R.
  • Reasoning paths: A reasoning path is a connected sequence of knowledge triples, with its length equal to the number of linked triples.The path connects a starting entity to an ending entity through successive relations.
  • Entity paths: An entity path connects an ordered list of entities through the reasoning paths between consecutive entities.It combines the component paths into one connected sequence of triples.
  • Privacy-preserving KGQA: Privacy-preserving KGQA requires answers supported by KG evidence while restricting information observable to a remote model.The setting distinguishes semantic exposure from raw identifiers and structural exposure from connectivity patterns and exploration traces.
  • Privacy constraints: PrivGemo keeps topic entities locally available, prohibits transmitting raw KG names and neighborhoods, and treats exploration and remote interactions as exposure events.A session-specific mapping creates an anonymized remote view while grounding and verification remain local.

4 Method

PrivGemo constructs a question-specific anonymized subgraph, separates remote analysis from local control and verification, and uses guided path exploration, pruning, and memory to limit exposure while supporting complex reasoning.

  • Initialization: PrivGemo identifies topic entities and builds a bounded question-centric subgraph around their local neighborhoods.A local LLM and dense retrieval align question entities with KG entities before bounded expansion.
  • Privacy-preserving KG construction: The framework sanitizes semantic and structural information by anonymizing identifiers, coarsening attributes, and reducing distinctive topology.Relations may retain original labels in utility mode or use clustered schema labels in privacy mode.
  • Dual-LLM reasoning: A remote Brain processes only privacy-preserving inputs, while a local Hand accesses raw evidence, controls exploration, and performs grounding.This separation makes the trust boundary explicit and keeps raw information local.
  • Memory-gated usage: Memory can gate Brain calls by reusing stored analysis artifacts, while otherwise the Brain receives anonymized questions, topic entities, and candidate paths.The local Hand can also produce analysis on-device when the remote call is skipped.
  • Hierarchical control: The controller traverses a reasoning tree using mode and depth budgets, prunes anonymized candidates, verifies selected evidence locally, and stops when evidence is sufficient.Failed verification updates the exploration state through the next-step policy, while repeated failures can prune branches.
  • Evidence retrieval and pruning: Evidence exploration combines topic-path, follow-up refinement, and prediction-driven phases before pruning and verification.Follow-up questions target missing evidence, while prediction-driven exploration uses path-level understanding and anonymized experience.
  • Experience memory: The experience memory stores verified decision traces that reduce repeated exposure and avoid unproductive exploration.It is used as reusable guidance rather than as a cache of answers.

5 Experiment

PrivGemo achieves strong KGQA performance across benchmarks, including under privacy constraints and with weaker reasoning models. Its accuracy remains relatively stable as anonymization increases, while ablations examine backbone capability and privacy–utility trade-offs.

  • Main Results: PrivGemo achieves SOTA performance across nearly all evaluated datasets, outperforming previous SOTA methods by significant margins.With GPT-3.5-Turbo, it improves over ToG by 12.4% on average and up to 30.2% on Simple Questions.
  • Main Results: PrivGemo with GPT-4o-mini surpasses the GPT-4-based ToG baseline on 5 of 6 datasets, with a 12.9% margin on Simple Questions.The result indicates that the framework can bring smaller models close to GPT-4-level reasoning performance.
  • Main Results: Under privacy constraints, PrivGemo with GPT-4o-mini outperforms ARoG across all reported datasets, improving WebQSP by up to 8.6%.Against LLM-only approaches, PrivGemo yields average improvements exceeding 40%.
  • Ablation Study: With Qwen3-4B as the Hand model, PrivGemo achieves a 192% improvement over UIO on GrailQA.With GPT-4-Turbo as the Brain and Qwen3-32B as the Hand, it reaches 88.0% on GrailQA, a 69.2% gain over UIO at 52.0%.
  • Ablation Study: CWQ accuracy decreases from 67% with plaintext to 59% with full anonymization, while WebQSP remains at 75% under the strictest privacy setting.GrailQA remains around 80% across anonymization ratios and peaks at 30% anonymization.

6 Conclusion

PrivGemo is a privacy-protected KG-based RAG framework that combines local raw-knowledge access with anonymized remote reasoning. It retrieves connected long-hop evidence and uses hierarchical control and memory to reduce unnecessary exposure and interaction.

  • Conclusion: PrivGemo keeps raw KG knowledge local while enabling remote reasoning over an anonymized view.It retrieves long-hop paths connecting all topic entities and uses hierarchical control with privacy-aware memory to reduce unnecessary exploration and remote interactions.

7 Limitation

The paper’s stated limitation is that PrivGemo considers only KG-based evidence. External modalities that may provide complementary factual signals are not incorporated.

  • Limitation: PrivGemo does not incorporate external modalities such as images, audio, or videos.The paper identifies multimodal retrieval and privacy-preserving grounding as future work.

A.4 Privacy-Aware Memory

PrivGemo combines privacy-aware KG initialization, memory-gated dual-LLM reasoning, and local verification to control exposure while reusing prior reasoning experience. Its pipeline anonymizes and sanitizes question-specific graph views, explores evidence hierarchically, and updates memory only after verified answers.

  • Privacy-aware initialization: Privacy-aware initialization grounds the question, builds a temporary entity map, anonymizes the KG view, sanitizes its structure, deletes the secret, and retains the raw subgraph locally.The construction includes semantic anonymization followed by structure sanitization and context minimization.
  • Hierarchical reasoning: Dual-LLM hierarchical reasoning builds a question tree, verifies nodes against local and anonymized views, attaches evidence, and synthesizes an answer only after a sufficiency check.The process supports top-down traversal, pruning, forward exploration, backtracking, and final evidence verification.
  • Experience update: Successful verified reasoning writes anonymized indicators, predicted depth, trajectories, templates, and outcomes back into encrypted, continuously maintained experience memory.The memory pool merges templates and prunes low-value records after successful verification.
  • Evidence exploration: Evidence exploration uses topic-path, refinement, and prediction-driven phases with bounded tree search and experience-guided pruning.Candidate paths are ranked using direct relevance and similarity to retrieved memory before optional Brain selection.
  • Memory-gated analysis: The memory-gated controller retrieves prior experiences and either reuses analysis artifacts locally or invokes the remote Brain model for anonymized question analysis.Retrieved experiences provide control hints such as whether to skip the Brain, initialization policy, and warnings.

C.1 Effectiveness Evaluation

PrivGemo remains effective as questions become more multi-entity and multi-hop, while structural sanitization substantially reduces exposed graph size. The evaluation reports strong performance on complex queries and near-70% reductions for large retrieved subgraphs.

  • Multi-entity reasoning: PrivGemo performs strongly on multi-entity questions, exceeding single-entity performance on CWQ, GrailQA, and WebQuestions while reaching 84.5% on WebQSP.The authors attribute this pattern to structure-based retrieval using multiple entities to narrow reasoning paths and reduce ambiguity.
  • Multi-hop reasoning: WebQSP reaches 100% accuracy at six relations and 71.43% at 8+ hops, while CWQ reaches 100% at two relations and 92.9% at six.The method also answers some 8+ hop questions through novel paths and synergy with LLM parametric knowledge.
  • Structural sanitization: Nearly 70% of nodes are removed from large CWQ and QALD10-en subgraphs, compressing over 2.4 million initial nodes to approximately 732K and 552K, respectively.Structure pruning and clustering remove redundant motifs and weakly connected entities while preserving core reasoning context.

C.2 Reasoning Faithfulness Analysis

PrivGemo primarily grounds correct answers in KG paths while using LLM knowledge selectively when graph evidence is incomplete. Its path overlap varies by dataset, reflecting precise matching on structured queries and exploratory reasoning on complex multi-hop questions.

  • Evidence sources: KG-only reasoning dominates across datasets, reaching 81.8% of correct answers on QALD10-en and 71.8% on GrailQA.
  • Evidence sources: WebQSP uses KG-inspired LLM reasoning for 20.75% of answers, while CWQ uses LLM-inspired KG reasoning for 20.41%.
  • Path overlap: Over 71% of WebQSP samples achieve perfect path overlap, indicating high structural fidelity to the ground-truth paths.
  • Path overlap: CWQ concentrates in the (0.25, 0.5] overlap interval, indicating more exploratory path construction for complex multi-hop questions.
  • Error analysis: CWQ errors are categorized as reasoning, KG/path retrieval, format, or hallucination/other failures.

C.4.1 LLM calls cost analysis

PrivGemo distributes work between an efficient remote Brain and a more interaction-heavy local Hand. Most questions are resolved within six calls, while complex CWQ questions produce a longer Hand-call tail.

  • LLM call distribution: The Brain resolves 77.67% of GrailQA, 57.50% of WebQSP, and 52.67% of CWQ questions within 0–3 calls.
  • LLM call distribution: The Hand shifts toward 3–6 calls, covering 40.50% of CWQ and 38.17% of WebQSP questions.
  • LLM call distribution: Only 5.00% of CWQ queries require 9–12 Hand calls, reflecting a longer interaction tail on the complex dataset.
  • LLM call distribution: The majority of questions across all benchmarks are resolved within six calls.

C.4.2 Cost-Performance analysis

PrivGemo shifts retrieval-heavy processing to a local Hand model and reserves cloud Brain calls for high-value reasoning. This reduces API usage while keeping private retrieval operations local.

  • Strategic offloading: The local Hand processes heavy retrieval workloads, including 10,710.8 input tokens on CWQ, without marginal API costs.
  • Cloud interaction cost: PrivGemo requires 3.5 cloud Brain calls on CWQ, compared with 21.7 calls for ToG.
  • Cost-performance trade-off: The dual-LLM architecture reduces cloud interaction while preserving privacy for retrieval-heavy operations.

D Experiment Details

PrivGemo is evaluated on six KGQA benchmarks under a privacy-protected dual-model setting. Its experiments cover accuracy, efficiency, interpretability, anonymized reasoning, entity extraction, and experience-gated control.

  • Datasets and baselines: The evaluation uses six benchmarks spanning multi-hop KGQA, single-hop KBQA, and open-domain QA over the full Freebase graph.
  • Evaluation protocol: Exact-match accuracy, reported as Hits@1, is the evaluation metric.
  • Privacy setting: The remote model never sees raw entity names, relation labels, or neighborhoods; local Hand processing performs grounding, verification, and answer generation.
  • Models: Experiments vary the Brain across GPT-3.5-Turbo, GPT-4o-mini, and DeepSeek-V3 while fixing the main Hand model as Qwen3-32B.
  • Models: Alternative instantiations include Qwen3-80B, Qwen3-32B, Qwen3-8B, and Qwen3-4B to test plug-and-play adaptability.
  • Interpretability: Case studies show interpretable multi-entity and multi-hop reasoning while keeping raw identities and neighborhoods local.
  • Pipeline components: The pipeline includes entity extraction, Brain-or-Hand analysis delegation, anonymized planning, local analysis, and experience-gated exploration.
Loading 2601.08739v1…