Semantic-TVM: Structure-Preserving Trustworthy Virtual Memory for Memory-Augmented and Tool-Using Agents

1. INTRODUCTION

• Remote requests may expose private values from retrieved memory and intermediate state, while one-way masking can remove exact identifiers needed by tools and later observations can reintroduce them.Protecting only the initial prompt does not cover the full execution cycle.Source: §1 · p. 1
• TVM projects retrieved memory into an online-safe view, stores exact bindings locally, resolves handles before tool execution, and reprojects observations before the next request.Permitted capsule metadata can remain in the projected view.Source: §1 · p. 1
• Rule-TVM replaces whole fields, whereas Semantic-TVM replaces only sensitive spans predicted by a trusted local model and preserves the remaining content within its field or schema boundary.The two methods share the lifecycle, resolution, and reprojection path but use different projection granularities.Source: §1 · p. 1
• Across Memory-EHR and Memory-RAP, both projection granularities reduce exposure, while Semantic-TVM recovers utility lost under whole-field replacement.The study covers exposure, execution, and task utility across two remote providers.Source: §1 · p. 1
• TVM contributes a local–remote virtual-memory lifecycle, two projection granularities with local recovery, and an evaluation spanning exposure, execution, task utility, component effects, and robustness.Fig. 1 depicts the lifecycle with local recovery and observation reprojection.Source: §1 · p. 1, Figure 1 · p. 2

2.1. TVM Abstraction and Trust Boundary

• The online model receives projected memory, not recovery state; TVM resolves tool actions locally and reprojects observations before subsequent requests.The resulting serialized outbound messages are captured for audit.Source: §2.1 · p. 2
• TVM defines protected values through a trial-specific manifest and measures detectable exposure as exact or normalized matches in serialized outbound requests.The manifest covers protected values from retrieved records, historical actions, and values reappearing in tool results.Source: §2.1 · p. 2

2.2. Rule-TVM: Whole-Field Projection

• Rule-TVM replaces protected fields with handles stored in trial-local state; the resolver validates tool, argument path, expected type, and scope before substituting exact values locally.Resolved actions and raw observations do not enter online history.Source: §2.2 · p. 2

2.3. Semantic-TVM: Span-Level Projection

• Semantic-TVM uses a trusted local model on each memory item, task, and scene adapter to replace predicted sensitive spans while retaining surrounding structure and local recovery state.Spans needed for exact execution receive scoped slots; other protected spans receive scoped placeholders.Source: §2.3 · pp. 2–3

3.1. Experimental Setup

• Memory-EHR uses four retrieved examples in a local query-program workflow; Memory-RAP uses three examples to construct search actions executed by a local tool.The comparison includes No Defense, one-way Masking, Rule-TVM, and Semantic-TVM; Execution Success measures successful execution, while Task Success measures correctness of the final outcome.Source: §3.1 · p. 3

3.2. Experimental Results

• Rule-TVM reaches at most 0.67% Leak Rate, while Semantic-TVM stays at most 1.00% on EHR and 12.00% on RAP; both remain above 99% RAP Execution Success.No Defense has protected-value matches in 78.50% of EHR trials and every RAP trial; Rule-TVM minimizes detected exposure while Semantic-TVM preserves more EHR executability.Source: §3.2 · p. 3, Table 1 · p. 4
• With fine-grained projection fixed, Recovery primarily improves EHR task utility, while the scene Adapter supplies privacy context for RAP exposure reduction.Source: §3.2 · p. 3, Table 2 · p. 4
• Semantic-TVM keeps EHR Leak Rate at most 1.00% with Execution Success above 87% and Task Success above 76%; on RAP it lowers Leak Rate to at most 12.00% with Execution Success above 99%.Source: §3.2 · p. 3
• 84.17% vs. 52.33% Task Success on DeepSeek and 76.83% vs. 6.17% on MiniMax show Semantic-TVM recovering EHR utility over Rule-TVM.The comparison is shown in Fig. 2; Leak Rate rises from 0.33% to 1.00% on DeepSeek and from 0.00% to 0.50% on MiniMax, versus 78.50% under No Defense.Source: §3.2 · p. 3, Figure 2 · p. 4

• TVM separates related approaches by transformed artifact and remote-visible granularity.Source: §4 · p. 4
• Agent memories, tool observations, and generated programs can be exposed to inference [2], while EHR execution may require record-specific values [3].Extraction, poisoning, and prompt-injection risks [7] motivate a trusted boundary; TVM applies the principle that authorization to use data need not imply authorization to release it.Source: §4 · p. 4
• MINIM, SlotGuard [13], and PlanTwin [14] mediate UI observations, transcript bindings, and private environments; Semantic-TVM projects sensitive spans in retrieved memory while preserving local recovery.Source: §4 · p. 4

• Exposure measurement counts detectable exact or normalized matches to manifest-listed values in completely captured provider-bound requests, excluding tool providers, local state, and the memory store.Source: §5 · p. 4
• The measure does not establish semantic privacy or prevent inference from retained context, and handles reveal per-type cardinality and repeated-record positions.Adversarial retrieved content can steer projection, reprojection may miss unanticipated structures, and recovery or schema repair can reintroduce original content.Source: §5 · p. 4
• Evaluation covers two providers and two pipelines with different utility definitions, but does not measure projection fidelity, local latency, throughput, or hardware cost.The authors recommend conservative projection with typed recovery bindings for high-assurance deployments.Source: §5 · p. 4

• 84.17% vs. 52.33% Task Success on DeepSeek shows Semantic-TVM recovering much of the EHR utility lost under Rule-TVM.Source: §6 · p. 4
• TVM applies one recovery-coupled projection operator at whole-field or sensitive-span granularity while keeping exact-value state local.Source: §6 · p. 4
• Granularity should be chosen per pipeline and coupled to validated local recovery: trusted recovery drives EHR utility, while scene-conditioned projection drives RAP exposure reduction.Source: §6 · p. 4