PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Extraction of Records of Processing Activities (RoPA) Using Hybrid RAG and Locally Deployed Large Language Models

To Duy Hinh, Nguyen Le Quoc Anh, Phan Van Tri, Khuong Nguyen-An

arXiv:2609.27359v1cs.CLcs.CRcs.IR

TL;DR

Vietnamese organizations must maintain RoPA, but manual preparation is labor-intensive and cloud LLM APIs may conflict with data-sovereignty requirements. RoPA Manager combines hybrid retrieval, locally deployable LLMs, and a Vietnamese benchmark; local Qwen showed no statistically significant coverage difference from cloud DeepSeek in the paired comparison.

  • Problem

    Manual RoPA preparation requires legal expertise and staff time, while cloud LLM APIs processing PII may conflict with data sovereignty.

    Source: §I · p. 1

  • Method

    RoPA Manager combines lexical and dense retrieval with RRF and LLM extraction, and introduces a Vietnamese benchmark with 32 organizations, 77 activities, 12 field groups, and 4,338 reference values.

    Source: §I · p. 1

  • Results

    The DeepSeek–Qwen token-coverage difference was +0.20 percentage points (95% CI [-0.93, +1.32], p = 0.72), with no significant difference found across 32 paired scenarios.

    Source: §V · p. 5

  • Takeaways & Limitations

    The reported findings support technical feasibility and coverage, but do not measure value-level precision on real model output.

    Source: §V · p. 6

    All data are synthetic, value-level precision is unmeasured, and the study’s conclusions are limited to technical feasibility and coverage.

    Source: §VI · p. 6, §V · p. 6

Abstract

from arXiv · show

Vietnam's Personal Data Protection Law (Law No. 91/2025/QH15) and Decree No. 356/2025/ND-CP, effective January 1, 2026, require organizations to establish and maintain Records of Processing Activities (RoPA). Manual RoPA preparation is labor-intensive, while cloud-hosted large language models (LLMs) may conflict with data-sovereignty requirements. We propose RoPA Manager, a system for automated RoPA information extraction using hybrid retrieval that combines lexical ranking over tsvector, dense-vector search, Reciprocal Rank Fusion (RRF), and locally deployed LLMs. We introduce a Vietnamese RoPA benchmark with 32 organizations, 77 processing activities, 12 field groups, and 4,338 reference values. Evaluation is reported at three distinct levels. The automated scorer, tested on perturbed data without invoking an LLM, achieved F1 = 0.9493 [0.9436, 0.9548]; this measures scorer robustness rather than end-to-end extraction accuracy. End-to-end extraction achieved token coverage of 50.04-55.25% against the reference labels. Two independent experts reviewed 1,558 reference values (35.9% of the benchmark), found no incorrect values, and achieved 99.68% agreement with PABAK = 0.9936. Value-level precision was not measured. Across 32 paired scenarios on a 24 GB GPU, locally deployed Qwen3.5-27B-GPTQ-Int4 showed no statistically significant difference from cloud-based DeepSeek-V4-Flash (difference 0.20 percentage points in favor of DeepSeek, 95% CI [-0.93, 1.32], p = 0.72), while Gemma-4-31B performed significantly worse (p < 0.01).

I. INTRODUCTION

Vietnamese RoPA extraction must address legal recordkeeping duties while limiting exposure of personal data to cloud services. RoPA Manager combines a Vietnamese benchmark, separated evaluation levels, and locally deployable LLMs to address a gap in Vietnamese RoPA automation.

I. INTRODUCTION

  • Law 91/2025/QH15 and Decree 356/2025/ND-CP require organizations processing personal data to maintain RoPA, while manual preparation demands legal expertise and staff time.The paper also describes manual preparation as error-prone, especially for organizations with many processing activities.Source: §I · p. 1
  • Cloud LLM APIs may conflict with data sovereignty when processing PII, motivating a solution that keeps personal data inside organizational infrastructure.RoPA Manager’s security design also addresses prompt injection and cloud-model data leakage.Source: §I · pp. 1–2
  • RoPA Manager contributes a Vietnamese benchmark of 32 simulated organizations, 77 activities, 12 field groups, and 4,338 reference values, with 35.9% independently reviewed.It also introduces a three-level reporting scheme separating scorer performance, end-to-end coverage, and reference-label review.Source: §I · pp. 1–2

II. RELATED WORK

  • The authors’ 2020–2026 search found no prior automated Vietnamese RoPA extraction work, while existing RoPA tools still require manual data entry.Related work also notes that legal extraction benchmarks and privacy-policy automation cited here focus on English or European legal requirements.Source: §II · p. 2
  • Hybrid retrieval combines exact keyword matching for Vietnamese legal terms with semantic similarity, and RRF merges rankings using rank positions rather than score scales.The system uses multilingual-e5-base for Vietnamese-English support and compatibility with pgvector.Source: §II · p. 2

III. SYSTEM DESIGN AND METHODS

RoPA Manager separates its application into four layers and keeps its AI layer independent of the model endpoint. Its extraction workflow combines lexical and dense retrieval, rank fusion, and schema-validated LLM output.

A. Overall architecture

  • The four-layer architecture separates presentation, business logic, artificial intelligence, and data, while endpoint independence permits switching between cloud and local LLMs through configuration.The design aims to keep personal data within internal infrastructure while minimizing components to operate.Source: A. Overall architecture · p. 2

B. Hybrid retrieval and extraction workflow

  • The six-stage workflow preprocesses documents, chunks and embeds text, retrieves context, calls an LLM, validates structured output, and stores results for review.Semantic chunking uses a 0.62 break threshold; each retrieval branch over-retrieves 30 candidates before weighted RRF fusion.Source: B. Hybrid retrieval and extraction… · p. 2
  • Lexical retrieval uses PostgreSQL tsvector ranking and dense retrieval uses cosine search through HNSW; weighted RRF uses k = 60 and lexical weight 0.4.The evaluated version keeps the top 8 chunks as context and does not use cross-encoder reranking; the RRF defaults have not been systematically tuned.Source: B. Hybrid retrieval and extraction… · p. 2, Table I · p. 3
  • The evaluation diagram distinguishes full-workflow LLM extraction, scorer testing on noise-modified reference labels without an LLM, and review of the reference labels themselves.The caption identifies these as levels 2, 1, and 3, respectively.Source: Figure 1 · p. 3

C. RoPA field groups

RoPA Manager maps statutory RoPA information into reporting groups and corresponding extraction queries. Its design combines layered security defenses with locally deployable model options.

C. RoPA field groups

  • The system organizes RoPA information into 12 reporting groups based on Law 91/2025/QH15 and Decree 356/2025/ND-CP, with reporting groups serving as the unit of labels and measurement.The mapping to extraction queries is many-to-many: some groups combine multiple queries, two share one query, and DOCUMENTATION is derived from description and cited sources.Source: C. RoPA field groups · p. 3

D. Information security design

  • Three workflow defenses use Pydantic schema validation, the model’s safety refusal mechanism, and constrained JSON output to address malformed output, out-of-task instructions, and free-text leakage.The design uses Zero Trust and STRIDE threat analysis to identify traditional and AI-specific risks.Source: D. Information security design · p. 3

A. Infrastructure and models

  • The evaluation uses cloud DeepSeek-V4-Flash and locally deployed Qwen3.5-27B-GPTQ-Int4 and Gemma-4-31B, with both local models served by vLLM on one 24 GB GPU.Experiments use synthetic data with the cloud API; the paper says real-data use must switch to local mode.Source: A. Infrastructure and models · p. 3

B. Benchmark dataset

The benchmark uses manually authored synthetic Vietnamese documents and labels across seven sectors, while evaluation separates scorer behavior, end-to-end coverage, and label reliability. Reproducibility is supported by frozen component versions and recorded run metadata.

B. Benchmark dataset

  • The benchmark covers 32 simulated organizations and 77 activities across seven sectors, with manually written Vietnamese source documents containing English terms and 12 labeled field groups.Activities are distributed across difficulty levels as 21 easy, 33 medium, and 23 hard; the authors’ labels were not generated by an LLM.Source: B. Benchmark dataset · pp. 3–4

C. Measurement method

  • Evaluation distinguishes scorer tests without LLM calls, full-workflow token coverage on 32 scenarios, and independent review of reference-label reliability.The review covers 1,558 values from 15 scenarios and 41 activities across all sectors and field groups; scenario-order sampling yields sector sampling rates from 33% to 100%.Source: C. Measurement method · p. 4

D. Reproducibility

  • Runs freeze dataset, scorer, code, retrieval parameters, and prompt versions, verify integrity with SHA-256, and record run identifiers and execution settings.The source code, benchmark, and expert review records are publicly available.Source: D. Reproducibility · p. 4

V. RESULTS AND DISCUSSION

End-to-end extraction produced 50.04–55.25% token coverage, with local Qwen statistically comparable to cloud DeepSeek and Gemma lower. Scorer robustness, reference-label agreement, and security were evaluated separately, with important limits on attribution and scope.

Scorer evaluation

  • F1 = 0.9493 [0.9436, 0.9548] in noisy scorer mode measures tolerance to imperfect data, not end-to-end extraction performance.The scorer was tested without an LLM; six of seven level-1 metrics met their thresholds.Source: Table II · p. 4, §V · p. 4

End-to-end extraction

  • 55.25% token coverage for DeepSeek-V4-Flash versus 55.06% for local Qwen3.5-27B; their +0.20-point difference was nonsignificant (95% CI [-0.93, +1.32], p = 0.72), while Gemma scored significantly lower.Gemma was 5.21 points below DeepSeek and 5.02 points below Qwen; its largest gap was in LEGAL_BASIS.Source: Table IV · p. 5, §V · p. 5

Chunking experiment

  • Semantic chunking averaged 42 chunks per document versus 6.9 for recursive chunking, while the eight-chunk context covered only about 19% of each document.The authors identify chunking as a competing explanation for 50.04–55.25% token coverage and call for an ablation study.Source: §V · p. 5

Retrieval limitation

  • Retrieval quality was not measured with Recall@K, MRR, or NDCG because no query–chunk relevance benchmark existed, so low coverage is not attributed to retrieval or generation.Source: §V · p. 5

Reference-label review

  • 99.68% expert agreement (1,553/1,558) and no values rated Wrong support reference-label consistency, with disagreements confined to one legal-interpretation issue.PABAK was 0.9936; both experts came from the same business ecosystem.Source: §V · pp. 5–6

Security evaluation

  • The security test blocked all eight prompt-injection scenarios, but n = 8 and one run make this proof of concept rather than strong evidence of attack resistance.Configuration review also found two failures: missing HSTS and an unauthenticated API endpoint, both requiring correction before operation.Source: §V · p. 6

Evaluation scope

  • The three-level evaluation does not measure value-level precision on real model output, limiting conclusions to technical feasibility and coverage.Source: §V · p. 6

VI. CONCLUSION AND FUTURE WORK

The study establishes a technical baseline for LLM+RAG RoPA automation and outlines further evaluation priorities. Its scope is bounded by synthetic data and other unmeasured or limited evaluation dimensions.

VI. CONCLUSION AND FUTURE WORK

  • The study contributes a Vietnamese RoPA benchmark and three-level reporting scheme as a technical baseline for automating compliance with Vietnam’s Personal Data Protection Law.It also reports GPU, DPO, and commercial GRC cost references.Source: §VI · p. 6
  • Future work prioritizes value-level content accuracy on real drafts, repeated multi-model experiments, retrieval benchmarking, hybrid-retrieval ablation, and confidence-score calibration.The proposed accuracy study would use stratified sampling, two reviewers, three labels, and precision and recall by field group.Source: §VI · p. 6
  • The cost reference places a 24 GB GPU at VND 50–65 million, a full-time DPO at VND 180–300 million per year, and commercial GRC at USD 10,000–50,000 per year.These are reported as cost references, not as a direct cost-effectiveness comparison.Source: §VI · p. 6
  • All data are synthetic, value-level precision is unmeasured, and the study does not claim full operational legal compliance.Other limits include single-run paired tests, no retrieval metrics or RRF ablation, and reference-label review of 35.9% of the benchmark by two experts using non-random scenario-order sampling.Source: §VI · p. 6

I. GIỚI THIỆU

Vietnamese organizations must maintain RoPA, while manual preparation is demanding and cloud LLM use for PII conflicts with data-sovereignty principles. RoPA Manager addresses this setting with a Vietnamese benchmark, separated evaluation reporting, and investigation of local deployment.

I. GIỚI THIỆU

  • Vietnam’s law and implementing decree require organizations processing personal data to create and maintain RoPA, while manual preparation demands legal expertise, staff time, and is error-prone.The burden is especially salient for organizations with many activities across business areas.Source: §I · p. 8
  • Cloud LLM APIs processing PII conflict with data sovereignty, motivating extraction that keeps personal data within organizational infrastructure.LLM+RAG can automate extraction from business documents into a standard RoPA structure.Source: §I · p. 8
  • RoPA Manager contributes a first Vietnamese RoPA benchmark, three-level reporting, and a local-deployment feasibility study in which one open-source model matched the cloud model’s coverage range.The benchmark includes 32 simulated organizations, 77 activities, 12 field groups, and 4,338 reference values; two open-source models completed 32 scenarios on a 24 GB GPU.Source: §I · p. 8

II. NGHIÊN CỨU LIÊN QUAN

  • Prior tools still require manual entry, while cited compliance and legal-extraction work is English- or Europe-focused; the authors’ search found no Vietnamese RoPA extraction work.The search covered Google Scholar, IEEE Xplore, and ACM Digital Library for 2020–2026.Source: §II · p. 9
  • The paper motivates LLMs for semi-structured, variable-layout source documents and hybrid retrieval for exact Vietnamese legal terms plus semantic similarity.RRF combines rankings without relying on their score scales; multilingual-e5-base is selected for Vietnamese-English support and pgvector compatibility.Source: §II · p. 9

III. THIẾT KẾ HỆ THỐNG VÀ PHƯƠNG PHÁP

RoPA Manager uses a layered architecture with an independently configurable AI endpoint. Its evaluation framework distinguishes end-to-end LLM workflow runs from scorer-only tests and reference-label review.

A. Kiến trúc tổng thể

  • The AI layer is model- and endpoint-independent, allowing configuration-based switching between cloud and locally deployed LLMs.The system is organized into presentation, business, AI, and data layers.Source: A. Kiến trúc tổng thể · p. 9
  • Architecture choices prioritize keeping personal data within internal infrastructure while minimizing operational components.The vector database uses pgvector integrated into PostgreSQL, and local inference uses vLLM with an OpenAI-compatible interface.Source: A. Kiến trúc tổng thể · p. 9

B. Quy trình truy xuất lai và trích xuất

  • Figure 1 distinguishes three evaluation tiers: full workflow with LLM calls, scorer testing without LLM calls, and review of the reference labels.The scorer-only branch compares reference labels with controlled modified versions.Source: B. Quy trình truy xuất lai và trích xuất · p. 9

C. Nhóm trường RoPA

The method defines 12 RoPA field groups and applies layered security measures. Evaluation uses three model configurations and a manually authored synthetic Vietnamese benchmark.

C. Nhóm trường RoPA

  • The 12 reporting field groups follow Article 31 and the decree, map many-to-many to extraction queries, and define the unit for reference labels and measurement.DOCUMENTATION is inferred from description and cited sources rather than extracted by a dedicated query.Source: C. Nhóm trường RoPA · p. 10

D. Thiết kế an toàn thông tin

  • The security design combines Zero Trust and STRIDE threat analysis with three defenses: schema validation, model refusal of out-of-scope instructions, and constrained JSON output.The identified threats include prompt injection, cloud-model data leakage, and hallucinated information.Source: D. Thiết kế an toàn thông tin · p. 10

A. Hạ tầng và mô hình

  • The evaluation compares cloud DeepSeek-V4-Flash with locally deployed Qwen3.5-27B-GPTQ-Int4 and Gemma-4-31B on a single 24 GB GPU.All models share code, prompt checksum, and scorer; cloud API experiments use wholly synthetic data, while real-data operation requires local mode.Source: A. Hạ tầng và mô hình · p. 10

B. Bộ dữ liệu kiểm chuẩn

  • The benchmark contains 32 simulated organizations across seven sectors, 77 activities, and 12 field groups, with source documents and labels authored manually rather than by LLM.Its three measurement units are 924 group instances, 4,338 homogeneous-mode leaf values, and 3,965 noisy/anonymized-mode leaf values after omitting 373 fields.Source: B. Bộ dữ liệu kiểm chuẩn · p. 10

C. Phương pháp đo lường

The section evaluates the scorer, end-to-end extraction, and reference-label reliability as separate measurement tiers, then discusses model comparisons and limits on interpreting them.

C. Phương pháp đo lường

  • Three tiers separately assess scorer robustness without an LLM, token coverage over 32 paired scenarios, and reference-label reliability.The 1,558-label expert sample was independently scored, without discussion and blind to model-generated labels.Source: C. Phương pháp đo lường · pp. 10–11

D. Khả năng tái lập

  • The study froze dataset, scorer, code, retrieval parameters, and prompts for comparisons, tracked run metadata, and publicly released code and benchmark materials.The configurations were integrity-checked using SHA-256.Source: D. Khả năng tái lập · p. 11

V. KẾT QUẢ VÀ THẢO LUẬN

  • F1 = 0.9493 [0.9436, 0.9548] on perturbed scorer inputs, while correct refusal scored 0.7874 against a 0.80 threshold.This tier did not invoke an LLM and measures scorer robustness, not extraction performance.Source: §V · p. 11
  • 55.25% vs 55.06% token coverage left DeepSeek–Qwen difference at +0.20 percentage points (95% CI [-0.93, +1.32], p = 0.72), with no significant difference.Gemma also completed all 32 scenarios, but performed significantly worse than both alternatives.Source: §V · pp. 11–12
  • Gemma trailed DeepSeek by 5.21 points (p = 0.0022) and Qwen by 5.02 points (p = 0.0014), with the gap concentrated in LEGAL_BASIS.There, Gemma achieved 41.6–56.1% coverage versus 83.5–100% for the other two models.Source: §V · p. 12
  • With eight retrieved chunks, semantic chunking supplies about 19% of each document as query context, a proposed but untested explanation for 50.04–55.25% token coverage.Both chunking approaches kept all segments within the 512-token embedding limit across 32 documents.Source: §V · p. 12
  • No reviewed reference value was marked incorrect; experts agreed on 99.68% (1553/1558), with PABAK = 0.9936.The sample covered 1,558 labels, 35.9% of the benchmark, across 15 of 32 scenarios and 41 of 77 activities.Source: §V · p. 12

V. KẾT QUẢ ו́À THẢO LUẬN

  • The system blocked all eight tested prompt-injection scenarios, but one run is only a proof of concept; two configuration failures require remediation before operation.The failed checks were missing HSTS and an API endpoint without authentication.Source: §V · p. 12

VI. KẾT LUẬN VÀ HƯỚNG PHÁT TRIỂN

The paper establishes a technical baseline for Vietnamese RoPA automation and identifies evaluation gaps that bound its conclusions and guide further work.

VI. KẾT LUẬN VÀ HƯỚNG PHÁT TRIỂN

  • The study establishes a technical baseline for applying LLM plus RAG to Vietnamese personal-data compliance and contributes the first Vietnamese RoPA benchmark.The authors do not claim that the system achieves full operational legal compliance.Source: §VI · pp. 12–13
  • Further work should repeat model experiments across at least three runs and isolate hybrid retrieval’s contribution with retrieval metrics and ablations.Proposed measures include Recall@K, MRR, and NDCG.Source: §VI · pp. 12–13
  • The evaluation uses fully synthetic data and has not been validated on real RoPA records or measured value-level precision.The authors identify value-level precision and recall on real drafts as a priority for future evaluation.Source: §VI · pp. 12–13
  • The expert label review covered 35.9% of the benchmark using two experts and scenario-order sampling rather than random sampling.The authors also call for assessing calibration of model-assigned confidence on real outputs.Source: §VI · pp. 12–13
Loading 2609.27359v1…