PaperlayerView exact revision on arXiv

Source-linked AI summary

Privasis: Synthesizing the Largest "Public" Private Dataset from Scratch

Hyunwoo Kim, Niloofar Mireshghallah, Michael Duan, Rui Xin, Shuyue Stella Li, Jaehun Jung, David Acuna, Qi Pang, Hanshen Xiao, G. Edward Suh, Sewoong Oh, Yulia Tsvetkov, Pang Wei Koh, Yejin Choi

arXiv:2602.03183v1cs.CLcs.AI

TL;DR

Privacy research is constrained by scarce data, while agentic systems increasingly process personal information under privacy requirements. Privasis addresses this gap with a million-scale synthetic dataset and sanitization corpus, and its compact models outperform tested frontier LLMs on text sanitization.

  • Problem

    Privacy research has long relied on small, narrowly scoped datasets because private information cannot be publicly shared, while agentic systems increasingly process personal records under privacy requirements.

  • Method

    Privasis synthesizes diverse private-information records from scratch using auxiliary control variables and diversity-preserving refinement, then constructs a parallel corpus for targeted text sanitization.

  • Results

    Small models of up to 4B parameters outperform tested frontier LLMs on text sanitization, with PRIVASIS-CLEANER-4B achieving 72.5% Full Success on the vanilla test set.

  • Takeaways & Limitations

    Privasis provides a privacy-safe, million-scale resource for research on privacy-preserving generation, controllable sanitization, and agentic systems processing sensitive data.

  • Takeaways & Limitations

    The dataset does not represent real-world events and should not be used for clinical decision-making, financial analysis, or actions involving real individuals.

Abstract

from arXiv · show

Research involving privacy-sensitive data has always been constrained by data scarcity, standing in sharp contrast to other areas that have benefited from data scaling. This challenge is becoming increasingly urgent as modern AI agents--such as OpenClaw and Gemini Agent--are granted persistent access to highly sensitive personal information. To tackle this longstanding bottleneck and the rising risks, we present Privasis (i.e., privacy oasis), the first million-scale fully synthetic dataset entirely built from scratch--an expansive reservoir of texts with rich and diverse private information--designed to broaden and accelerate research in areas where processing sensitive social data is inevitable. Compared to existing datasets, Privasis, comprising 1.4 million records, offers orders-of-magnitude larger scale with quality, and far greater diversity across various document types, including medical history, legal documents, financial records, calendars, and text messages with a total of 55.1 million annotated attributes such as ethnicity, date of birth, workplace, etc. We leverage Privasis to construct a parallel corpus for text sanitization with our pipeline that decomposes texts and applies targeted sanitization. Our compact sanitization models (<=4B) trained on this dataset outperform state-of-the-art large language models, such as GPT-5 and Qwen-3 235B. We plan to release data, models, and code to accelerate future research on privacy-sensitive domains and agents.

1 Introduction

PRIVASIS addresses privacy research’s data scarcity by synthesizing diverse private-information records from scratch and using them to train sanitization models. Its pipeline combines controlled generation, diversity-preserving refinement, and decomposition-based sanitization.

  • Dataset and motivation: PRIVASIS is a million-scale synthetic dataset built entirely from scratch for privacy research, spanning diverse private-information records.Its pipeline generates medical, legal, financial, and communication documents with detailed annotations.
  • Dataset and motivation: Auxiliary control variables guide record initialization, while iterative rejection sampling combines LLM quality scoring with Vendi diversity metrics.The approach aims to preserve realism and diversity without reference data.
  • Dataset and motivation: PRIVASIS domain subsets achieve MATTR values of 0.807–0.823 versus 0.700–0.794 for existing human-written datasets, alongside higher bigram diversity and Shannon entropy.The subsets also show lower cosine similarity, indicating reduced semantic redundancy.
  • Sanitization corpus and models: PRIVASIS-SANITIZATION decomposes records into chunks and applies targeted abstraction or removal while preserving selected non-sensitive information.The resulting triplets contain an original record, sanitization instruction, and sanitized record.
  • Sanitization corpus and models: The 4B-parameter PRIVASIS-CLEANER achieves a 72.5% full success rate on the vanilla test set, surpassing o3 at 70.3%.It maintains competitive performance on the hard set at 12.4% versus GPT-5’s 13.1%.
  • Dataset and motivation: PRIVASIS is presented as a privacy-safe, privacy-rich resource intended to accelerate research on sanitization and other privacy-sensitive methods.The authors plan to release the data, models, and code.

2 Synthesizing Privacy-rich Text Data from Scratch

PRIVASIS synthesizes privacy-rich records from scratch using auxiliary variables, iterative refinement, and diversity selection, without real-world reference data. The resulting dataset spans diverse domains and achieves million-scale size, broad attribute coverage, and naturalness comparable to human-written records.

  • Synthesis Pipeline: PRIVASIS uses profiles, record types, and background contexts to initialize diverse records without relying on real-world reference data.Profiles include attributes such as gender, ethnicity, and date; record types and contexts are derived from those profiles.
  • Synthesis Pipeline: Iterative refinement combines an LLM judge’s specificity and realism assessment with Vendi-based diversity selection to prevent records from collapsing into similar patterns.Candidate drafts are compared with the current draft, and acceptance uses a weighted score; refinement runs for up to three steps with threshold τ = 0.5.
  • Synthesis Pipeline: 94% generation success and post-generation filtering produce records while excluding short records, underage profiles, and degenerate cases.The filtering removes 66,894 records under 64 words, 1,180 profiles under age 18, and 1,232 degenerate cases.
  • Statistics & Analysis: 1,414,871 records contain 55,092,084 annotated attributes, averaging approximately 39 attributes per record, with multiple models contributing stylistic and distributional diversity.The records include basic profile details, richer dates and locations, grouped attributes, and contextual metadata.
  • Statistics & Analysis: 95% of generated profiles are non-U.S., while the sex distribution is 59% female and 41% male; ethnicity labels have heterogeneous granularity.The paper attributes the non-U.S. proportion to using first names from the diverse U.S. SSN applicant database.
  • Statistics & Analysis: PRIVASIS records cover 10 primary categories and 42 subcategories, with Health & Wellness most common at 20.7%, followed by Government & Civic at 13.5% and Business & Finance at 13.4%.The categories were created by reclustering 592 automatically generated categories and manually refining the hierarchy.
  • Statistics & Analysis: PRIVASIS subsets consistently show greater diversity than related human-written datasets across MATTR, bigram diversity, Shannon entropy, and cosine similarity.The comparison reports richer vocabulary and syntactic variation, more uniform word use, and less repetition across PRIVASIS domains.
  • Statistics & Analysis: 113 of 128 PRIVASIS records were judged natural and coherent, compared with 111 of 128 human-written records in a blind evaluation.Seven annotators evaluated randomly sampled records from both collections.

3 Building a Sanitization Parallel Corpus

PRIVASIS-SANITIZATION is a parallel corpus for contextual, instruction-following sanitization that balances privacy with retained utility. Its decomposition-based pipeline supports abstraction or deletion across long, diverse records and produces training data for lightweight local models.

  • Corpus Design: PRIVASIS-SANITIZATION represents sanitization as triplets of an original record, instruction, and sanitized record, treating sensitivity as contextual.The corpus supports flexible strategies that balance privacy with utility rather than limiting sanitization to fixed PII categories.
  • Sanitization Pipeline: The pipeline recursively decomposes records into locally coherent chunks of at most 512 characters before sanitization.Natural boundaries such as double newlines and EOS markers guide variable-length decomposition.
  • Sanitization Pipeline: Target selection assigns sensitivity weights to annotated attributes, samples targets, and labels each for abstraction or dropping.The weighting prioritizes highly sensitive information over relatively benign details that are difficult to sanitize.
  • Sanitization Pipeline: For each target, the pipeline identifies relevant chunks, extracts matching spans, and generates context-grounded abstraction or deletion instructions.Abstraction instructions use all relevant chunks, whereas DROP targets use a fixed instruction.
  • Sanitization Pipeline: Retention targets explicitly preserve selected attributes, and the pipeline combines target instructions into a final coherent instruction and sanitized record.Retention attributes are selected to minimize lexical overlap with sanitization targets, supporting utility during rewriting.
  • Corpus Statistics: The corpus contains 100K examples, including 37K training examples, with instructions averaging 69.4 words and specifying 2.9 targets.Evaluation uses 2.1K records generated with GPT-5, Gemini-2.5-Pro, Qwen3-235B, and LLaMA-4 Maverick.
  • Comparison with Existing Resources: Compared with existing privacy resources, PRIVASIS-SANITIZATION targets longer, cross-domain text and supports rewritten alternatives beyond deletion and predefined PII categories.PRIVASIS records average 527 words, while annotated spans average 35.3 characters.

4 Experiments

Experiments evaluate sanitization using hierarchical leakage metrics, retention metrics, Vanilla and Hard test sets, robustness checks, failure analyses, and zero-shot generalization. PRIVASIS-CLEANER models achieve competitive or superior performance to frontier LLMs while retaining more non-target information.

  • Evaluation: Sanitization evaluation detects direct, inference, and proximity leaks, and counts a record successful only when no target attribute leaks.Exact string matching checks direct leaks; evaluator-LLM predictions assess inference and proximity leaks.
  • Test Sets: 1,042 Vanilla records have perfect pipeline scores, whereas 1,149 Hard records contain pipeline failures and more grouped attributes.Grouped attributes comprise 60% of Vanilla records and 87% of Hard records.
  • Performance: 72.50% Full Success Record on Vanilla makes PRIVASIS-CLEANER-4B outperform o3 at 70.3% and all tested frontier models.Frontier-model Full Success Record rates range from 64.4% for Qwen3-235B to 70.3% for o3.
  • Performance: 12.80% Full Success Record on Hard ranks PRIVASIS-CLEANER second only to GPT-5 at 13.14%.Hard-set performance declines sharply because of more challenging attribute selections and longer contexts.
  • Retention: 99.2% attribute retention for PRIVASIS-CLEANER-4B exceeds GPT-5’s 93.4% on Vanilla, despite most models exceeding 90% Successful Attribute/Record.Full-record success remains around 70% because models often miss at least one target attribute.
  • Evaluation Robustness: Human evaluation judged 98% of o3 proximity-leak failures correct, while two evaluator models reached 97% inter-model agreement over 5K cases.These checks support the reported precision and robustness of the proximity-leak evaluation.

5 Related Work

Related privacy datasets cover synthetic generation, anonymization, and data minimization, but remain limited in scale, domain coverage, or supervision. PRIVASIS combines large-scale multi-domain data with PII and sensitive-span labels and parallel abstraction/removal pairs.

  • Synthetic Data Generation: Prior synthetic-data approaches include differentially private generation, public-to-private pipelines, self-instruction, targeted prompting, and automated refinement.These approaches differ in whether they use private training data, public generators, or prompting and refinement procedures.
  • PII Removal Datasets: Classic anonymization corpora provide span-level detection, while newer datasets use synthetic documents and LLM interactions.The cited PII-removal datasets remain small-scale or domain-specific.
  • Data Minimization: Data-minimization work studies disclosure rewrites, naturalness benchmarks, LLM anonymization, and generalization strategies.PRIVASIS unifies these directions with fine-grained span labels and parallel abstraction/removal pairs across diverse document types.

6 Conclusion

PRIVASIS addresses the scarcity of large-scale public datasets with rich sensitive attributes by providing a fully synthetic, million-scale resource and a sanitization corpus. Its construction uses controllable generation without real-world private data, while intended use remains limited to privacy-preserving research and evaluation.

  • PRIVASIS contains over 1M synthetic records spanning medical, financial, legal, email, calendar, and meeting domains.The dataset is presented as a resource for addressing data scarcity in privacy-sensitive research.
  • PRIVASIS-SANITIZATION provides triplets for developing compact models that balance privacy with textual utility.The corpus is constructed from original records, sanitization instructions, and sanitized records, with targeted abstraction or removal strategies.
  • The dataset is generated from scratch using profiles, record types, background contexts, formats, and prompts that produce detailed records across diverse settings.Profile generation populates demographic and identifying attributes, after which record types, contexts, formats, and records are generated sequentially.
  • All personal identifiers are fictitious, and the pipeline does not incorporate or reference real-world private data.The stated safeguards are intended to reduce privacy infringement and re-identification risks.
  • The dataset and models are intended for non-commercial research and evaluation, not clinical decisions, financial analysis, or actions involving real individuals.The paper also commits to releasing the data, models, and code to the research community.

A.1 Analysis

The analysis compares language models for PRIVASIS synthesis using diversity, lexical, length, and manual-quality measures, and examines multilingual generation. GPT-OSS-120B is reported as consistently strong and cost-effective, while Chinese and Korean outputs receive positive native-speaker evaluations.

  • GPT-OSS-120B performs consistently strongly across diversity and lexical measures and generates longer text than other compared models.Manual review found its generations highly coherent.
  • GPT-OSS-120B offers a substantially better price-performance ratio than Gemini-2.5-pro, GPT-5, and Qwen3-235B for large-scale generation.
  • The analysis includes figures for ethnicity distribution, record subcategories, and model diversity and lexical metrics.
  • Native-speaker reviewers found Chinese and Korean records coherent, contextually appropriate, and diverse, with cosine similarity scores of 0.34 and 0.36, respectively.The records were generated with Exaone 3.5 32B, Qwen3 80B, and GPT-4.1 through the pipeline.

B Building the Sanitization Parallel Corpus

The sanitization corpus is built with a decomposition-based pipeline that selects sensitive targets, applies abstraction or removal, preserves selected attributes, and recombines the text. Experiments identify 512-character chunks as the strongest setting because smaller or larger chunks lose context or overwhelm models.

  • The pipeline decomposes each record into chunks of at most 512 characters before sanitization.The algorithm uses τ = 512 as its chunk threshold and merges processed chunks into the sanitized record.
  • It samples sensitive attributes with sensitivity weights, assigns each target an abstraction or drop label, and applies targeted operations to relevant chunks.Relevant spans are extracted, then either abstraction instructions or fixed drop instructions are applied consistently across chunks.
  • The pipeline optionally selects retention attributes and generates a user-style instruction alongside the original and sanitized records.This produces training triplets that preserve non-sensitive information for utility.
  • Retention targets are selected to minimize lexical overlap with sanitization targets, measured using ROUGE.The design avoids overlap that could make correct sanitization remove retention targets as well.
  • 512-character chunks yield peak sanitization performance, while 128-character chunks lose context and 2048-character chunks make models struggle.Performance gradually declines for larger chunks after the 512-character peak.

C Training

The training setup fine-tunes compact Qwen3 and Llama-3.2 models with DeepSpeed on NVIDIA H100 GPUs using specified optimization, scheduling, precision, and ZeRO configurations.

  • Qwen3 4B and 0.6B are fine-tuned with DeepSpeed on NVIDIA H100s, while Llama-3.2 4B is trained on two GPUs.The configurations use AdamW, WarmupDecayLR, bf16 precision, and ZeRO optimization stages.

D Evaluation

The evaluation uses attribute-matching procedures to assess sanitization and retention, with vanilla examples requiring direct substitutions and harder examples requiring contextual understanding. The supplied records illustrate diverse medical, financial, and legal content alongside sanitization instructions.

  • Evaluation procedure: Retention attributes are first checked by exact string matching, then inferred by GPT-OSS-120B when no exact match is found.The evaluator may subsequently compare inferred values against the sanitized text.
  • Test sets: Vanilla examples contain one attribute occurrence and direct instructions, whereas hard examples use longer instructions requiring contextual understanding.
  • Evaluation examples: The examples span forensic financial records, medical session notes, and targeted instructions to generalize, remove, or retain sensitive attributes.
  • Evaluation examples: The medical example includes occupational burnout, work demands, emotional effects, cognitive symptoms, and a treatment plan.

E Error Analysis

Error analysis identifies difficult attribute classes and several leakage patterns. Models particularly struggle with names and dates, while surface-level removal can leave sensitive information inferable from contextual clues.

  • Failed attributes: Models struggle most with name-related attributes, including last names, full names, and user handles, as well as dates.
  • Error-analysis setting: Sanitization instructions can combine abstraction, removal, and retention requirements across employment, publication, and medical records.
  • Leakage patterns: Inference leaks occur when GPT-5 removes explicit journal-name occurrences but leaves an email domain that allows the evaluator to infer the journal.
  • Leakage patterns: A proximity leak occurred when scattered temporal cues allowed reconstruction of a removed lithium dose time.Exact string matching did not detect the reconstructed value, but the evaluator marked the case as a sanitization failure.

F Extended Related Work

Related work covers privacy-preserving synthetic data, general synthetic-data generation, PII removal, abstraction, and behavioral privacy benchmarks. Privasis combines fine-grained supervision with paired abstraction and removal across diverse document types.

  • Privacy-preserving synthesis: Differentially private synthetic-data methods commonly train generative models on private data with DP-SGD, while public-to-private pipelines reduce training cost through repeated public-model queries.
  • Synthetic data: General synthetic-data methods often rely on fixed prompts, seed data, or reference trajectories, whereas Privasis synthesizes million-scale data from scratch using auxiliary control variables.
  • PII removal: PII-removal work focuses on span detection and redaction for concrete identifiers such as names, addresses, dates, and account numbers.
  • Abstraction: Data-minimization methods preserve meaning by deleting, obscuring, or abstracting sensitive details without necessarily targeting strict PII.
  • Privasis contribution: Privasis unifies fine-grained labels for PII and non-PII sensitive content with parallel abstraction and removal pairs across diverse document types.
  • Behavioral benchmarks: Behavioral privacy benchmarks expose leakage and defense failures but do not provide the aligned span-level supervision needed to train sanitizers.

Monthly Summary with Key Metrics

The supplied monthly records report financial activity, health and lifestyle updates, civic actions, and a commercial legal case. Key figures include the loan balance, payments, interest, fees, weight change, and settlement outcome.

  • Financial summary: LAK 2,450,300 was the current balance on 2026-04-30, with a fixed 4.2% interest rate and a LAK 14,600 payment due on 2026-05-07.
  • Financial summary: LAK 43,800 in payments produced LAK 15,720 interest, LAK 1,200 fees, and a LAK 26,880 net reduction in principal.
  • Health and lifestyle: The health update records a 20 kg weight loss from 88 kg to 68 kg after a six-month phase.
  • Civic activity: The civic plan proposed three phases, including source-separation bins, expanded collection routes, composting, and a city-wide recycling hub.
  • Legal case: The commercial case ended with court-approved settlement terms totaling RWF 38,500,000 after mediation.
  • Legal case: The case closure report recorded a 14-month lawsuit duration and total legal fees of RWF 4,200,000.

To the Admissions Committee,

The applicant describes progressive experience in environmental research, renewable energy, data analysis, and teamwork. They report leading a prize-winning project and seek further study in environmental physics and renewable energy systems.

  • Scientific preparation: The applicant began developing scientific interests through school research on LED energy efficiency and renewable energy systems.Their experience included a 27% reduction in electricity consumption and leadership of a 5.2 kW solar-array project.
  • Scientific preparation: They gained practical research experience analyzing urban microclimate data from 18 rooftop sensors using RStudio and ArcGIS.
  • Research leadership: They led a three-person project that collected 2,400 measurements to optimize urban energy mixes across solar, wind, and geothermal sources.The project involved field measurements at three Helsinki sites over 11 weeks.
  • Research leadership: The team addressed equipment failure and disagreements over analysis by adopting a hybrid regression and machine-learning approach.The data-logger malfunction caused the loss of 18 hours of data.
  • Achievements: The project won first prize, a €3,000 scholarship, and a research internship at Aalto University’s Department of Energy Technology.The jury cited the integration of real-time data and predictive modeling, along with the applicant’s leadership in overcoming adversity.
  • Academic goals: The applicant seeks further study in environmental physics and renewable energy systems while contributing to sustainable urban solutions.
Loading 2602.03183v1…