Source-linked AI summary
Domain Agnostic Text Redaction from Natural Language Rules using Instruction Tuning
Aravindhan Arunagiri, Ayaan Khan, Udayaadithya Avadhanam, SaiBarath Sundar
TL;DR
Existing sanitization methods struggle with semantically rich, variably phrased sensitive content and often provide limited explanations. This paper instruction-tunes a smaller language model to apply user-defined natural-language redaction rules, achieving high redaction precision and reconstruction error while supporting domain-agnostic sanitization.
Problem
Existing rule-based and manual sanitization methods are costly or poorly suited to semantically rich, variably phrased sensitive content and provide limited explanation.
Method
The method generates synthetic natural-language redaction rules, instruction-tunes a smaller language model, and applies the trained model to redact sensitive content with explanations.
Results
The solution shows high reconstruction errors and redacts only sensitive content, with sensitive and redacted spans matching in at most 99% of evaluated samples.
Takeaways & Limitations
The system supports user-defined redaction of free-text sensitive content across document domains with less manual intervention and high utility.
Takeaways & Limitations
Sketch-based approaches remain non-scalable for complex data semantics because each distinct representation of sensitive information requires a custom sketch.
Abstract
from arXiv · showhide
With the increasing digitization of personal and corporate communication, the automatic sanitization of textual data has become a crucial component of data privacy and compliance frameworks. Traditional text sanitization solutions are majorly suitable for obscuring sensitive data with standard structure such as Personal Identifiable Information (PII). These solutions do not provide transparent justification for their redaction, which makes it difficult to audit them. This paper introduces an explainable, domain-agnostic text redaction solution that uses natural language rules of redaction, applied via an instruction-tuned language model, to identify and redact sensitive information in unstructured documents. Unlike traditional text sanitization, this method enables a user to conveniently define any sensitive information; which may be structured (e.g.\ PII) or unstructured (e.g.\ legal terms and conditions) in natural language. A general-purpose LLM generates or augments these natural language rules of redaction from the user's definition, which are then used to instruction-fine-tune a smaller language model that reasons the rules step-by-step over any given document to identify and redact the corresponding sensitive content, while providing transparent justifications for each redaction and highlighting the specific rule that triggered the decision. This explanation is generated in natural language to support human reviewers and auditors in understanding why specific content was redacted. A reconstruction-based metric is used to estimate the probability of recovering redacted information from the sanitized document, quantifying redaction coverage. The solution shows high reconstruction error and high redaction precision, making it suitable for automated text sanitization in critical applications such as legal discovery, medical documentation, and corporate information governance.
1 Introduction
The introduction motivates text sanitization as essential for protecting sensitive information under expanding textual-data use and privacy regulations. It identifies limits of pattern-based, manual, and opaque approaches for irregular, context-dependent content, then presents an explainable language-model solution using natural-language redaction rules.
- Motivation: Growing use of large-scale textual data for analytics, legal discovery, and digital services makes protecting sensitive information essential under laws including GDPR, HIPAA, and CCPA.The passage identifies PII and commercial confidential information as examples of regulated sensitive data.
- Problem: Existing sanitization methods handle regular patterns such as PII, but business, regulatory, and legal requirements also define irregular content including legal clauses, medical diagnoses, and contract terms.These non-regular forms are presented as a central limitation of current methods.
- Problem: Manual irreversible redaction is labor-intensive, skill-dependent, and costly as data variety or volume increases, while redaction quality depends on accurately identifying sensitive information.The passage connects identification quality directly to overall redaction quality.
- Limitations: Phrase and speech-pattern variation makes one sensitive context appear in multiple forms, requiring rule-driven sketch approaches to create custom sketches for distinct textual patterns.The passage states that pattern variety grows with the variety of sensitive content and its context.
- Contribution: The proposed solution uses a smaller fine-tuned language model to reason over documents, dynamically generate or select natural-language redaction rules, redact sensitive information, and provide natural-language explanations.The approach is described as combining rule-based interpretability with contextual understanding while addressing the limitations of black-box and rigid rule-based systems.
2 Preliminaries and Related Work
Prior redaction methods range from manual review and pattern-based techniques to NER, differential privacy, and LLM-based approaches, but remain limited by scalability, semantic coverage, domain flexibility, or changing rules. Recent work therefore emphasizes instruction-based redaction and quantitative evaluation balancing sensitive-content removal with preservation of non-sensitive content.
- Manual and pattern-based methods: Manual redaction is time-consuming and costly because it depends on subject-matter expertise as document variety and volume increase.The passage also links automated redaction’s importance to privacy regulations including GDPR and HIPAA.
- Manual and pattern-based methods: Regular expressions handle regularly formatted data but fail on rich-semantic content spanning sentences or phrases.Sketch-based methods likewise become unsuitable when sensitive-content patterns and their document representations vary.
- Learning-based methods: NER and transformer models improve contextual identification of entities such as names, addresses, and medical terms, yet remain limited for irregular, semantically rich information.The passage specifically describes BERT as improving NER performance while retaining limitations beyond regular entity patterns.
- Learning-based methods: Differential privacy approaches lack flexibility across multiple domains and when the criteria defining sensitive content change dynamically.These methods rely on domain-specific ML knowledge and require additional models when broader domain coverage is needed.
- LLM-based methods: LLMs offer cross-domain knowledge for instructed text rewriting and de-identification, but changing sensitivity criteria requires retuning instructions.The paper positions generalized instruction-tuning as a response to this limitation.
- Evaluation: Evaluation has shifted from SME-based qualitative assessment toward recall and precision, while redaction quality must balance privacy against document utility.The passage frames this balance as removing sensitive content while preserving non-sensitive content.
3 Methodology
The methodology lets users define domain-agnostic redaction policies in natural language, generates synthetic rule-based training data, instruction-tunes a smaller language model, and applies it to documents. The pipeline redacts sensitive spans while producing natural-language explanations and auditable rule traces.
- User-Defined Policies: Users define sensitivity criteria and redaction rules in natural language for structured or unstructured content across domains and document types.The system is designed for documents ranging from financial reports to medical records and supports diverse sectors.
- Method Overview: The method has three stages: synthetic rule generation, instruction-tuning a smaller LLM, and inference-time redaction of sensitive document content.Synthetic rules and documents teach the smaller model to identify sensitive content and redact it during inference.
- Synthetic Data and Fine-Tuning: A general-purpose LLM generates broad synthetic redaction rules, which instruction-tune Phi 3.5 to reason step-by-step over document content.Training data includes redaction rules, sensitive document content, and explanations; preprocessing checks formatting and may chunk documents into fixed-length segments.
- Inference Pipeline: During inference, the fine-tuned model identifies sensitive words or phrases, replaces them with generalized tokens such as [Redacted], and outputs the sanitized document.Inference uses the document, user-defined rules, and scenario as inputs, then merges post-processed chunks into the redacted document.
- Explainability and Auditability: Each redaction includes a natural-language rationale stating what the data is, why it is sensitive, and which rule or policy it violates.The system also produces a trace of redaction rules to support compliance auditing and reviewer validation.
4 Experiments and Results
The experiments evaluate the redaction system qualitatively against human ground truth and quantitatively using Reconstruction Accuracy and Redaction Recall. Results indicate precise span redaction, strong resistance to reconstruction attacks, and coverage exceeding 95% of sensitive instances.
- Validation dataset: Because no suitable open-source benchmark existed, the authors created a manually redacted validation dataset spanning diverse domains.Each data point contains the domain, user rule, original document, human-redacted content, model-redacted content, and an explanation.
- Qualitative evaluation: Subject matter experts found that only sensitive content was redacted, with sensitive and redacted spans matching in at most 99% of evaluated samples.Non-sensitive content remained intact in the redacted documents.
- Quantitative metrics: The evaluation uses Reconstruction Accuracy to measure adversarial recovery and Redaction Recall to measure adherence to human-annotated redaction requirements.Claude-sonnet-3.5 attempts to reconstruct redacted spans, while Redaction Recall measures the proportion of ground-truth redacted tokens successfully removed.
- Privacy evaluation: Reconstruction Accuracy was less than 0.1, indicating that Claude-based inference attacks recovered little sensitive information from the redacted documents.Lower reconstruction accuracy indicates stronger privacy preservation, and the result held across cross-domain document instances.
- Redaction coverage: The fine-tuned SLM redacted sensitive content in over 95% of sensitive instances in the input documents.The paper presents this result as evidence of redaction correctness and coverage for large-volume documents.
5 Summary
The solution enables user-defined redaction of both regularly and non-regularly structured sensitive content through automatically generated natural-language rules and step-by-step SLM reasoning. It supports preprocessing for ill-formatted documents, evaluates redaction correctness and coverage, and transfers to unseen domains with minimal adaptation.
- Capabilities: The solution handles non-regularly structured information, including legal clauses and medical information, beyond solutions limited to regularly structured data such as PII.Users can custom-define sensitive content such as legal clauses, medical therapy, or business contract deals.
- Method: An LLM automatically generates redaction rules, while an SLM reasons over them step-by-step to identify and redact sensitive content in documents.The SLM is trained on synthetic samples from Legal, Business, and Medical domains, mitigating inference attacks on the model.
- Evaluation: The solution preprocesses ill-formatted documents with large blanks and spaces, while reporting redaction correctness and coverage through two dedicated metrics.A customized manually redacted dataset validates testing performance, and the reconstruction metric estimates redaction coverage.
- Generalization: The SLM can redact documents from domains absent during training using minimal few-shot learning and/or fine-tuning.The solution supports on-the-fly custom definitions of free-text sensitive content.
- Practical impact: Automated redaction improves the productivity of sanitizing large documents through less manual intervention, high accuracy, and high utility.The solution is described as versatile for user-defined free-text sensitive content.