PaperlayerView exact revision on arXiv

Source-linked AI summary

ASIRF: An Agentic Framework for Context-Dependent Sensitive Information Redaction

Sudha Priyadarshini, Mohamed Chahine Ghanem

arXiv:2609.29191v1cs.AIcs.IRcs.LG

TL;DR

Sensitive information depends on domain and intent, but trained detectors fix their taxonomy at training time, making new categories costly to add. ASIRF retrieves domain-specific definitions at inference time; its recall exceeds OPF’s in 68 of 80 model–domain combinations (85%).

  • Problem

    Sensitivity varies by domain and intent, while trained detectors fix their taxonomy at training time and require new labeled data and retraining to add domains or attribute types.

    Source: §1 · p. 1

  • Method

    ASIRF classifies an input’s domain, retrieves relevant sensitive-attribute definitions from a knowledge base, and extracts matching values.

    Source: §3 · p. 2

  • Results

    ASIRF exceeds OPF’s recall in 68 of 80 model–domain combinations (85%), by at least one architecture.

    Source: §4.2.1 · p. 3

  • Takeaways & Limitations

    For categories absent from training corpora, a knowledge base can be given a definition, whereas a fixed classifier cannot recognize them without expensive training.

    Source: §4.2.2 · p. 4

    ASIRF-Multi and ASIRF-Single suit different domains and model families rather than either winning uniformly.

    Source: §5 · p. 5

Abstract

from arXiv · show

Sensitive information is defined by domain and intent, not a universal category, yet redaction systems such as privacy filters and named-entity recognizers fix a taxonomy at training time, requiring retraining for each new domain. We introduce ASIRF (Agentic Sensitive Information Redaction Framework), which retrieves domain-specific definitions based on the input's domain from a flexible knowledge base at inference time, needing no retraining to adapt. Two architectures, a three-call multi-agent pipeline and a single-agent variant, are evaluated across ten small open-weight models and eight datasets, including out-of-distribution fictional domains, against the OpenAI Privacy Filter (OPF) as a trained-classifier baseline. With only a few dozen expert-authored definitions per domain and no training data, ASIRF's recall exceeds OPF's in 68 of 80 model-domain combinations (85 percent), by at least one of the two architectures, with shortfalls confined mostly to OPF's training-distribution domains.

1 Introduction

Sensitive information depends on domain and intent, but trained detectors fix their taxonomy at training time, making new domains or attribute types costly to add. The paper evaluates whether an inference-time agentic framework can improve domain-specific filtering and generalize to unseen definitions and fictional domains.

1 Introduction

  • Sensitivity depends on domain and intent, while adding a domain or attribute type to trained detectors requires new labeled data and retraining.ASIRF instead puts the taxonomy in an inference-time knowledge base, which can be adapted through a knowledge-base edit without labeled examples or a training run.Source: §1 · p. 1
  • RQ1 asks whether a test-time agentic framework combining an agentic harness with domain-conditioned retrieval improves domain-specific filtering over training-centric approaches.The comparison includes ASIRF’s multi-agent and single-agent architectures against OPF [1], as well as a retrieval-removal ablation.Source: §1 · p. 1
  • RQ2 asks how well the framework generalizes to out-of-distribution domains and previously unseen sensitive-attribute definitions.The evaluation addresses this question using two out-of-distribution fictional domains.Source: §1 · p. 1

2 Related Work

Prior sensitive-information detectors generally use fixed entity lists or label schemas, while retrieval-augmented generation supplies information at inference time. ASIRF combines retrieval with domain-conditioned routing for sensitive-information filtering.

2 Related Work

  • Trained detectors use fixed entity lists or label schemas, and adding categories requires new labeled data and retraining.Examples include OPF [1], Presidio-style recognizers, GLiNER [3], and clinical taggers; retrieval-augmented generation instead grounds outputs in documents fetched at inference time [5].Source: §2 · p. 2
  • ASIRF composes retrieval and domain-conditioned routing into a pipeline for sensitive-information filtering.It applies inference-time retrieval to entity definitions, making the sensitivity taxonomy an inference-time input.Source: §2 · p. 2
  • ASIRF-Multi uses staged, tool-using agent calls, while ASIRF-Single tests whether the benefit persists when that structure is collapsed [6] [7] [8].The cited work reports that decomposing tasks into staged, tool-using calls improves multi-step reasoning.Source: §2 · p. 2

3 Methodology

ASIRF classifies each input’s domain, retrieves relevant sensitive-attribute definitions, and extracts matching values; its knowledge base supports domain additions without retraining. The two architectures are evaluated across open-weight models, with recall reported by model and domain.

3 Methodology

  • ASIRF classifies the domain, retrieves relevant sensitive-attribute definitions, and extracts matching values.ASIRF-Multi assigns these steps to three chained agents; ASIRF-Single performs them in one call while retaining retrieval access.Source: §3 · p. 2
  • Both architectures are evaluated across ten open-weight models from three families, spanning 1B–14B parameters and two inference providers.The models are Gemma-3 [12], Qwen3 [13], and Ministral [14].Source: §3 · p. 2
  • New domains are onboarded by adding knowledge-base entries instead of retraining.The knowledge base includes cross-domain entity types, statutory-text entries, and dataset-specific entries, including two fictional domains.Source: §3 · p. 2
  • For both ASIRF architectures, the top-performing model exceeds OPF on every dataset.Table 1 reports ASIRF-Multi recall by model and domain; Table 2 reports the same for ASIRF-Single.Source: Table 1 · p. 3, Table 2 · p. 3

4 Evaluation

The evaluation compares ASIRF with OPF across eight datasets, using token-level fuzzy overlap and emphasizing recall. ASIRF’s recall is higher in most model–domain combinations, including every model on the fictional domains, while an ablation tests retrieval’s contribution.

4.1 Experimental setup

  • The evaluation covers eight datasets, including three synthetic real-world domains, two fictional domains, and three established privacy datasets; recall is foregrounded.Precision, recall, and F1 use token-level fuzzy overlap rather than exact span matching.Source: §4.1 · p. 3

4.2.1 ASIRF vs. OPF

  • 68 of 80 model–domain combinations (85%) show higher recall for at least one ASIRF architecture than for OPF; shortfalls concentrate on pii-masking and SPY.Only the largest models close the gap on those two datasets.Source: §4.2.1 · pp. 3–4, Figure 2 · p. 4

4.2.2 Out-of-distribution generalization

  • Every model under both architectures exceeds OPF’s recall on the two fictional domains, including Gemma-3-1B, the weakest model elsewhere.These domains use invented categories that could not have appeared in a training corpus.Source: §4.2.2 · p. 4

4.3 Ablation

  • Removing the Entity Retriever from ASIRF-Multi confirms that retrieval improves recall across all three tested models on Archive of Forgotten Futures.The ablation uses a scoped grid on the custom dataset.Source: §4.3 · p. 4, Table 3 · p. 4

5 Conclusion

ASIRF achieves higher recall than OPF in most model–domain combinations with a small fraction of a trained detector’s preparation cost, but its LLM calls cost more at inference. Neither ASIRF architecture wins uniformly, and retrieval improves recall across models.

5 Conclusion

  • ASIRF’s recall exceeds OPF’s in 85% of model–domain combinations, using a small fraction of a trained detector’s preparation cost.Its LLM calls nevertheless cost more at inference than OPF’s single lightweight forward pass.Source: §5 · p. 4
  • ASIRF-Multi and ASIRF-Single suit different domains and model families rather than one architecture winning uniformly; knowledge-base retrieval improves recall across models.Future work includes improving knowledge-base curation, including a web-search fallback to drive new-domain preparation time toward zero.Source: §5 · p. 5

A.1 Precision and F1

The appendix provides architecture-specific precision and F1 tables, each organized by model and domain. The supplied passages identify the tables but do not report their cell values.

  • Table 4 reports ASIRF-Multi precision by model and domain.Source: Table 4 · p. 6
  • Table 5 reports ASIRF-Single precision by model and domain.Source: Table 5 · p. 6
  • Table 6 reports ASIRF-Multi F1 by model and domain.Source: Table 6 · p. 7
  • Table 7 reports ASIRF-Single F1 by model and domain.Source: Table 7 · p. 7

A.2 Compute resources

Inference used paid hosted APIs rather than self-hosted GPU compute; local work was limited to lightweight orchestration, retrieval, and scoring. Processing time varied by dataset, with one representative model averaging 5 s per row.

  • Model inference ran through paid-tier Amazon Bedrock and Hugging Face APIs, not self-hosted GPU compute.Source: §A.2 · p. 7
  • Local compute handled lightweight orchestration, ChromaDB retrieval, and scoring without GPU acceleration.Source: §A.2 · p. 7
  • Processing time varied with dataset input length and reasoning complexity; Qwen3-14B Single-Agent averaged 5 s/row.Source: §A.2 · p. 7
Loading 2609.29191v1…