Source-linked AI summary
Masking Is Not Enough: Generative Restoration for Multimodal De-Identification in Medical AI
Shiva Shrestha, Zongxing Xie, Chen Zhao, Liran Ma, Zhipeng Cai, Honghui Xu
TL;DR
Medical image-text systems can leak PHI through both visual content and accompanying text, while existing benchmarks and de-identification methods do not jointly establish privacy and downstream utility. ClinX provides coordinated image-text sanitization using OCR-guided masking, privacy-oriented generative restoration, and progressive text de-identification, and finds that restoration-based processing reduces recoverable PHI while retaining higher utility than blackout masking. The framework remains bounded by OCR coverage and evaluations using synthetic PHI and OCR-based leakage detection.
Problem
Medical image-text data can expose PHI through images and text, but existing evaluation largely separates benchmark utility from de-identification effectiveness.
Method
ClinX combines OCR-based image PHI localization, ClinX-PRISM no-skip generative restoration with post-processing, and progressive text-side masking and rewriting.
Results
Restoration-based sanitization substantially reduces OCR-recoverable PHI while retaining higher MedVQA utility than blackout-masking methods.
Takeaways & Limitations
Privacy-preserving MedVQA should be evaluated as a joint multimodal problem rather than image redaction alone.
Takeaways & Limitations
Privacy guarantees are limited because leakage detection uses OCR-based recovery and synthetic PHI, while image masking remains bounded by OCR coverage.
Abstract
from arXiv · showhide
Medical image-text data can expose protected health information (PHI) through both visible image content as well as accompanying text, creating a barrier to privacy-preserving medical AI systems. This risk is especially prominent in multimodal systems, where images, questions, reports, and clinical context may enter training, evaluation, or inference pipelines. Existing medical vision-language benchmarks primarily emphasize task utility, while de-identification methods are often evaluated separately from downstream reasoning. We introduce ClinX, an end-to-end multimodal PHI sanitization framework for medical image-text data. ClinX detects visible identifiers with optical character recognition (OCR), constructs binary PHI masks, and applies ClinX-PRISM, a no-skip generative restoration module with privacy-oriented post-processing for burned-in identifier suppression. In parallel, text-side PHI is reduced through progressive de-identification levels: regex masking, context-aware masking, and rewrite-based sanitization. We evaluate ClinX in medical visual question answering (MedVQA), jointly measuring PHI leakage and downstream utility across image-side, text-side, and combined de-identification settings. Results show that OCR-only masking is not sufficient as a standalone solution, and restoration-based sanitization better preserves clinically relevant visual context while sharply reducing recoverable PHI.
1 Introduction
ClinX addresses PHI exposure in multimodal medical AI by sanitizing image-text data before training, evaluation, or inference. It jointly evaluates privacy leakage and downstream MedVQA utility across image-side, text-side, and combined settings.
- Multimodal medical AI connects images with questions, reports, instructions, and clinical contexts, increasing the need to protect sensitive patient information.The systems support image-grounded reasoning, report understanding, instruction following, and clinical question answering.
- ClinX sanitizes PHI-contaminated image-text pairs through coordinated image-side and text-side branches.The image branch uses OCR, PHI matching, binary masks, and generative restoration; the text branch progresses from regex masking to context-aware masking and rewriting.
- ClinX-PRISM extends image de-identification beyond OCR-only masking with mask-conditioned no-skip generative restoration and privacy-oriented post-processing.The framework targets burned-in identifiers before model training, evaluation, or inference.
- The framework jointly measures PHI leakage and answer utility in MedVQA across image-side, text-side, and combined multimodal de-identification settings.This evaluation is designed to connect privacy protection with downstream reasoning performance.
2 Background and Related Work
Medical vision-language benchmarks largely emphasize utility, while clinical de-identification and image masking have developed as more separate lines of work. Prior masking approaches can suppress identifiers but may damage visual context needed for medical reasoning, motivating restoration-based sanitization.
- Medical VQA datasets primarily evaluate answer accuracy, reasoning ability, and closed-ended question performance rather than privacy leakage.Examples include VQA-RAD, PathVQA, SLAKE, and PMC-VQA across radiology, pathology, and knowledge-enhanced image understanding.
- Clinical de-identification has traditionally addressed PHI protection through regulatory and methodological pathways separate from downstream multimodal reasoning evaluation.HIPAA-related de-identification includes Safe Harbor and Expert Determination pathways.
- OCR-based blackout masking can remove visible identifiers while disrupting nearby visual structures useful for diagnosis and interpretation.This creates a privacy–utility concern for image-text reasoning systems.
- Generative inpainting reconstructs masked regions from learned visual context, but generic restoration must additionally suppress identifiers and avoid misleading artifacts.The distinction separates ordinary image reconstruction from privacy-preserving sanitization.
3 ClinX
ClinX sanitizes multimodal medical VQA inputs by localizing image-side PHI with OCR and applying either blackout masking or privacy-oriented generative restoration. Its design limits changes to PHI regions, preserves non-masked clinical content, and explicitly measures residual leakage because OCR coverage is incomplete.
- ClinX Overview: ClinX processes PHI-contaminated image-question pairs before downstream medical VLM use, targeting both recoverable PHI suppression and preservation of answer-relevant clinical information.The pipeline addresses image and question inputs within a controlled PHI-augmented VQA setting.
- Image-Side PHI Localization: OCR detects candidate image text, which is matched against a shared PHI manifest to select identifier regions instead of removing all text-like content.The manifest contains inserted identifiers and placement metadata; matching tolerates minor OCR errors while protecting meaningful labels and descriptive text.
- Image-Side PHI Localization: OCR blackout masking preserves pixels outside the PHI mask while replacing only localized identifier regions.The baseline uses a binary mask and a constant blackout value for the selected regions.
- ClinX-PRISM Restoration: ClinX-PRISM uses a no-skip, mask-conditioned generator to restore PHI-bearing regions, then composites the generated content with the original unmasked pixels.The generated image is not used to replace the full input; generation is confined to the masked region.
- ClinX-PRISM Restoration: Mask-guided post-processing refines the composited no-skip restoration to produce the final ClinX-PRISM output.The refinement mask is resized to image resolution and feathered at its boundary.
- Image-Side Evaluation: The image-side evaluation compares raw input, OCR blackout, skip-connected restoration, and ClinX-PRISM conditions.The comparison isolates the proposed no-skip restoration and post-processing pathway from alternative image-side sanitization conditions.
4 Experiment and Evaluation
ClinX is evaluated on synthetic-PHI versions of three MedVQA datasets using joint privacy-leakage and answer-utility measurements across image and text sanitization settings. Restoration generally preserves utility better than blackout masking while substantially reducing OCR-recoverable PHI.
- Experimental setup: ClinX was evaluated on PathVQA, VQA-RAD, and SLAKE with four image conditions and four progressive text-privacy levels.The full evaluation used the Cartesian product of image and text conditions.
- Experimental setup: Utility was measured with EM, token-level F1, yes/no EM, and LLM-as-a-Judge scores, while privacy used OCR token recall and character overlap.
- Overall tradeoff: OCR blackout masking sharply reduced recoverable PHI but often matched or fell below raw performance on EM and F1.
- Overall tradeoff: Restoration conditions preserved high utility while keeping OCR leakage close to zero, with g-skip and PRISM outperforming raw and ocr on lexical metrics under stricter text privacy.
- Image-side sanitization: Restoration conditions consistently outperformed raw and ocr on EM and F1 across L0–L3, especially when text-side privacy was stricter.
- Progressive query de-identification: Utility remained relatively constant from L0 to L2 but was most affected at L3, while restoration still outperformed raw and ocr.
- OCR-based privacy protection: More than 37% token recall was observed for raw images, whereas both masking and restoration drastically reduced OCR leakage.
5 Discussion
The discussion frames PHI sanitization as a multimodal privacy–utility problem: blackout masking removes identifiers but can discard visual context, whereas restoration better balances leakage reduction and reasoning utility.
- Privacy–utility framing: OCR-blackout masking removes visible identifiers but fails to restore local visual context useful for downstream reasoning.
- Privacy–utility framing: Restoration-based sanitization reduces OCR-recoverable PHI while preserving more image evidence for downstream reasoning.
- Architecture: ClinX-PRISM remains competitive with g-skip while using a no-skip pathway that prevents direct feature reuse from PHI-contaminated input.
- Limitations: ClinX offers no complete privacy guarantees because evaluation was limited to OCR-based recovery and synthetic PHI, requiring additional audits with actual PHI patterns.
6 Conclusion
ClinX jointly sanitizes image-side and text-side identifiers for MedVQA. Across datasets, privacy settings, and evaluated medical VLMs, restoration reduced OCR-recoverable PHI while retaining higher utility than blackout masking.
- Conclusion: ClinX is a multimodal PHI sanitization framework that jointly addresses image-side and text-side identifiers in MedVQA.
- Conclusion: Across datasets, privacy settings, and evaluated medical VLMs, restoration substantially reduced OCR-recoverable PHI while consistently retaining higher utility than blackout masking.
- Conclusion: The paper concludes that privacy-preserving MedVQA should be evaluated as a joint multimodal problem rather than image redaction alone.
- Future work: Future work targets stronger privacy audits, non-text PHI, and broader clinical multimodal settings.
A.1 Dataset-Specific Utility Breakdown
The appendix separates aggregate findings into dataset-specific utility breakdowns and diagnoses residual OCR leakage in ClinX-PRISM. PRISM-flat isolates the effect of removing final post-processing from the no-skip restoration pipeline.
- Dataset-specific utility: Dataset-specific tables disaggregate utility across PathVQA, VQA-RAD, and SLAKE, text privacy levels, and image conditions.They assess whether the aggregate privacy-utility pattern holds within each benchmark.
- PRISM-flat retest: PRISM-flat retains the no-skip restoration backbone while removing the final mask-guided post-processing stage.This diagnostic separates restoration architecture from rendered-output post-processing.
- PRISM-flat retest: PRISM showed slightly higher residual OCR leakage than g-skip, but the authors do not interpret this as evidence that no-skip restoration is intrinsically less private.Faint boundary, grain, or compression-like artifacts from post-processing may be parsed as text-like structure.
- PRISM-flat retest: Under both EasyOCR and RapidOCR, PRISM-flat produced comparatively lower residual leakage than primary PRISM.PRISM-flat was evaluated only for OCR leakage, not downstream MedVQA utility, so it remained a diagnostic control.
A.3 Expanded Model-Wise Results
The expanded model-wise results report utility across all text privacy levels and image conditions. Their purpose is to test whether restoration trends remain consistent across model families and privacy levels.
- Expanded model-wise results: Table 3 covers question-side privacy levels L0 through L3 and all image conditions for each evaluated medical VLM.Each sub-table corresponds to one text privacy level.
- Expanded model-wise results: The model-wise breakdown reports exact match, token-level F1, and judge-based utility for every evaluated medical VLM.All reported metrics are utility measures in the expanded tables.
- Dataset-specific utility: Table 2 presents dataset-specific utility breakdowns across text privacy levels and image conditions, with higher values indicating better utility.The caption defines the table’s direction of interpretation.
- Condition definitions: PRISM denotes ClinX-PRISM, the proposed no-skip restoration pathway with mask-guided post-processing.The abbreviation distinguishes PRISM from OCR masking and the skip-connected restoration baseline.
B Benchmark Construction and Sanitization Details
The benchmark constructs controlled multimodal privacy stress tests by injecting shared synthetic identifiers into images and questions, then matching OCR detections to manifest-defined PHI. Text sanitization progresses from regex masking to contextual masking and rewriting.
- Synthetic PHI injection: Each sample uses a shared manifest containing synthetic names, birth dates, medical record numbers, hospital identifiers, and image-placement metadata.The manifest coordinates image and question injection and supports leakage measurement without real patient information.
- Synthetic PHI injection: The benchmark creates PHI-overlaid images from clean images by rendering synthetic identifiers into predefined placement regions and producing binary masks.Figure 2 depicts the clean image, overlaid image, and localized sensitive-region mask.
- OCR matching and masking: OCR detections are normalized and matched against manifest identifiers using string overlap and confidence thresholds before matched regions are projected into image space.A small dilation covers boundary pixels and keeps non-PHI supporting information unmasked.
- OCR matching and masking: If OCR misses a PHI token, that token is omitted from the mask, so localization is not assumed to be perfect.Residual leakage is measured after sanitization with OCR-based privacy metrics.
- Question-side sanitization: Question-side de-identification uses progressive levels: L1 regex masking, L2 context-aware masking, and L3 rewrite-based sanitization that preserves the medical question.The rewrite retains clinical findings, anatomy, modality, disease cues, and question intent while removing identity-bearing information.
C.1 ClinX-PRISM Architecture and Optimization
ClinX-PRISM restores masked image regions with a mask-conditioned no-skip generator and patch-level adversarial supervision. Training combines restoration, identity preservation, and regularization objectives, while evaluation also uses blinded clinical-equivalence judging.
- Architecture: ClinX-PRISM encodes the PHI-overlaid image while using the binary mask to condition SPADE blocks in the bottleneck and decoder.Removing skip connections forces reconstruction through the mask-conditioned bottleneck rather than direct low-level feature reuse.
- Architecture: The generated image is composited only inside the PHI mask, while pixels outside the mask are retained from the input.Primary PRISM adds mask-region post-processing; PRISM-flat omits that stage.
- Optimization: PatchGAN produces a spatial realism map that encourages locally plausible restoration in the PHI region.Both the skip-connected baseline and ClinX-PRISM use the same adversarial training setting.
- Optimization: The training objective combines adversarial supervision, masked reconstruction inside the PHI region, identity preservation outside it, and total-variation regularization.Training tuples contain the overlaid image, mask, and clean image.
- Evaluation: LLM-as-a-Judge scoring evaluates clinical consistency from the question, ground-truth answer, and model prediction without exposing image, privacy condition, LLM identity, or PHI information.Scores are reported separately for the two judges rather than averaged.
- Qualitative evaluation: Qualitative examples show reconstructed images after PHI removal while preserving other image regions; clean targets are used only for paired restoration training.The clean target is unavailable to the sanitizer during evaluation.
D.2 Limitations of the Privacy Evaluation
ClinX’s privacy evaluation has important scope boundaries: it uses synthetic PHI and OCR-dependent leakage measures rather than formal cryptographic protection or comprehensive re-identification assessment. Qualitative restoration examples also rely on clean targets used only during paired restoration training.
- ClinX provides no explicit formal cryptographic privacy protection.
- Synthetic PHI injection enables repeatable evaluation but may miss visual and linguistic variation in real medical images.
- Leakage measurement focuses on OCR-dependent token recovery, not broader re-identification risks from non-textual visual or demographic cues.
- ClinX-PRISM qualitative examples compare restored outputs with clean targets used only for paired restoration training.