PaperlayerView exact revision on arXiv

Source-linked AI summary

AutoMIA: Improved Baselines for Membership Inference Attack via Agentic Self-Exploration

Ruhao Liu, Weiqi Huang, Qi Li, Xinchao Wang

arXiv:2604.01014v1cs.CRcs.CV

TL;DR

Membership inference methods often depend on static, handcrafted heuristics that do not adapt readily across models, while automated discovery must handle noisy feedback and a large strategy space. AutoMIA addresses this gap by generating and executing logits-level strategies in a closed loop, refining them through feedback. Across multiple models and datasets, it achieves strong performance and broad applicability, while its scope is the grey-box setting.

  • Problem

    Existing membership inference attacks rely on handcrafted, task-specific heuristics and lack a unified mechanism for exploring strategies across different large models.

  • Method

    AutoMIA uses history-aware agents to generate executable logits-level attack strategies, execute them, and refine a dynamic strategy library through closed-loop evaluation feedback.

  • Results

    AutoMIA achieves strong performance across multiple vision-language models and datasets, adaptively generating strategies tailored to diverse experimental settings.

  • Takeaways & Limitations

    AutoMIA supports scalable and systematic privacy evaluation by reducing reliance on handcrafted heuristics while remaining model-agnostic.

  • Takeaways & Limitations

    The evaluation deliberately focuses on the grey-box setting, assuming access to model logits and related interfaces under favorable auditing conditions.

Abstract

from arXiv · show

Membership Inference Attacks (MIAs) serve as a fundamental auditing tool for evaluating training data leakage in machine learning models. However, existing methodologies predominantly rely on static, handcrafted heuristics that lack adaptability, often leading to suboptimal performance when transferred across different large models. In this work, we propose AutoMIA, an agentic framework that reformulates membership inference as an automated process of self-exploration and strategy evolution. Given high-level scenario specifications, AutoMIA self-explores the attack space by generating executable logits-level strategies and progressively refining them through closed-loop evaluation feedback. By decoupling abstract strategy reasoning from low-level execution, our framework enables a systematic, model-agnostic traversal of the attack search space. Extensive experiments demonstrate that AutoMIA consistently matches or outperforms state-of-the-art baselines while eliminating the need for manual feature engineering.

1. Introduction

AutoMIA reframes membership-inference strategy discovery as closed-loop self-exploration to address the limitations of handcrafted, task-specific heuristics. It generates executable logits-level attacks and refines them using evaluation feedback, with experiments showing stronger performance across multiple settings.

  • Membership inference attacks determine whether a sample was used during model training and help evaluate data privacy leakage.
  • Existing attacks rely on handcrafted statistical heuristics that are task-specific, require expert feature engineering, and lack unified strategy exploration.
  • Automating membership-inference strategy discovery is difficult because MIAs provide noisy distribution-level feedback rather than explicit success boundaries.
  • AutoMIA generates executable logits-level attack code and iteratively refines strategies using aggregated dataset-level feedback such as AUC scores.
  • AutoMIA substantially outperforms baselines across multiple evaluation tasks, achieving higher success rates and broad applicability.

2. Related Work

Prior work studies membership inference under multiple access assumptions and commonly uses handcrafted metrics or shadow models. Agentic security research has begun automating inference and privacy attacks, but the cited work positions AutoMIA as extending automation to logits-level MIA strategy discovery.

  • Membership Inference Attacks: Membership inference attacks are studied under white-box, black-box, and grey-box access assumptions.
  • Membership Inference Attacks: Metric-based attacks use handcrafted statistics such as confidence, entropy, or Min-K%, while shadow-model attacks approximate target-model behavior.
  • LLM-Based Agents and Safety: Large language model agents support autonomous planning and multi-step reasoning for complex workflows.
  • LLM-Based Agents and Safety: Prior agent-based security systems iteratively refine queries for inference attacks or conduct privacy red-teaming against training-data leakage and retrieval-augmented systems.

3. Problem Setting and Challenges

The paper formulates grey-box membership inference against a target vision-language model as binary classification from logits-level outputs. AutoMIA addresses the resulting challenge of searching a large strategy space through a closed-loop system that generates, executes, and refines candidate attacks.

  • Problem Setting: A target vision-language model maps multimodal input x = (I, Xins) to logits-level features o, with Dtrain denoting its training dataset.
  • Problem Setting: The adversary implements membership inference as binary classification, using executable strategy p to compute an inference score S = p(o).
  • Problem Setting: The detector decides membership by comparing the strategy score against a threshold, with the indicator function assigning member or non-member labels.
  • AutoMIA Framework: AutoMIA forms a closed loop in which an agent generates strategies, code execution runs attacks against target VLMs, and guidance feedback updates the strategy library.
  • Adversary’s Knowledge: The paper assumes grey-box access to model queries, tokenizer, logits, and generated text, but not training algorithms, gradients, or specific model parameters.
  • Adversary’s Knowledge: The framework deliberately focuses on grey-box attacks to explore an upper bound under favorable access conditions and support internal auditing and privacy-risk assessment.
  • Challenges: Automated attack design must navigate a vast combinatorial space of logits-level operations across heterogeneous model architectures without known discriminative features.

4. Method

AutoMIA automates membership-inference strategy discovery through iterative generation, execution, evaluation, and reflection. A dynamic strategy library and feedback-guided selection support focused exploration under noisy evaluation signals.

  • Overview: AutoMIA combines an agent, a Guidance agent, and a dynamic strategy library in a perception–reasoning–action–reflection loop.The library stores generated strategies with categorized guidance and evaluation statistics for subsequent iterations.
  • Execution and evaluation: AutoMIA aggregates the evaluation tuple into a weighted Composite Effectiveness Score to guide subsequent strategy refinement.The weights calibrate the relative importance of AUC, Accuracy, and TPR.
  • Strategy library and selection: A fixed-size sliding window exposes high- and low-quality strategies instead of the entire library, reducing redundant exploration and contextual burden.The selected subset changes as the library accumulates experience and balances promising and unsuccessful directions.
  • Strategy synthesis: Each iteration generates K high-level strategy specifications with associated logits-level runnable code for membership inference.Candidate strategies are conditioned on retrieved prior strategies and previous-round guidance.
  • Execution and evaluation: Candidates transform reusable target-model logits into per-sample membership scores, which are evaluated using AUC, Accuracy, and TPR@5%FPR.The resulting metrics form an evaluation tuple for each candidate strategy.
  • Guidance and library update: Guidance categorizes evaluated strategies and supplies feedback that conditions the next generation cycle, enabling systematic exploration of the attack space.The categorized strategies are incorporated into the library for later retrieval.

5. Experiment

AutoMIA is evaluated against handcrafted membership-inference metrics across three vision–language models and text, image, and multimodal benchmarks. It remains robust across models, text lengths, modalities, and input compositions, whereas handcrafted baselines vary substantially by setting.

  • Experimental setup: Experiments use VL-MIA/Text, VL-MIA/DALL·E, and VL-MIA/Flickr to assess membership inference across text, image, and multimodal settings.The benchmarks target instruction-tuning text, image modality, and Flickr-based multimodal conditions.
  • Experimental setup: AutoMIA is compared with representative handcrafted metrics on MiniGPT-4, LLaVA-1.5, and LLaMA-Adapter under a grey-box threat model.The agent can observe logits or confidence-related outputs but not model parameters or training data.
  • Experimental setup: Experiments use a consistent protocol across benchmarks, including ten rounds per configuration and an initially empty strategy library.The strategy library is updated after each round from evaluated candidate strategies.
  • Overall performance comparison: Across text lengths and models, AutoMIA consistently achieves near-optimal AUC and outperforms the strongest baseline by a clear margin.Handcrafted metrics show inconsistent performance as model architecture and text length change.
  • Overall performance comparison: Across image-centric and multimodal benchmarks, AutoMIA frequently achieves the best AUC across different modality compositions.Handcrafted metrics vary substantially depending on whether image, instruction, description, or combined inputs are used.
  • Overall performance comparison: Overall, AutoMIA maintains stable and competitive performance across tested scenarios while existing methods remain sensitive to architecture, modality, and evaluation setting.The reported robustness is associated with automatically exploring, evaluating, and refining attack strategies rather than using a fixed metric.

6. Ablation Study

Ablations show that AutoMIA benefits from multiple evaluation signals, iterative exploration, and guidance, while retaining robustness under stricter and held-out evaluation settings.

  • 6.1. Impact of Agent Backbone: All agents synthesize comparable high-AUC strategies under the shorter text setting, despite using different LLM backbones.The evaluated backbones are Gemini 3 Flash, Grok 4.1 Fast, Qwen3-Max, and DeepSeek-V3.2-Reasoner.
  • 6.2. Impact of Exploration Rounds: Performance improves substantially during rounds 1–5, typically peaks around round 15, and then shows diminishing returns.The reported trajectory indicates near-optimal performance within approximately 15 exploration rounds.
  • 6.3. Impact of Scoring Function Weights: The default scoring configuration balances global ROC separation with sensitivity under low-FPR constraints better than single-criterion weightings.The pattern appears in both linear- and logarithmic-FPR visualizations, while agent-generated strategies outperform handcrafted baselines across variants.
  • 6.4. Evaluation under a Near-IID Setting.: 0.240 vs. 0.216 TPR@5%FPR: the best AutoMIA strategy surpasses the strongest baseline in the stricter near-IID setting.Members and nonmembers are sampled from the same Dolma source with matched length and preprocessing to reduce distribution artifacts.
  • 6.5. Generalizability of Top Strategies: Top validation strategies generalize to the 50% hold-out test split, although performance drops moderately on unseen data.Hold-out AUCs remain above random guessing and competitive with strong static baselines.
  • 6.6. Impact of Guidance agent on Metric Exploration: Removing the Guidance Agent causes a consistent performance drop across text lengths, indicating that feedback-driven exploration contributes to discovery efficiency and stability.Without guidance, the agent searches candidate logit transformations with less directional bias.

7. Conclusion

AutoMIA reframes grey-box membership inference for vision–language models as automated strategy generation and execution. Iterative closed-loop exploration reduces reliance on handcrafted heuristics while supporting strong performance across models and datasets.

  • AutoMIA reframes grey-box membership inference against vision–language models as an automated strategy generation and execution process.
  • Iterative exploration, evaluation, and refinement of logits-level attack strategies reduces reliance on handcrafted heuristics and remains model-agnostic.
  • Experiments across multiple vision–language models and datasets show adaptive strategy generation with strong performance across diverse conditions.

A. Additional Experimental Results

The appendix supplements AUC with accuracy and low-FPR sensitivity metrics to evaluate membership-inference performance under different operating conditions.

  • AUC provides the main threshold-independent measure of membership-inference discriminative power.
  • Accuracy measures overall member-versus-nonmember correctness at an optimal threshold selected by Youden’s J statistic.
  • TPR@5%FPR measures attack sensitivity when false alarms are restricted to 5%, targeting high-confidence operating conditions.
  • The supplementary metrics are reported for both text-based and multimodal benchmarks.

A.1. Results on Text-Based Benchmarks

Additional text-benchmark results compare accuracy and TPR@5%FPR across three vision–language models and two text lengths. AutoMIA maintains robust performance where handcrafted baselines vary substantially.

  • Tables 6 and 7 compare accuracy and TPR@5%FPR across LLaVA, MiniGPT-4, and LLaMA-Adapter at text lengths 32 and 64.
  • AutoMIA consistently maintains high performance, while Perplexity and Min-k% Prob exhibit significant volatility across models and text lengths.

A.2. Results on Multimodal Benchmarks

The multimodal evaluation uses Flickr and DALL·E benchmarks across input modalities and reports Accuracy and TPR@5%FPR comparisons. The section also specifies token-level metric generation, implementation constraints, and feedback indicators.

  • Flickr benchmark: Flickr evaluation compares image, instruction, description, and instruction-plus-description modalities on vision–language models.Tables 8 and 9 report performance across these input compositions.
  • DALL·E benchmark: DALL·E evaluation measures Accuracy and TPR@5%FPR for image, instruction, description, and combined instruction-plus-description inputs.The benchmark uses synthetic non-member images generated from BLIP captions.
  • Evaluation findings: Handcrafted metrics vary substantially across models and modalities, with no single baseline generalizing consistently across heterogeneous multimodal settings.The reported indicators are AUC, Accuracy, and TPR@5%FPR, with higher values preferred.
  • Metric generation: AutoMIA generates metrics from token-level probabilities, log probabilities, and token ids while requiring efficient, numerically stable implementations.The specification forbids nested token loops, sorting, ranking, and unnecessary recomputation.
  • Metric requirements: Generated metrics must distinguish members from non-members and provide reasonable statistical meaning while following the required executable-code format.Existing system metrics include perplexity, modified entropy, and Rényi entropy, which should not be recreated.

C. Example for strategy library

The strategy-library example contrasts a gradient-based metric that consistently performs well with a geometric-spread metric that fails to discriminate membership effectively.

  • Strategy 1: log probability gradient field helicity: Strategy 1 measures second-order structure in true-token log-probability gradients and consistently outperforms alternative metrics.Its results suggest higher-order gradient properties capture memorization patterns absent from non-member samples.
  • Strategy 2: token distribution geometric spread: Strategy 2 computes geometric probability-mass dispersion as a measure of overall token-distribution uncertainty.The strategy is categorized as weak in the strategy library.
  • Strategy 2: token distribution geometric spread: The geometric-spread strategy performs worse than random guessing and lacks meaningful low-FPR discrimination.Its coarse geometric signal does not reliably correlate with membership, limiting entropy-like global uncertainty measures in this setting.

D. Why the Discovered Metrics Capture Memorization Rather than Spurious Correlations

AutoMIA’s discovered metrics are presented as mathematically inspectable and are tested in a controlled logit-level simulation. The results indicate that top metrics respond to structured memorization signals rather than superficial benchmark correlations.

  • Mathematical interpretability: AutoMIA produces explicit executable formulas, allowing direct inspection of their consistency with memorization intuitions.The paper illustrates this interpretability with the Avg true max log gap metric.
  • Mathematical interpretability: Avg true max log gap measures the average positive log-probability gap between the most confident prediction and the ground-truth token.Lower values correspond to stronger memorization signals because members more often receive their highest probability on the true token.
  • Targeted mathematical simulation: The controlled simulation boosts the ground-truth logit for synthetic members while sampling non-member logits from a standard Gaussian distribution.Softmax probabilities are then used to evaluate the discovered metrics.
  • Targeted mathematical simulation: AUC = 0.915, Cohen’s d = −1.97, and p < 0.001 for avg true max log gap, which clearly separates the synthetic member and non-member distributions.Members receive significantly lower scores under this construction.
  • Targeted mathematical simulation: Theory and controlled simulation together support the interpretation that the discovered formulas capture statistically meaningful and mechanistically interpretable memorization signatures.The paper reports similar separability for other top-ranked discovered metrics.
Loading 2604.01014v1…