PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaptPrint: Response-Adaptive Fingerprinting of Black-Box LLM Services

Yilin Li, Yifei Zhang, Guozhu Meng

arXiv:2608.22213v1cs.CR

TL;DR

Black-box LLM fingerprinting is challenged by opaque services and complex, variable configurations that weaken fixed-query approaches. AdaptPrint uses response-adaptive consistency probing and similarity matching, significantly outperforming state-of-the-art methods while remaining robust across defenses and decoding parameters.

  • Problem

    Complex system prompts and decoding parameters can make fixed-query fingerprinting unreliable for identifying models in black-box LLM services.

  • Method

    AdaptPrint adaptively generates probes anchored in target responses and combines Direct, Continuation, and Follow-up Probing for progressive response matching.

  • Results

    AdaptPrint significantly outperforms state-of-the-art methods and demonstrates strong robustness across different defense strategies and decoding parameters.

  • Takeaways & Limitations

    AdaptPrint supports practical security-risk assessment and copyright auditing by revealing hidden model identities in black-box LLM services.

  • Takeaways & Limitations

    AdaptPrint is affected by probe question quality and assumes a stable underlying model within each probing session.

Abstract

from arXiv · show

Black-box LLM services have emerged as a practical deployment paradigm. Nevertheless, their opacity also hinders the systematic assessment of security risks and complicates copyright auditing for model owners. Black-box LLM fingerprinting, which identifies the underlying LLM identity through query-response interactions, offers a promising way to bridge this gap. Existing approaches typically collect responses from target LLM services using a fixed set of queries and perform poorly in the presence of realistic and complex configurations (e.g., system prompt and sampling settings). To overcome these limitations, we propose AdaptPrint, a response-adaptive fingerprinting method for revealing hidden LLM identities in black-box LLM services. AdaptPrint integrates three progressive response consistency probing strategies: Direct Probing, Continuation Probing, and Follow-up Probing. AdaptPrint determines the final LLM identity by performing similarity matching among candidate LLMs. Experimental results show that AdaptPrint significantly outperforms state-of-the-art methods among 27 candidate models, achieving Top-1, Top-3, and Top-5 accuracies of 80.6%, 90.3%, and 92.1%. AdaptPrint also demonstrates strong robustness across different defense strategies and decoding parameters.

I. INTRODUCTION

Black-box LLM services expose only query–response interfaces, making their underlying models opaque for security assessment and copyright auditing. AdaptPrint addresses limitations of fixed-query fingerprinting with response-adaptive probing and similarity matching.

  • Problem: Black-box LLM services let users interact through interfaces without revealing the exact underlying model identity.Such services may self-host models or wrap responses from upstream providers.
  • Problem: Fixed-query fingerprinting struggles with service-specific guardrails and complex configurations such as system prompts, sampling settings, and prompting strategies.These factors can alter outputs and reduce fingerprinting effectiveness.
  • Method: AdaptPrint adaptively generates benign, domain-relevant probes and uses target responses as anchors for fingerprint analysis.This response-adaptive perspective replaces fixed predefined queries as the starting point.
  • Method: AdaptPrint combines Direct Probing, Continuation Probing, and Follow-up Probing for progressive, multi-dimensional response matching.It aggregates evidence from these strategies to match the target against candidate models.
  • Evaluation: AdaptPrint is evaluated across diverse system prompts, sampling parameters, prompting strategies, and LLMs, with reported robustness across scenarios.The paper positions these evaluations as evidence of reliability and general applicability.

C. Motivation

Prior black-box fingerprinting methods rely on predefined queries and universal configurations that may conflict with service domains, guardrails, or variable settings. The motivation for AdaptPrint is to use informative, domain-relevant probes and responses produced under the target service’s current configuration.

  • Query limitations: Predefined queries may fail when they conflict with a target service’s domain, safety guardrails, or intent-detection mechanisms.Code-completion or injection prompts can trigger refusal behavior instead of useful responses.
  • Configuration limitations: Universal base-configuration comparisons become questionable when system prompts and decoding parameters substantially influence generated responses.The target’s observable behavior may reflect configuration-dependent information beyond the user query.
  • Empirical motivation: Experiments compare four representative models across system prompts and temperature/top-p settings using average negative log-likelihood per token.The setup collects 50 responses per model from five questions and ten decoding configurations.
  • Design motivation: AdaptPrint aims to reduce the impact of complex and variable service configurations through response-adaptive fingerprinting.Its design follows the principle of matching behavior under the target service’s current setting.

III. APPROACH

AdaptPrint identifies a hidden LLM by using the target service’s responses to construct probes, compare candidate responses, and aggregate consistency evidence across multiple strategies.

  • AdaptPrint comprises initial question generation, candidate model pool construction, multi-strategy fingerprint collection, and fingerprint matching.
  • The method models the target service as llm_black and compares candidate responses generated under corresponding response-conditioned probes.
  • AdaptPrint uses the target response as a reference because unknown service configurations can strongly affect responses to straightforward fixed-query comparisons.
  • Fingerprint identification selects the candidate whose responses are most consistent with black-box responses across multiple probing strategies.

B. Initial Question Generation

AdaptPrint generates domain-relevant questions designed to elicit stable, non-trivial explanatory structures, then standardizes candidate-side conditions before collecting adaptive fingerprints.

  • Initial Question Generation: AdaptPrint generates low-entropy, non-trivial questions tailored to the black-box service’s task domain.
  • Initial Question Generation: Open-ended questions can yield divergent outputs from the same model, whereas trivial factual questions can produce highly similar responses across models.
  • Initial Question Generation: An auxiliary LLM guides questions toward a single stable explanatory backbone, such as a process, constraint-resolution path, or decision dependency path.
  • Candidate Model Pool Construction: Candidate models are organized into a comparison pool using known identities rather than attempting to reproduce the black-box service’s hidden system prompt.
  • Multi-Strategy Fingerprint Collection: AdaptPrint collects fingerprints through Direct Probing, Continuation Probing, and Follow-up Probing.
  • Continuation Probing: Continuation probing uses positions where the response’s second half can serve as a continuation of its first half, improving cross-model comparability.
  • Continuation Probing: Continuation starting points are selected by scoring token positions with a local LLM, aggregating scores over sliding windows, ranking positions, and clustering them into three groups.

E. Fingerprint Matching

AdaptPrint matches fingerprints by combining semantic and textual response similarities across three probing strategies, while weighting continuation evidence from multiple starting points.

  • Similarity Measures: AdaptPrint compares each candidate model with the black-box model using semantic similarity and textual similarity.
  • Similarity Measures: Semantic similarity uses embedding vectors and cosine similarity to measure high-level meaning closeness.
  • Similarity Measures: Textual similarity uses n-gram Jaccard similarity to capture local contiguous patterns, lexical choices, and phrase-level preferences.
  • Score Aggregation: For each strategy, AdaptPrint computes candidate scores from both semantic and textual similarities and aggregates evidence into a final identity score.
  • Score Aggregation: Continuation Probing evaluates three starting-point branches and retains the top two scores to reduce the influence of uninformative or unstable branches.

IV. EVALUATION

The evaluation examines AdaptPrint across model selections, service settings, defense strategies, ablations, and applications using a candidate-model matching algorithm.

  • Research Questions: The evaluation asks whether AdaptPrint remains effective across models, service settings, and security defenses.
  • Research Questions: It also studies the contribution of each AdaptPrint component and its performance in downstream and real-world scenarios.
  • Experimental Setup: Experiments use 12 representative black-box targets from diverse model families, while all evaluated models form the candidate pool.
  • Fingerprint Matching: Algorithm 2 initializes candidate scores, evaluates Direct, Continuation, and Follow-up strategies, and returns the model with the highest aggregated score.
  • Fingerprint Matching: For Continuation Probing, the algorithm selects the two highest scores among three branches before adding their weighted contribution.
  • Experimental Setup: The study includes diverse system prompts, sampling parameters, prompting strategies, and LLMs.

B. RQ1: Effectiveness and Robustness

AdaptPrint is evaluated against black-box fingerprinting baselines across diverse models, prompts, defenses, decoding settings, and open-set conditions. It consistently outperforms baselines, while revealing trade-offs from similarity choices and rejection thresholds.

  • Baseline comparison: AdaptPrint consistently outperforms MET, LLMmap, and ZeroPrint on Top-1, Top-3, and Top-5 identification accuracy.ZeroPrint reaches 48.1%, 69.0%, and 76.9% on Method-Gen and 78.2%, 84.7%, and 88.9% on Method-All; AdaptPrint improves Top-1 accuracy by 32.5 and 8.8 percentage points, respectively.
  • Generalization: All evaluated LLMs except mimo-v2.5 correctly solve at least 10 of 18 instances, while all service configurations solve at least 8 of 12.Four LLMs achieve perfect identification accuracy; mimo-v2.5 failures are associated with verbosity-induced mismatch under Jaccard similarity.
  • Similarity analysis: 13 of 18 mimo-v2.5 cases are identified with target-side containment, but uniform use lowers Top-1, Top-3, and Top-5 accuracy to 32.4%, 64.8%, and 81.5%.Containment better accommodates longer responses, whereas Jaccard is more broadly applicable across models.
  • Defense robustness: Only 3 of 54 AdaptPrint initial questions are off-topic, and about 1% of responses are flagged for potential safety risks.Most safety flags arise from medical and fitness domains rather than explicitly harmful content.
  • Open-set performance: Increasing the rejection threshold raises KAP and URR but lowers KCAR, with τ = 0.01 and τ = 0.0125 reported as balanced settings.Higher thresholds improve accepted-prediction reliability and unknown-target rejection while removing more correct known-target predictions.

C. RQ2: Ablation Study

The ablation study examines AdaptPrint’s probing strategies, similarity components, and weighting choices. Results indicate that the strategies are complementary, while combining semantic and textual similarity provides competitive performance.

  • Ablation of Probing Strategies: AdaptPrint’s probing strategies provide complementary support and jointly enhance fingerprinting effectiveness.The study decomposes AdaptPrint-Triple into individual strategies and pairwise combinations.
  • Embedding Models and Metrics: Different embedding models achieve comparable performance for semantic similarity extraction.The evaluation compares multiple embedding models for extracting semantic representations.
  • Embedding Models and Metrics: Jaccard similarity performs better than edit distance and longest common substring for textual similarity measurement.Jaccard focuses on overlap of local textual units rather than strict sequence-level alignment.
  • Impact of Weighting Ratios: Combining semantic and textual similarities consistently achieves competitive results, whereas using either measure alone performs relatively worse.The weighting ablation evaluates different ratios between the two similarity measures.
  • Impact of Weighting Ratios: Equal weights of 0.5 are adopted as the default because they provide a general effective setting without additional parameter tuning.The authors state that equal weighting is not intended to be optimal.

D. RQ3: Applications

The application study demonstrates AdaptPrint on a publicly disclosed Python Expert Chatbot and evaluates whether it can identify fine-tuned models through their source models. The results support its feasibility for both real-world service tracing and fine-tuned-model identification.

  • Case Study: AdaptPrint is demonstrated on a Python Expert Chatbot hosted on POE whose disclosed underlying model is gpt-4o-mini.The disclosed identity provides a reliable ground truth while other configuration details remain black-box.
  • Fine-tuned LLMs: AdaptPrint correctly identifies 8, 9, and 11 models at Top-1, Top-3, and Top-5, respectively, for fine-tuned LLMs.These results demonstrate tracing fine-tuned models back to their source models.

V. DISCUSSION

The discussion positions AdaptPrint as a practical black-box fingerprinting method with security and copyright-auditing applications. It also identifies probe quality, dynamic routing, and query cost as important boundaries for deployment.

  • Limitations and Future Directions: Different probe questions can produce different identification results for the same service and setting, making informative and discriminative probes important.The paper suggests systematic probe optimization, including reinforcement learning for targeted probe generation.
  • Limitations and Future Directions: Dynamic routing across sub-LLMs can reduce fingerprinting reliability because AdaptPrint assumes a stable underlying model within each probing session.Two consecutive topic-consistent queries may be more likely to reach the same sub-LLM when routing depends on semantics or task type.
  • Security Implications: Knowing a service’s underlying model can facilitate targeted attacks such as adaptive jailbreak-prompt optimization.The paper connects model identity exposure with security-risk assessment and service hardening.
  • Copyright Auditing: AdaptPrint can support model copyright auditing and intellectual-property protection for model owners.The discussion identifies copyright auditing as a practical application of black-box fingerprinting.
  • Black-box Setting: Black-box fingerprinting relies on query–response interactions because model parameters and internal representations are unavailable.This distinguishes the black-box setting from white-box fingerprinting.
  • Scope and Contribution: AdaptPrint targets hidden LLM identity identification in black-box services using response-adaptive fingerprinting and three response-consistency probing strategies.The strategies are Direct Probing, Continuation Probing, and Follow-up Probing.

APPENDIX

The appendix specifies prompt templates for generating initial queries, continuing truncated answers, and producing follow-up questions. These templates emphasize realistic, domain-relevant, context-preserving interactions rather than benchmark-like prompts.

  • Initial Question Generation: Initial-question generation asks for one natural user request tailored to the target task domain.The request should resemble something a real user would ask an AI assistant.
  • Initial Question Generation: Initial queries should include 3–5 contextual details, at least one non-obvious constraint or risk boundary, and a practical judgment request.The prompt also requires the question to be self-contained and answerable without clarification.
  • Initial Question Generation: The initial-question template requires returning only the final user question.No additional explanation is requested.
  • Continuation Probing: Continuation prompting instructs the model to resume a cut-off answer in the same context without repeating prior content or asking for clarification.It also preserves the answer’s language, tone, and formatting style.
  • Follow-up Probing: Follow-up-question generation produces one concise, natural question grounded in a concrete point from the preceding answer.The follow-up should continue the same scenario and request practical clarification, an action, a trade-off, or an edge-case application.
  • Follow-up Probing: The follow-up template requires exactly one question without explanations, prefixes, or quotation marks.These output constraints keep the generated interaction focused.

B. CoT and RAG Prompt Templates

The paper provides prompt templates for Chain-of-Thought and Retrieval-Augmented Generation, alongside a generic system prompt and example initial questions. It also reports that average candidate rank by target model is shown in Figure 10.

  • B. CoT and RAG Prompt Templates: The appendix introduces prompt templates for Chain-of-Thought and Retrieval-Augmented Generation.
  • B. CoT and RAG Prompt Templates: Fitness Tracker, Travel Planning, and Tutoring Assistance prompts request step-by-step explanations, while other examples include contextual details.
  • B. CoT and RAG Prompt Templates: The Retrieval-Augmented Generation template instructs the model to answer based on provided context.
  • C. Generic System Prompt: The generic system prompt frames the assistant as helpful, honest, and efficient, emphasizing clear, accurate, concise, practical, and step-by-step responses.
  • 1) Initial Question Examples:: Initial question examples include shopping and pet-care scenarios with detailed constraints, preferences, and decision-making concerns.
  • 1) Initial Question Examples:: The shopping example compares two espresso machines for different coffee preferences, limited counter space, rapid heating, and long-term usefulness.
  • 1) Initial Question Examples:: The pet-care example asks how to prioritize veterinary care for a cat urinating outside its litter box after the owner’s work schedule changed.
  • E. Candidate Model Rank: Figure 10 shows the average candidate rank by target model.
Loading 2608.22213v1…