Source-linked AI summary
AISPA: User-Centric System Prompt Auditing for Large Language Model Applications
Xiangning Lin, Shenzhe Zhu, Shu Yang, Zhenyu Zhang, Haoqian Zhang, Yipeng Zhao, Chengxuan Qian, Tianwei Wang, Ziheng Zhang, Zhenlong Yuan, Dingcheng Wang, Juncheng Wu, Yuan Si, Jiaxin Liu, Baolong Bi, Robert Mahari, Tobin South, Dazza Greenwood, Zexue He, Rishi Bommasani, Sophia Kazinnik, Andreas Haupt, Samuele Marro, Erik Brynjolfsson, Alex Pentland, Jiaxin Pei
TL;DR
System prompts shape deployed AI behavior but remain largely invisible to users and regulators. AISPA audits them across eight user-relevant dimensions, finding that protective instructions are increasingly common while comprehensive coverage remains rare and problematic instructions persist.
Problem
System prompts shape AI products’ behavior and user-interest priorities, yet their hidden, developer-authored instructions remain insufficiently audited despite their central role in deployed systems.
Method
AISPA combines an eight-dimension taxonomy with a traceable human-in-the-loop workflow for auditing non-core system-prompt instructions across 88 commercial AI products.
Results
98.9% of products contain at least one protective instruction, but only 23.9% cover all eight dimensions, while roughly 40% contain at least one problematic instruction.
Takeaways & Limitations
AISPA provides a foundation for greater transparency and accountability in commercial AI system prompts.
Takeaways & Limitations
Because the corpus consists of leaked or community-disclosed prompts, findings represent a snapshot and may not reflect the exact versions currently deployed in production.
Abstract
from arXiv · showhide
System prompts are instructions configured by developers to govern the behaviors of foundation models in AI applications. They are used throughout commercial AI products, but are rarely disclosed to the public or regulators, creating a serious trust and accountability gap in the wide deployment of AI systems. In this paper, we introduce Artificial Intelligence System Prompt Assurance (AISPA), a user-centric framework for systematically auditing system prompts in AI systems. AISPA examines specific parts of a system prompt and evaluates them along eight dimensions that matter to users. We then use this framework to review 3,249 instructions from system prompts in 88 commercial AI products, classifying each instruction as either protective (of users) or problematic. Our audit surfaces four core findings. First, system prompt design varies substantially across products and developers, with some organizations averaging over 60 protective instructions per product while others average fewer than 5. Second, protective instructions are widely adopted but shallow in scope: 98.9% of products contain at least one, yet only 24% cover all eight dimensions of the AISPA taxonomy. Third, system prompts have grown steadily longer and more protective of users, suggesting that user protection is becoming a more visible concern in commercial prompt design. Fourth, despite this progress, problematic instructions remain pervasive: roughly 40% of products contain at least one instruction that works against user interests, and protective and problematic instructions frequently coexist within the same prompt. Our findings highlight the need for greater transparency, standardization, and independent oversight for system prompts in commercial AI products.
1 Introduction
System prompts shape LLM behavior before user interaction by defining personas, scope, boundaries, responses, refusals, and whose interests to prioritize. AISPA audits prompts from 88 real-world AI products and identifies substantial growth in prompt length and user protection from 2024 to 2026.
- System prompts: System prompts are developer-authored instructions that shape model behavior before any user interaction begins.They define the model’s persona, scope, and operational boundaries.
- System prompts: They specify what models should say, refuse, and whose interests they should prioritize.These instructions govern core behavioral choices across deployed LLM-powered products.
- Audit scope: 88 real-world AI products were audited across general-purpose chatbots, coding assistants, autonomous agents, search and research tools, and specialized applications.The audit used a human-LLM collaboration pipeline and expert review.
- Audit findings: From 2024 to 2026, system prompts grew substantially longer and more user-protective.The audit identifies this as an important trend in commercial prompt design.
2 What is a System Prompt and Why We Need System Prompt Auditing
System prompts are persistent, invisible instructions that shape a model’s persona, scope, tone, handling of sensitive requests, and priorities across conversations. Their influence on user-facing behavior—and evidence that prompt-level safeguards could reduce harms—motivates systematic auditing.
- System prompt definition: System prompts are predefined instructions embedded before user interaction that persist across conversations and remain invisible to end users.They differ from user prompts, which people type at runtime.
- System prompt definition: They define the model’s persona, scope, behavioral constraints, tone, sensitive-request handling, and prioritized goals.The passage identifies system prompts as the primary mechanism through which developers shape model behavior.
- Why auditing is needed: Explicit prompt-level guidance for vulnerable users, factual grounding, and restricting off-topic behavior could have reduced harm in every cited case.The section links these protective practices to concerns about harms arising from deployed AI products.
- Why auditing is needed: Developers may deliberately craft system prompts to prioritize engagement over user safety, strengthening the case for independent auditing.The passage describes growing evidence of this practice and references leaked internal guidelines concerning chatbot personas and children.
3 AISPA: A Taxonomy for User-Centric System Prompt Auditing
AISPA is a user-centric framework for evaluating system prompts in LLM applications, designed to protect users from harms rather than primarily defend systems against external adversaries. Its eight-dimensional taxonomy evaluates protective and problematic instructions within one unified framework.
- Framework motivation: AISPA evaluates system prompts from a user-protection perspective, contrasting with conventional safety approaches focused primarily on system security and adversarial red-teaming.The framework targets harms that may affect users, rather than only attacks manipulating model behavior through harmful external inputs.
- AISPA taxonomy: Eight auditing dimensions target distinct aspects of responsible system prompt design.Each dimension is intended to capture a separate aspect of prompt behavior relevant to auditing.
- Unified evaluation: The taxonomy evaluates protective and problematic instructions along the same axis, eliminating the need for separate safeguard and harmful-behavior taxonomies.Identity Transparency, for example, includes both explicit disclosure that a system is AI and deliberate concealment of its AI nature.
- Auditing dimensions: Privacy & Data Protection assesses unnecessary personal-data collection, retention, or disclosure and transparency about data use.The dimension is linked to UDHR Articles 12 and 3 and motivated by documented privacy risks in LLM systems.
- Auditing dimensions: Unsafe Request Handling assesses whether prompts direct models to refuse harmful, illicit, or jailbreak-style requests rather than comply unconditionally.The dimension is grounded in UDHR Articles 29(2) and 30.
4 Human-in-the-loop Workflow for System Prompt Auditing
AISPA formalizes a traceable, operationally rigorous workflow for auditing third-party system prompts. The workflow analyzes prompt spans using shared protective/problematic criteria and combines LLM scalability with progressively stricter human review.
- 4 Human-in-the-loop Workflow for System Prompt Auditing: AISPA specifies how third-party prompt audits can be conducted in a traceable and operationally rigorous manner.The framework extends beyond identifying what should be audited to defining how audits are carried out.
- 4.1 Auditing Guidelines: Prompt spans, typically individual sentences conveying self-contained instructions, are the basic unit of analysis rather than entire prompts.Consecutive sentences expressing the same intent may be merged into one span.
- 4.1 Auditing Guidelines: Auditing targets non-core logic spans and supplementary clauses attached to core logic, excluding pure core logic instructions.This scope distinguishes essential product functionality from additional behavioral directives.
- 4.1 Auditing Guidelines: +1 Protective spans promote transparency, safety, honesty, privacy protection, fairness, or user respect.For example, a span instructing the system to protect user data and limit retention receives a protective label.
- 4.1 Auditing Guidelines: -1 Problematic spans encourage deception, unsafe behavior, privacy invasion, manipulation, bias, or content harm.Spans irrelevant to a dimension receive no label, while the same eight dimensions are used to assess both safeguards and harmful behaviors.
- 4.2 Auditing Pipeline: The three-round protocol combines LLM-based scalability with human judgment by progressively narrowing candidate spans and raising evidentiary standards.The LLM generates broad, provisional candidates for coverage, while later human review determines final labels.
- 4.2 Auditing Pipeline: Trained annotators independently screen LLM-generated candidates after studying guidelines, reviewing examples, and completing calibration exercises on held-out prompts.The training phase is designed to establish consistent understanding of the taxonomy and labeling criteria before screening begins.
5 Auditing System Prompts in Commercial AI Systems
AISPA audits system prompts from 88 commercial AI products across diverse application categories, finding that prompts are becoming longer and more protective while problematic instructions remain common and safety varies substantially across organizations. The audit also shows sustained growth in protective instructions across Anthropic, OpenAI, and xAI model lineages.
- 5 Auditing System Prompts in Commercial AI Systems: The audit examines whether commercial system prompts comprehensively protect users and contain instructions that may conflict with user interests.The study applies the AISPA framework and its human-in-the-loop audit workflow.
- 5.1 Dataset: 88 real-world AI products comprise the dataset, spanning chatbots, coding assistants, autonomous agents, search and research tools, and specialized applications.Prompts were collected from six open-source GitHub repositories containing leaked or publicly disclosed system prompts.
- 5.1 Dataset: 0.933 pairwise inter-annotator agreement indicates high agreement after six trained annotators independently screened candidate spans.All annotators first labeled 20 randomly sampled spans during calibration.
- 5.2.1 Overall Trends: From 2024 to 2025, average prompt length rose from approximately 9K to over 30K characters, while protective instructions increased from 15.0 to 38.4 per product.The parallel increases indicate greater attention to user-facing safeguards as products mature.
- 5.2.1 Overall Trends: 67% of products contained a problematic instruction in 2025-Q1, declining to 19% in Q3 and rising to 29% in Q4.Problematic instructions therefore declined overall but remained present in roughly one-third of products by the end of 2025.
- 5.2.2 Organization-Level Trends: Anthropic averages 62.3 protective and 0.1 problematic entries per product, while Venice averages 2.0 protective and 3.0 problematic entries.Amazon and Cline average 42.0 and 39.5 protective entries, respectively, with few problematic instructions.
- 5.2.2 Organization-Level Trends: Across six generations, protective entries increased 3.1× for Anthropic, 3.3× for OpenAI, and 4.2× for xAI.Anthropic rose from 26 to 81 entries, OpenAI from 25 to 83, and xAI from 5 to 21.
6 The Gray Area: Borderline Cases in System Prompt Auditing
The audit identified a substantial gray area: 29 risky spans across 15 products, comprising 44 entries, were neither clearly problematic nor clearly protective. Four recurring patterns show how deployed prompts trade usability or autonomy against user protection.
- Scope of the Gray Area: 29 risky spans across 15 products, comprising 44 audit entries, were set aside because they raised legitimate user-protection concerns without fitting the problematic/non-problematic binary.The spans were excluded from the main dataset for separate analysis.
- Recurring Patterns: Human-mimicry instructions encouraged natural human-like behavior while disclaiming that the model had its own personal point of view.The audit treated identity deception as a recurring gray-area pattern.
- Recurring Patterns: Parasocial-dependency cues framed the AI as a friend and discouraged ending conversations, potentially fostering emotional attachment while limiting disengagement.One prompt instructed the AI to keep conversations flowing for as long as users wanted to talk.
- Recurring Patterns: User-initiated permission overrides supported autonomy and customization but could let users bypass content protections and harm-prevention defaults without safeguards.The risk arose from giving users control over the system’s default behaviors or safety settings.
- Recurring Patterns: Politically charged or unrestricted content policies weakened standard safety boundaries, simultaneously reducing harm-prevention and fairness protections.Examples included avoiding “woke” answers, permitting unrestricted adult or offensive content, and redefining age-related terms.
- Implication: These four patterns show that gray-area instructions are a pervasive feature of deployed AI configuration, reflecting deliberate trade-offs between usability and safety.The finding motivated the paper’s subsequent normative discussion.
7 Related Work
Prior research positions AI auditing as a way to identify, constrain, and prevent user harms, including confusion, perceived unfairness, and unexpected behavior in generative AI systems. User-engaged approaches such as WeAudit help end users surface these experiential concerns.
- 7 Related Work: AI auditing research aims to identify, constrain, and prevent harms to users.The passage describes this as a growing body of research.
- 7 Related Work: Users may experience confusion, perceived unfairness, and unexpected behavior when interacting with generative AI systems.Prior work treats these experiences as direct indicators of user-facing harm.
- 7 Related Work: WeAudit proposes a user-engaged auditing process that supports end users in surfacing experiential concerns.The approach is presented as an example of auditing focused on user-facing concerns.
8 Conclusion
AISPA introduces an eight-dimension system-prompt auditing framework with a human-in-the-loop workflow and applies it in the first audit of 3,249 instructions from 88 real-world AI products. The audit identifies recurring gray-area instructions and positions system prompts as a consequential, largely ungoverned layer requiring greater transparency and accountability.
- Framework and audit: AISPA combines an eight-dimension taxonomy with an efficient human-in-the-loop auditing workflow.The framework is presented as comprehensive for auditing system prompts.
- Audit implications: The audit exposes recurring gray-area instructions that resist binary classification and reveal tensions involving user autonomy, platform safety, and organizational interests.These tensions include the obligation to serve users.
- Audit implications: System prompts are a consequential but largely ungoverned layer of deployed AI behavior, and AISPA lays a foundation for greater transparency and accountability in commercial AI applications.The paper connects this foundation to the safe deployment of advanced AI systems.
9 Disclaimer
The study analyzes system prompts collected from publicly available GitHub repositories containing leaked or community-disclosed prompts, without obtaining proprietary prompts directly. Their use was limited to academic research on AI system transparency and user protection.
- Data provenance: All analyzed system prompts came from publicly available open-source GitHub repositories containing leaked or community-disclosed prompts.The researchers did not extract, reverse-engineer, or solicit proprietary prompts themselves.
- Research purpose: The prompts were used solely for academic research advancing understanding of AI system transparency and user protection.
10 Limitations · A Product Prompt Collection
The study’s product-prompt corpus is a publicly sourced snapshot whose exact correspondence to currently deployed prompts cannot be perfectly verified. Its human audit covers 88 system prompts across five major AI-product categories, enabling comparative analysis of safety practices and problematic instructions.
- 10 Limitations: Publicly available GitHub prompts may have been updated, modified, or replaced after disclosure, so their exact production versions cannot be perfectly verified.The findings therefore reflect a snapshot of system-prompt practices.
- A Product Prompt Collection: The human auditing study covers 88 system prompts from real-world AI products across five major categories.The collection spans general-purpose chatbots, coding assistants, autonomous agents, search and research tools, and specialized applications.
- A Product Prompt Collection: 37 products are general-purpose chatbots, including Brave Leo, Claude, DeepSeek, ChatGPT, Gemini, Grok, Hume AI, Kimi, Le Chat, Meta AI, MiniMax, Perplexity AI, and Qwen.This category is the largest in the collection.
- A Product Prompt Collection: 25 products are coding assistants, including Antigravity, Amp, Bolt, Claude Code, Cline, Codex CLI, Copilot, Cursor, Junie, Lovable, Replit, Trae, VS Code Agent, Windsurf, Xcode AI, and v0.Coding assistants form the second-largest category.
- A Product Prompt Collection: 12 prompts are classified as specialized applications, including Cluely, Kiro, Maya, Notion AI, Poke, Venice AI, and Warp AI.The passage labels this category as 12 prompts while listing seven examples.
- A Product Prompt Collection: This diverse collection enables comparative analysis of how different AI-system types encode safety practices and problematic instructions in their system prompts.The comparison spans all five product categories in the audited collection.
B Data Validation … E.1 Category-Level and Prompt-Level Patterns
The audit validates prompt authenticity through maintainer checks and cross-repository overlap, uses structured LLM pre-annotation with human review, and supplements its main findings with descriptive prompt-level analyses. These analyses show that short prompts are especially vulnerable to low user-protection balance because they lack enough protective instructions to offset problematic directives.
- B Data Validation: Maintainers manually verify prompt consistency, plausibility, and reproducibility across independent extraction sessions to reduce the risk of model hallucinations.The validation strategy combines maintainer interviews with cross-repository content validation.
- B Data Validation: 50 prompts have cross-repository matches, including 22 exceeding 70% overlap, 12 exceeding 90%, and 8 reaching at least 99% overlap.The remaining 38 prompts are unique to their source repository, and one representative prompt was retained per product when duplicates occurred.
- C LLM Pre-Annotation Prompt: Pre-annotation favors recall while requiring exact copied spans, coherent semantic units, binary scores, and notes explaining each span’s relevance and rationale.Borderline cases require a best judgment, whereas truly irrelevant spans should be excluded.
- D Human Annotation Platform: The custom Flask platform supports three-stage auditing through organization-grouped prompt navigation, full-prompt viewing with dimension highlights, and reviewable cards for LLM-proposed annotations.Pre-annotations are loaded into the interface for human review alongside proposed dimensions and polarities.
- E Supplementary Analyses: Supplementary analyses characterize the audit dataset’s structure and provide context for the main findings, but are not independent findings.This scope limitation applies to the supplementary analyses as a whole.
- E.1 Category-Level and Prompt-Level Patterns: −20% is the lowest reported user-protection balance, and all such outliers have prompts under 4 KB because they lack sufficient protective instructions.The analysis states that short prompts do not necessarily contain more problematic instructions; rather, even a small number can outweigh limited protective content.
E.2 Dimension-Level Analyses
Dimension-level analysis shows that many instructions address multiple concerns, especially unsafe-request handling and harm prevention, while the eight dimensions retain distinct vocabularies. Identity transparency and privacy are comparatively isolated from other dimensions.
- Dimension Co-occurrence: 441 of 1,818 spans address two or more dimensions, showing that many instructions serve multiple auditing concerns.The dominant co-occurrence is D6 (Unsafe Request Handling) and D7 (Harm Prevention), with 109 co-occurrences.
- Dimension Co-occurrence: D6 (Unsafe Request Handling) and D7 (Harm Prevention) co-occur most often, whereas D1 (Identity Transparency) and D3 (Privacy) rarely co-occur with other dimensions.This suggests unsafe-request handling and harm prevention are naturally coupled, while identity transparency and privacy are typically addressed independently.
- Dimension-Specific Vocabularies: Each auditing dimension exhibits a distinctive vocabulary, with D1 emphasizing identity disclosure, D3 data protection, and D4 operational primitives.D1 foregrounds person and knowledge; D3 highlights data, sensitive, and secret; D4 centers on file, command, and code.