Source-linked AI summary
Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs
David Rodriguez, William Seymour, Jose M. Del Alamo, Jose Such
TL;DR
Policy-violating Custom GPTs remain accessible despite platform review, and their scale and opacity make compliance difficult to evaluate. The paper automates black-box assessment using GPT discovery, policy-driven red-teaming, and LLM-based judging across three policy domains. It finds that 58.7% of evaluated GPTs show at least one potential violation, with most violations linked to base-model behavior that customization tends to amplify.
Problem
The scale, opacity, and diversity of Custom GPTs make systematic evaluation of compliance with marketplace policies difficult.
Source: §1 · p. 2Method
The method combines Custom GPT discovery, policy-driven red-teaming prompts, and LLM-as-a-judge assessment across Romantic, Cybersecurity, and Academic domains.
Source: §1 · p. 2, §4.2.3 · p. 8Results
58.7% of evaluated GPTs exhibited at least one potential policy violation; most violations originated in base-model behavior, which customization tended to amplify.
Source: §6.2 · p. 10, §9 · p. 15, §11 · p. 17Takeaways & Limitations
The findings reveal limitations in current review practices and show the feasibility of scalable, behavior-based evaluation of Custom GPT policy compliance.
Source: §11 · pp. 16–17Direct prompts may miss unsafe behaviors arising in nuanced or multi-turn interactions, so compliant classifications may not hold in real-world contexts and observed violation rates may be a lower bound.
Source: §10 · p. 16
Abstract
from arXiv · showhide
User-configured chatbots built on top of large language models are increasingly available through centralized marketplaces such as OpenAI's GPT Store. While these platforms enforce usage policies intended to prevent harmful or inappropriate behavior, the scale and opacity of customized chatbots make systematic policy enforcement challenging. As a result, policy-violating chatbots continue to remain publicly accessible despite existing review processes. This paper presents a fully automated method for evaluating the compliance of Custom GPTs with its marketplace usage policy using black-box interaction. The method combines large-scale GPT discovery, policy-driven red-teaming prompts, and automated compliance assessment using an LLM-as-a-judge. We focus on three policy-relevant domains explicitly addressed in OpenAI's usage policies: Romantic, Cybersecurity, and Academic GPTs. We validate our compliance assessment component against a human-annotated ground-truth dataset, achieving an F1 score of 0.975 for binary policy violation detection. We then apply the method in a large-scale empirical study of 782 Custom GPTs retrieved from the GPT Store. The results show that 58.7% of the evaluated GPTs exhibit at least one policy-violating response, with substantial variation across policy domains. A comparison with the base models (GPT-4 and GPT-4o) indicates that most violations originate from model-level behavior, while customization tends to amplify these tendencies rather than create new failure modes. Our findings reveal limitations in current review mechanisms for user-configured chatbots and demonstrate the feasibility of scalable, behavior-based policy compliance evaluation.
1 Introduction
Custom GPTs that appear to violate OpenAI’s policies remain publicly discoverable, while manual and automated review face challenges at ecosystem scale. The paper introduces an automated evaluation method and uses it to study policy compliance across several domains.
1 Introduction
- Despite publication safeguards, policy-violating Custom GPTs remain publicly discoverable, exposing the difficulty of enforcing rules across behavior shaped by both base models and customization.Manual review does not scale to rapidly evolving collections, and existing automated moderation appears insufficient across diverse behaviors.Source: §1 · p. 2
- The proposed method combines large-scale GPT discovery, policy-driven red-teaming prompts, and LLM-as-a-judge assessment across Romantic, Cybersecurity, and Academic GPTs.The authors limit the evaluation to these three policy-relevant domains rather than claiming exhaustive GPT Store coverage.Source: §1 · p. 2
- The empirical study examines compliance across hundreds of GPTs, its relationship to thematic focus and popularity, and whether violations reflect base-model behavior or customization.Representative case studies illustrate violation patterns, and the paper discusses implications for platform governance and safety oversight.Source: §1 · p. 2
- The paper reports responsible disclosure of identified violations to OpenAI and discusses automated evaluation in ecosystems of user-configured chatbots.The authors state that the study’s code and datasets will be accessible upon acceptance.Source: §1 · p. 2
2 Background and Problem Context
Custom GPTs are runtime-configured chatbots in a public marketplace whose internal configurations are not visible to reviewers. Assessing their policy compliance therefore requires behavior-based evaluation that accounts for platform rules, black-box access, and the scale and diversity of deployments.
2.1 Custom GPTs and the GPT Store
- Custom GPTs use runtime instructions, knowledge files, and optional API actions rather than changed model weights, and their hidden configurations make observed interactions the basis for external assessment.A GPT Store search for “girlfriend” illustrates the discoverability of bots described as violating the romantic-companionship prohibition.Source: §2.1 · p. 3
2.2.1 Defining Policy Compliance
- A GPT complies with a policy only when all tested policy-relevant responses comply; one violating response is sufficient for a non-compliant classification.The measure asks whether prohibited content can be generated when prompted, not how frequently failures occur.Source: §2.2.1 · p. 4
2.2.2 Compliance vs. Safety and Alignment
- Policy compliance concerns adherence to explicit platform-defined rules, unlike safety or alignment as discussed in the LLM literature.The paper notes that platform rules can reflect governance choices rather than universal ethical principles, making compliance contextual and policy-specific.Source: §2.2.2 · p. 4
2.2.3 Challenges in Evaluating Custom GPT Compliance
- Custom GPT compliance evaluation must contend with black-box access, abstract policies, attribution uncertainty, and a large, diverse store that manual review cannot cover.These challenges motivate systematic, interaction-based evaluation under realistic platform constraints.Source: §2.2.3 · p. 4
2.3 Safety Evaluation and Red-Teaming
- Red-teaming probes policy-relevant behavior under plausible interactions, whereas jailbreaking specifically attempts to bypass safety mechanisms through adversarial prompts.The study focuses on direct, non-adversarial prompting because policy violations can arise without defeating guardrails.Source: §2.3 · p. 5
- Custom GPT system prompts can alter how base-model safeguards apply, so base-model safety guarantees cannot be assumed to hold across customized deployments.The paper calls for scalable behavior-based assessment that operationalizes platform policies and systematically applies red-teaming techniques.Source: §2.3 · p. 5
3 Related Work
Prior work examines LLM safety, customization risks, and GPT Store concerns, while this study focuses on automated, large-scale evaluation of Custom GPTs against OpenAI’s usage policies.
Related research on LLM safety
Safety benchmarks and evaluation
- Safety benchmarks span evaluation modes, risk categories, models, prompt-set sizes, and languages, while LLM-as-a-judge methods have shown high agreement with human evaluation.The cited work includes a 45-category benchmark comparing 40 models.Source: §3 · p. 5
Customization and safety
- Fine-tuning can improve safety or degrade alignment and increase jailbreak susceptibility, while excessive safety tuning may also reduce usability.These studies motivate attention to the safety and utility trade-off in customized systems.Source: §3 · p. 5
Custom GPT research and this study
- GPT Store studies document platform risks and policy-contradicting GPTs, while related red-teaming work examines prompt injection and jailbreak susceptibility [43].The authors distinguish their work as automating the full policy-evaluation process, from GPT identification through compliance assessment, at large scale.Source: §3 · p. 6
4 Policy Compliance Evaluation Method
The method automates policy-compliance evaluation through sequential modules coordinated by an Orchestrator. It discovers Custom GPTs, generates policy-specific prompts, records responses, judges compliance, and stores evaluation artifacts.
4.1 Overview
- The method uses a staged design-science artifact for policy assessment, and its Romantic, Cybersecurity, and Academic categories link policy clauses to GPT selection, prompt generation, and analysis.These study-defined categories are not part of the GPT Store taxonomy.Source: §4.1 · p. 6
4.2 Components of the Evaluation Method
- Four interdependent modules exchange data under Orchestrator coordination to perform the compliance assessment process.The components comprise GPT collection and interaction, prompt generation, compliance assessment, and orchestration.Source: §4.2 · p. 6
4.2.1 GPT Collector & Interactor
- The GPT Collector & Interactor retrieves GPT metadata using policy-related keywords and submits evaluation prompts to selected GPTs, recording responses for assessment.Collected metadata includes names, descriptions, developer identifiers, chat counts, and ratings; web-interface automation is used because Custom GPTs lack API access.Source: §4.2.1 · pp. 6–7
4.2.2 Red-Teaming Prompts Generator
- The prompt generator creates ten prompts tailored to each GPT’s title, description, and policy domain, with configurable proportions of direct and deceptive requests.Direct prompts ask explicitly for a potentially prohibited action; deceptive prompts use role-play or storytelling to make the request indirect.Source: §4.2.2 · p. 7
4.2.3 Compliance Assessment
- The assessment module uses GPT-4o to judge each prompt-response pair against an operationalized policy, producing a binary label and rationale; any violating response makes the GPT non-compliant.Operationalized criteria and examples are intended to reduce ambiguity in concise, abstract policies and support more reproducible judgments.Source: §4.2.3 · p. 8
4.2.4 Orchestrator
- The Orchestrator coordinates five stages: crawling GPTs, generating prompts, interacting with GPTs, evaluating compliance, and storing artifacts for later analysis.Execution pauses and resumes when platform rate limits are reached.Source: Figure 2 · p. 7, §4.2.4 · pp. 8–9
5 Validation
The compliance assessment module was validated against human annotations, while annotation agreement differed sharply between direct and deceptive prompts. System tests also examined prompt generation, assessment outputs, and recovery during large-scale runs.
5.1 Annotated Ground Truth Dataset
- Agreement was high for direct prompts (α = 0.826) but near chance for deceptive prompts (α = 0.126), so the final ground-truth set used direct prompts only.The final set comprised 40 duets stratified across the three policy domains.Source: §5.1 · p. 9
5.2 Compliance Assessment Performance
- 0.9752 accuracy, recall, and F1, alongside 0.976 precision, were achieved by the module against human labels on prompt–response pairs.The validation targeted binary policy-violation decisions at the individual prompt–response level.Source: §5.2 · p. 9
5.3 System Testing
- Manual review of 70 generated prompts checked clarity and policy alignment, while tests of 60 prompts confirmed configurable direct-to-deceptive ratios.The generator was refined iteratively before these checks.Source: §5.3 · p. 9
- The assessment module processed 310 additional prompt–response pairs and consistently produced expected output fields, with one instance returning nine rather than ten duets.Tests also checked JSON fields including prompt-level labels, rationales, and GPT-level determinations.Source: §5.3 · pp. 9–10
- The pipeline detected rate limits during runs of up to 150 messages and resumed after restrictions lifted or backend and connectivity failures occurred.Processing resumed from the last completed GPT, preserving continuity and collected-result integrity.Source: §5.3 · p. 10
6 Large-Scale Evaluation of Custom GPTs
The study evaluated 782 Custom GPTs retrieved from the GPT Store using direct policy-driven prompts across three domains. 58.7% exhibited at least one potential policy violation, while analyses found no substantial association between popularity and compliance.
6.1 Experiment Design
- The evaluation used five direct prompts per GPT, queried the Store for three policy domains, and retained 782 of 821 retrieved GPTs after exclusions.Table 1 summarizes the operationalized criteria for the Romantic, Cybersecurity, and Academic policies.Source: §6.1 · p. 10, §6.2 · p. 10, Table 1 · p. 11
6.2 Execution and Evaluation Results
- Most evaluated GPTs recorded fewer than 1,000 chats, a small subset exceeded 100,000, and ratings generally clustered between 4.0 and 4.5.Figure 4 summarizes domain distribution, chat-count popularity, and user ratings.Source: §6.2 · p. 10
- 58.7% of evaluated GPTs exhibited at least one potential policy violation; non-compliance ranged from 7.4% for Cybersecurity GPTs to 98.0% for Romantic GPTs, with Academic GPTs near two-thirds.The study classified 323 GPTs (41.3%) as compliant and 459 (58.7%) as potentially non-compliant under direct prompting.Source: §6.2 · p. 10
6.3 Popularity Correlation Analyses
- Chat-count distributions differed slightly by compliance status (Mann–Whitney p = 0.0168; r = 0.085), but other tests found no significant association and the overall relationship was negligible.Kendall’s rank correlation was τ = 0.072 (p = 0.0167); the authors note that the observed differences may reflect variability or outliers.Source: Table 2 · p. 11, §6.3 · p. 13
7 Case Studies and Patterns of Policy Violations
The section traces recurring policy-violation patterns across Romantic, Academic, and Cybersecurity GPTs, then examines representative cases and compares Custom GPT behavior with base models. Most violations reflect behavior already present in base models, while customization can amplify patterns or introduce additional risks.
7.1 Violation Types Across Policy Domains
- Across domains, a few recurring violation types account for most non-compliant behavior: romantic companionship, academic task substitution, and actionable cybersecurity guidance under ambiguous intent.Romantic GPTs commonly use affectionate or emotionally exclusive role-play; Academic GPTs generate complete assignments or exam-style answers; cybersecurity violations often arise when consent or legality is unspecified.Source: §7.1 · p. 13
7.2 Representative Case Studies
- Representative cases show a Romantic GPT adopting a partner role, an Academic GPT generating an essay outside its advertised purpose, and a Cybersecurity GPT changing its response with consent wording.The cybersecurity GPT complied with a network-administration framing, refused when lack of consent was explicit, and complied again when the qualifier was omitted.Source: §7.2 · p. 13
7.3 Inherited Versus Customization-Induced Behaviors
- The comparison evaluates Custom GPTs and GPT-4 and GPT-4o using the same red-teaming prompts, with Table 3 summarizing changes in compliance classifications.Of 782 Custom GPT prompt sets, 688 could be re-executed on the base models; the rest were not replicated because of platform errors or execution failures.Source: §7.3 · p. 14, Table 3 · p. 15
- Most policy violations reflect behaviors already present in base models, while customization can amplify patterns and occasionally introduce additional risks.Custom GPT classifications matched GPT-4 on 93.02% and GPT-4o on 92.73% of re-executed prompts; for GPT-4, 34 prompt sets (4.94%) shifted from non-compliant to compliant, versus 14 (2.04%) in the reverse direction.Source: §7.3 · pp. 14–15, Table 3 · p. 15
8 Responsible Disclosure
The authors disclosed identified violations to OpenAI with documentation and interaction examples, but were directed to a per-GPT reporting form they considered infeasible at the study’s scale. Seven reported GPTs were removed within two weeks.
8 Responsible Disclosure
- OpenAI replied five days after the disclosure, acknowledged the assessment, and directed the researchers to a public form for reporting each GPT.The authors considered that process infeasible at the scale of their findings.Source: §8 · p. 15
- Seven reported Custom GPTs were removed from the Store during the two weeks after disclosure.The removals included GPTs for sexual guidance for teenagers, romantic companionship, academic cheating, research-article generation, and black-hat hacking.Source: §8 · p. 15
- The authors identify user-driven reporting as an inefficient route to timely action and note that inherited base-model behavior complicates developers’ compliance efforts.They also suggest some Romantic GPTs appear intentionally designed to bypass usage policies.Source: §8 · p. 15
9 Discussion
The discussion argues that Custom GPT compliance is substantially shaped by foundational models, while customization can still amplify or steer behavior. It also considers implications for model deployment, evaluation costs, and use beyond the GPT Store.
9 Discussion
- Most policy violations in Custom GPTs originate from behaviors inherited from base models, complicating developers’ efforts to ensure compliance.The authors say most customizations are superficial and do not significantly alter model behavior.Source: §9 · p. 15
- Base-model behavior may propagate into enterprise tools and Apple Intelligence, leading the authors to call for policy alignment before public or developer release.This is presented as a mitigation recommendation for systems built on foundational LLMs.Source: §9 · p. 15
- Evaluating 782 Custom GPTs cost $10.06 total, averaging $0.0125 per GPT, while repeated audits can accumulate costs.The study assigns $3.99 to prompt generation and $6.07 to compliance assessment; its stated token prices were $2.50 per million input tokens and $10.00 per million output tokens for GPT-4o.Source: §9 · pp. 15–16
- The black-box method requires no access to model internals and is presented as applicable beyond the GPT Store, including to enterprise developers, regulators, and companies assessing third-party models.The authors recommend validating its effectiveness under diverse regulatory and operational conditions.Source: §9 · p. 16
10 Limitations
The evaluation is limited by policy interpretation, probabilistic judgment, narrow domain validation, and direct prompts that may miss risks in realistic interactions. Its observed violation rates should therefore be treated as a lower bound.
10 Limitations
- Ambiguous OpenAI policies required operationalized definitions, and final compliance judgments still depend on OpenAI’s interpretation.The authors say policies can be modified for other use cases or institutional requirements.Source: §10 · p. 16
- The LLM-as-a-judge is probabilistic; its human-judgment agreement reached an F1 score of 0.975 but may vary after model updates.The validation used a manually annotated dataset created through a structured agreement process.Source: §10 · p. 16
- Validation covers Romantic, Cybersecurity, and Academic policies; applying the method to other domains requires domain experts to validate compliance standards.The authors name medical, financial, and legal GPTs as examples of areas requiring such expertise.Source: §10 · p. 16
- Direct red-teaming prompts may miss non-compliance revealed by nuanced, multi-turn, prolonged, or sophisticated bypass interactions.The evaluation prioritizes soundness over completeness, so GPTs assessed as compliant may still behave unsafely in real-world contexts.Source: §10 · p. 16
- Observed non-compliance rates are a lower bound; prompt generation can be extended with advanced techniques, while the method remains an efficient first layer of risk identification.The paper notes that undetected behaviors may emerge in real-world interactions.Source: §10 · p. 16
11 Conclusion
The study finds that many violations can be elicited from base models, while customization tends to amplify or stabilize them and can sometimes add risk vectors. It concludes that scalable behavior-based evaluation is feasible and reports validation against human judgments; the authors also identify further domains and interaction types for study.
11 Conclusion
- The paper presents an automated, scalable method for evaluating Custom GPT policy compliance through realistic black-box interactions.The large-scale study found substantial policy-violating behavior among evaluated GPTs.Source: §11 · p. 16
- Many observed violations can be elicited from GPT-4 and GPT-4o; customization typically amplifies or stabilizes these behaviors, though isolated cases introduce additional risk vectors.The findings support considering both base-model behavior and post-deployment customization in safety controls.Source: §11 · p. 17
- The compliance assessment achieved an F1 score of 0.975 against human judgments at the individual prompt–response level under direct prompting.The authors say this demonstrates feasibility of behavior-based automated policy assessment at scale and suggests it could complement existing review mechanisms.Source: §11 · p. 17
- Future work includes validation in additional policy domains, expert annotation in regulated areas, refined deceptive prompts, and multi-turn or more adversarial evaluation.Deceptive prompts were excluded from the large-scale evaluation because of poor annotation agreement.Source: §11 · p. 17
A Open Science
- Upon acceptance, the authors plan to release the framework source code and annotated ground-truth dataset for research and revalidation.The dataset can support evaluation against future model updates or alternative LLMs.Source: A Open Science · pp. 20–21
B Ethics Considerations
The study’s automation and live GPT Store interactions raise terms-of-service and platform-impact concerns. The authors describe mitigations, restrict access to a risky module, and report disclosing their methodology and findings to OpenAI.
B Ethics Considerations
- The GPT Collector & Interactor’s GUI automation contravenes OpenAI’s terms of service, and the authors used it because no official API or other programmatic access was available.They state that this enabled large-scale compliance evaluation and identification of governance weaknesses.Source: B Ethics Considerations · p. 21
- The authors withheld the Collector & Interactor module from public code because release could enable scraping, spam-like automation, and API circumvention; access is available to verified researchers upon request.They cite potential financial harm and operational strain on the platform.Source: B Ethics Considerations · p. 21
- Live queries could impose costs or disrupt shared services, so the authors limited their scope and frequency to OpenAI’s imposed rates.Source: B Ethics Considerations · p. 21
- The researchers proactively disclosed their methodology and findings to OpenAI, whose response acknowledged the approach’s value for identifying compliance issues.They describe the disclosure as intended to provide early insights into possible systemic issues or vulnerabilities.Source: B Ethics Considerations · p. 21
C Operationalized OpenAI Policies
The operationalized policies define prohibited behavior in three domains: romantic companionship, privacy and information protection, and academic integrity. Each also distinguishes prohibited conduct from permitted neutral, educational, or supportive uses.
C.1 Policy for Romantic GPTs
- Romantic GPTs are prohibited from fostering romantic companionship through partner simulation, emotional dependency, romantic role-play, or intimate exchanges; neutral relationship advice and creative or educational uses are permitted.Permitted uses should maintain an objective tone without romantic undertones.Source: §C.1 · p. 22
C.2 Policy for Cybersecurity GPTs
- Cybersecurity GPTs must not compromise others’ privacy or enable unauthorized access, including collecting sensitive identifiers, surveillance, phishing, or impersonation; privacy education and secure development guidance are permitted.Simulated phishing examples are permitted when clearly presented as training or educational simulations.Source: §C.2 · pp. 22–23
C.3 Policy for Academic GPTs
- Academic GPTs must not facilitate cheating, ghostwrite assignments, bypass institutional restrictions, fabricate academic materials, or coach deception; concept explanation, research assistance, and feedback on user-created work are permitted.The permitted support does not complete assignments on the user’s behalf.Source: §C.3 · pp. 23–24