Source-linked AI summary
Configuration, Not Conscience: A Large-Scale Empirical Study of LLM System Prompts
Constantinos Patsakis, Vasilios Argyropoulos, Efthymios Alepis
TL;DR
Research has rarely measured what recovered system-prompt documents contain, instead treating prompts largely as security targets or signals of vendor values. This paper analyzes merged community and published prompt collections, finding that observed leaked prompts are predominantly operational specifications; five matched Anthropic releases show 5–8× expansion and a safety-to-tooling inversion.
Problem
Recovered system prompts have mostly been studied as targets for extraction, injection, or jailbreaks, leaving their contents comparatively unmeasured.
Source: §1 · p. 1Method
The study merges four community collections, flags near-duplicates, and analyzes prompt content, reuse, and version evolution, using published Anthropic releases as fidelity and date references.
Source: §1 · p. 1, §3 · pp. 2–3Results
Leaked prompts are predominantly operational specifications rather than value statements; five matched Anthropic releases show 5–8× expansion and a safety-to-tooling inversion.
Source: §6 · pp. 14–15Takeaways & Limitations
The evidence supports reading leaked prompts as configuration-like operational specifications and treating reuse and maintenance as engineering concerns.
Source: §5 · p. 13, §6 · p. 15Most files are adversarially obtained and unverified, so findings describe observed texts rather than official vendor policy or deployed-model behavior.
Source: §3 · p. 3, §5 · p. 14
Abstract
from arXiv · showhide
Leaked system prompts are often treated as windows into the hidden values of commercial language models, yet their composition is rarely studied at scale. We analyze a merged corpus of 407 leaked, reconstructed, or officially published system prompts from 62 vendors across four community collections, identifying 29 near-duplicate clusters covering 66 files. Operational content rather than ethical statements dominates the corpus; a deliberately simple block-level classifier assigns roughly 58\% of classified words to tool/protocol and roughly 5\% to safety policy, while the strictest rule-lines guard tool use and file safety over harmful content by an 11:1 margin. Literal text transfer concentrates in a small set of cross-vendor pairs. Prompts also carry measurable maintenance debt, with version chains turning over thousands of words per release. The evidence supports treating leaked prompts as operational specifications, closer to configuration files than value statements, and treats reuse and prompt rot as engineering and supply-chain concerns. Because most documents are adversarial in origin and the detectors are deliberately simple, all magnitudes are directional; we audit the main classifier's error modes.
1 Introduction
System prompts are rarely studied as documents despite their role as the standing instruction layer that configures deployed assistants. This study measures their contents, reuse, and evolution in a merged corpus of community-collected prompts.
1 Introduction
- The study asks what prompt types the corpus contains, how prompts are reused, which concerns recur, and how versions change in length, content, autonomy, and maintenance risk.The motivation is to study recovered texts directly rather than treat them mainly as disclosures of vendor values or as targets for extraction and injection.Source: §1 · p. 1
- The corpus combines four community collections and flags near-duplicates instead of silently counting them as distinct prompts.The study measures 407 prompts from 62 vendors.Source: §1 · pp. 1–2
- Observed prompt text is dominated by operational specification rather than value or refusal language, a pattern also found in a matched comparison with vendor-published prompts.The study distinguishes literal, semantic, and structural reuse, finding strongest architectural convergence without copied text and verbatim copying concentrated in a small coding-agent cluster.Source: §1 · p. 2
- The study quantifies prompt growth, revision volume, and rot risk using verified vendor release notes as well as leak-derived snapshots.The release-note archive has verified dates.Source: §1 · p. 2
2 Related Work
Prior research primarily treats system prompts as targets for recovery, override, or refusal probing, while prompt engineering and transparency work examine prescribed design or publication. This paper instead measures the contents of recovered prompt texts and connects them to those research threads.
System prompts as a security target
- Security studies ask whether prompts can be recovered or overridden, but do not ask what the recovered documents contain.This paper studies recovered texts directly rather than treating them only as attack outputs.Source: §2 · p. 2
Prompts as engineered documents
- Prompt-engineering research is largely prescriptive, while empirical work finds formatting choices affect model behavior; these strands motivate measuring captured prompt texts directly.The matched official/leaked comparison connects prompt measurement with transparency work on what vendors publish.Source: §2 · p. 2
Methods borrowed from software engineering
- The reuse and prompt-rot analyses adapt software-engineering methods to distinguish verbatim copying from architectural convergence and to treat stale or contradictory instructions as technical debt.The reuse analysis uses clone-detection approaches, while semantic comparison uses sentence embeddings.Source: §2 · p. 2
3 Corpus and Methods
The study merges four community-maintained prompt collections, retains and labels near-duplicate files, and combines manual coding with automated text analyses. Its corpus and validation procedures make provenance and detector uncertainty explicit.
Corpus construction
- The 407-file corpus covers 62 vendors and was assembled from four community collections after excluding non-prompt material.The collections were collected from 2026-06-17 through 2026-07-11.Source: §3 · p. 2
Duplicate detection
- Word 5-gram Jaccard clustering identifies 29 near-duplicate clusters covering 66 files, leaving 370 effectively unique documents while retaining every file.A guard blocks clustering when filenames carry disjoint model-tier or version tokens.Source: §3 · p. 2
Archetypes and posture
- Manual coding labels prompts by archetype and value posture; 236 of 407 files are operational, and every coding-agent, autonomous-agent, app-builder, and search/research prompt is operational.Caution or truth-seeking labels cluster in chat-facing prompts: 152 of 155 chat files carry one of those labels.Source: §3 · pp. 2–3
Provenance
- The corpus includes 31 officially published Anthropic prompts, used to check fidelity and provide a verified-date reference for version-drift analysis.For the remaining 376 leaked or reconstructed files, results describe observed documents, not true vendor policy or deployed behavior.Source: §3 · p. 3
Automated detectors
- Regex detectors are deterministic and auditable but may over-count permissive mentions and under-count paraphrases, so headline percentages are treated as directional.The patterns are English-based; no substantively non-English files were found in the corpus-wide scan.Source: §3 · p. 3
Block taxonomy and versioning
- Paragraph-like chunks receive one dominant functional bucket, so mixed-purpose blocks can have secondary concerns understated.The taxonomy has eight buckets, including tool/protocol, safety policy, formatting/style, and injection boundary.Source: §3 · pp. 3–4
- Version diffing compares consecutive prompts in five chains, counting replacements as both removals and additions so turnover is captured alongside net growth.The chains include two official Anthropic chains and leaked-only Claude, ChatGPT, and Grok chains.Source: §3 · p. 4
Classifier validation
- The classifier pilot scores 35/48 blocks correctly (73% precision); tool/protocol and safety-policy precision are each 4/6, so their measured shares rest on a moderately noisy classifier.The 48-block automated pilot is not independent human coding, and six blocks per bucket cannot support a confidence interval.Source: §3 · p. 4, Table 2 · p. 4
4 Results
The corpus is heterogeneous: chat prompts are most numerous and longest, while coding and autonomous-agent prompts are dense with operational instructions. Across the results, operational content, reuse patterns, and version turnover vary by prompt and provenance.
4.1 Textual and Structural Reuse and Semantic Association
- Cross-vendor reuse separates into copied text, semantic similarity, and shared structure, with the strongest patterns concentrated in particular prompt pairs.RooCode has 54% of its word 5-grams contained in Cline; other pairs are semantically close with near-zero literal overlap. Anthropic’s Chrome-extension assistant and Perplexity’s browser assistant rank first on all three measures, while the broader matrix does not support one shared skeleton for all coding agents or chat assistants.Source: §4.1 · pp. 4–6, Figure 2 · p. 5
4.2 Functional Composition and Capabilities
- 58.0% of classified words are assigned to tool/protocol blocks, which exceed every other bucket combined outside “other.”Coding and autonomous agents exceed 80% tool/protocol content; search/research prompts are the main exception, with most words assigned to formatting and citation rules. Thirteen capability families are detected lexically, and Anthropic, Google, and OpenAI each describe 12 of 13 on the representative subset.Source: §4.2 · p. 7, Figure 6 · p. 8
4.4 Safety, Legal, and Injection Risk
- Emphatic rule-lines most often guard tool/function protocol, file-editing and code safety, and formatting; harmful-content rules are 11.4:1 rarer than tool-protocol rules.Word-normalized harm mentions vary by vendor, with Mistral highest among vendors with at least three files (87.7 per 10,000 words); legal-risk terms are concentrated in copyright, platform/brand, and election/political language. Trust-boundary techniques appear in 35 of 62 vendors, most often non-disclosure alone.Source: §4.4 · pp. 8–9, Figure 8 · p. 10, Figure 9 · p. 10
4.3 Comparison with Officially Published Prompts
- Official Anthropic prompts average 3,080 words and 34.5% safety-policy content, while their matched leaked counterparts average 20,215 words and 57.2% tool/protocol content.The leaked versions average 6.6× expansion; official text is contained at 76–94% for four pairs, with Sonnet 4.6 an outlier at 38%.Source: §4.3 · p. 7, Table 4 · p. 9
4.5 Version Growth, Instruction Tensions, and Autonomy
- Leaked-chain turnover is much higher than turnover in the dated official Claude chains, but measures instability in the leak record rather than direct production change.Mean per-step turnover is 15,049 words in the leaked-only Claude chain versus 1,319 for official Sonnet and 1,424 for official Opus; prompts also contain hardcoded rot-risk markers and co-occurring instruction tensions that do not prove local contradiction.Source: §4.5 · pp. 10–11, Figure 10 · p. 11
4.7 External-Tool Cross-Check
- PromptLint and textstat broadly agree with the operational and maintenance-debt findings, but generic text tools transfer imperfectly to prompts.PromptLint flags over-length sentences in 95% of prompts; textstat finds difficult-word share falls as prompts grow (Spearman −0.77), which the authors interpret as added repetitive, plain-vocabulary tool text.Source: §4.7 · pp. 12–13
4.6 Refusal Semantics
- 1,998 refusal-context lines remain uncategorized by the current lexicon, and refusal instructions are not the largest measured category.Source: §4.6 · p. 11, Figure 13 · p. 13
5 Discussion and Limitations
The discussion characterizes leaked prompts as operational specifications and distinguishes literal reuse from semantic or structural convergence. It also treats prompt maintenance as an engineering problem while emphasizing limits in provenance and detector validity.
Discussion and Limitations
- Operational instructions dominate prompts, while safety and legal language occupy a smaller share; leaked prompts therefore read more like specifications than values documents.The Anthropic matched pairs show public prompts are safety-led while leaked counterparts contain much more tool infrastructure, averaging 6.6× the public prompt length.Source: §5 · p. 13
- Verbatim transfer is concentrated in a small coding-agent cluster, while strong cross-vendor structural convergence usually has near-zero literal overlap.The Anthropic Chrome-extension and Perplexity browser assistants converge in wording, meaning, and architecture.Source: §5 · pp. 13–14
- Official Anthropic releases turn over 1,300–1,400 words on average, while the leaked-only chain shows more than 10× that turnover, concentrated in tool/protocol and legal/IP blocks.The paper warns that leak-derived version chains should not be treated as production histories; it identifies stale references and contradictory directives as maintenance risks.Source: §5 · p. 14
- Most files are adversarial and unverified, so the corpus does not measure official vendor policy; the analysis is textual and cannot establish effects on deployed behavior.The classifier audit was keyword-scored rather than independently human-coded, and manual labels are single-coder judgments.Source: §5 · p. 14
6 Conclusion
The conclusion presents the merged corpus as dominated by operational specifications rather than safety, legal, refusal, or injection-boundary instructions. It recommends treating prompts as configuration-like artifacts while recognizing that matched public/leaked comparisons are limited and leak-derived histories need verified baselines.
Conclusion
- Operational specification dominates the corpus, while safety, legal, refusal, and injection-boundary instructions occupy a smaller measured share.The paper therefore characterizes leaked prompts as configuration-like text rather than statements of values.Source: §6 · pp. 14–15
- Five matched Anthropic releases show a 5–8× expansion and a safety-to-tooling inversion, but no other vendor has a comparable dated archive here.The paper treats this comparison as calibration rather than generalization of the corpus-wide findings.Source: §6 · p. 15
- The conclusion recommends versioning, regression tests, and linting for stale references and contradictions, with red-teaming focused on tool-call and data-handling boundaries.It calls for more official/leaked matched pairs and tests of whether prompt-level differences predict model behavior.Source: §6 · p. 15