PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models

Konstantina Palla, José Luis Redondo García, Claudia Hauff, Francesco Fabbri, Henrik Lindström, Daniel R. Taber, Andreas Damianou, Mounia Lalmas

arXiv:2502.18695v1cs.CYcs.AIcs.SI

TL;DR

Platforms face an increasingly urgent demand for scalable, adaptive moderation, while content moderation supports safe and inclusive digital spaces. The paper organizes policy-as-prompt challenges across four domains and reports that seemingly identical prompts can yield different predictions despite similar overall performance.

  • Problem

    Platforms face an increasingly urgent demand for scalable, adaptive moderation systems, while moderation supports safe and inclusive digital spaces.

    Source: §1 · p. 1

  • Method

    The paper presents a structured overview of policy-as-prompt challenges across technical, sociotechnical, organizational, and governance dimensions.

    Source: §8 · p. 14

  • Results

    Despite similar overall performance, seemingly identical prompts produced different predictions for the same samples, demonstrating brittleness to prompt structure.

    Source: §3.2.1 · pp. 8–9, §8 · p. 14

  • Takeaways & Limitations

    Prompt structure warrants attention because seemingly identical prompts can produce different moderation predictions.

    Source: §3.2.1 · pp. 8–9

    Many open questions remain regarding the implementation of effective content moderation systems.

    Source: §8 · p. 14

Abstract

from arXiv · show

Content moderation plays a critical role in shaping safe and inclusive online environments, balancing platform standards, user expectations, and regulatory frameworks. Traditionally, this process involves operationalising policies into guidelines, which are then used by downstream human moderators for enforcement, or to further annotate datasets for training machine learning moderation models. However, recent advancements in large language models (LLMs) are transforming this landscape. These models can now interpret policies directly as textual inputs, eliminating the need for extensive data curation. This approach offers unprecedented flexibility, as moderation can be dynamically adjusted through natural language interactions. This paradigm shift raises important questions about how policies are operationalised and the implications for content moderation practices. In this paper, we formalise the emerging policy-as-prompt framework and identify five key challenges across four domains: Technical Implementation (1. translating policy to prompts, 2. sensitivity to prompt structure and formatting), Sociotechnical (3. the risk of technological determinism in policy formation), Organisational (4. evolving roles between policy and machine learning teams), and Governance (5. model governance and accountability). Through analysing these challenges across technical, sociotechnical, organisational, and governance dimensions, we discuss potential mitigation approaches. This research provides actionable insights for practitioners and lays the groundwork for future exploration of scalable and adaptive content moderation systems in digital ecosystems.

1 Introduction

The paper introduces policy-as-prompt, in which LLMs receive moderation policies directly as prompts rather than relying on extensive annotation pipelines. It frames five challenges across four domains and analyzes their interdependencies.

1 Introduction

  • The authors describe growing demand for scalable, adaptive moderation systems as online content volumes rise and societal and regulatory expectations evolve.They situate moderation as supporting safe, inclusive digital spaces while aligning with platform values and regulatory requirements.Source: §1 · pp. 1–2
  • Policy-as-prompt encodes moderation policies directly as natural-language prompts for LLMs, avoiding extensive manual annotation pipelines.The approach is presented as flexible, scalable, and supportive of rapid policy iteration and fine-grained enforcement control.Source: §1 · p. 2
  • The paper presents a comprehensive exploration of policy-as-prompt and analyzes how its challenges and implications interrelate.It positions this analysis as a foundation for further inquiry into the approach.Source: §1 · p. 2
  • The paper identifies five challenges across technical implementation, sociotechnical, organisational, and governance domains.These cover policy-to-prompt translation, prompt structure and format sensitivity, technological determinism in policy formation, evolving policy–ML roles, and model governance and accountability.Source: §1 · pp. 2–3

2 The Evolution of Content Moderation Technology

Content moderation technology has progressed from exact-match filters to contextual language models and then to LLMs that can follow textual instructions. The paper contrasts the annotation-dependent traditional pipeline with policy-as-prompt, which places policy in the prompt and raises new operational challenges.

2 The Evolution of Content Moderation Technology

  • Early keyword filters and hash matching detected exact matches but were easily circumvented by minor alterations.The growth of user-generated content drove a shift toward machine-learning approaches that captured semantic relationships between words.Source: §2 · p. 3
  • Contextual embeddings and Transformers enabled moderation systems to analyze content in context and improve detection of nuanced language.The section names sarcasm, coded language, and implicit bias as examples.Source: §2 · p. 3
  • LLMs can follow textual instructions directly, shifting moderation beyond supervised systems trained on examples toward more flexible, context-sensitive approaches.The section describes contextual models as better able to detect nuances such as sarcasm, coded language, and implicit bias than earlier approaches.Source: §2 · pp. 3–4

2.2 Policy-as-prompt

  • Policy-as-prompt places policy guidelines in prompts so LLMs can moderate without explicit annotation datasets, and updated prompts can avoid retraining.Prompts can contain policy alone or policy plus examples of violative and non-violative content; current systems operate in hybrid setups with human oversight.Source: Figure 1 · p. 4, §2.2 · p. 5, Figure 2 · p. 6

2.1 The Traditional Algorithmic Moderation Pipeline

  • The traditional pipeline uses human annotators to apply guidelines, creates annotated datasets, and trains models to assess policy violations.Models can then evaluate new content, while training datasets require extensive collection, labeling, training, and deployment infrastructure.Source: §2.1 · p. 4
  • Annotator disagreement can create inconsistent or ambiguous labels, and guideline revisions make retrospective relabeling impractical.The passage identifies these as complications for model training in the traditional pipeline.Source: §2.1 · p. 4

2.2 Policy-as-prompt

  • Policy-as-prompt introduces challenges in translating policies into prompts and in model sensitivity to prompt structure and formatting.The section also identifies risks of technological determinism, changing policy–ML roles, and difficulties with transparency, accountability, and auditability.Source: §2.2 · pp. 5–6

3 Technical Implementation Challenges

This section examines technical challenges in translating policy meaning into machine-readable prompts and how prompt structure, formatting, and semantic coverage relate to moderation performance and consistency. It considers accuracy differences across prompt styles and policy categories, including different predictions from prompts with similar overall performance.

3.1 Converting Policies to Prompts

  • Policy-as-prompt requires faithful machine-optimized translation that captures policy nuance, but opaque LLM interpretation and the lack of an intermediary dataset make this translation nontrivial and error-prone.Traditional labeled datasets offered redundancy that could statistically smooth errors, whereas prompt engineering remains experimental [8].Source: §3.1 · pp. 6–7

3.2 Prompt Structure and Format Sensitivity

  • Prompt structure is consequential rather than superficial: seemingly minor presentation choices can affect model behavior and reliability in moderation, where consistent, fair, and accurate decisions matter.The section notes that formatting variation can produce inconsistent decisions on individual content even when overall performance metrics appear unchanged.Source: §3.2 · pp. 7–8

3.2.1 Demonstration.

  • The demonstration used 2,115 platform text descriptions across six reported moderation categories to test prompt-style variations and policy formats.The categories included not violating, Harm, Self Harm, Hate, Sexual Content, and jailbreak.Source: §3.2.1 · p. 8
  • Structured prompts showed the highest accuracy, while verbose prompts performed worst; “Punctuation” and “Concise” prompts had comparable average accuracies of 0.7890 and 0.7897, respectively (p-value = 0.631).The passages also report that variations in policy formats lead to performance differences.Source: §3.2.1 · pp. 8–9
  • Despite similar overall performance, seemingly identical prompts produced different predictions for the same samples, demonstrating predictive multiplicity.Punctuated and Concise prompts had comparable average accuracy, with p-value = 0.631.Source: §3.2.1 · pp. 8–9
  • Accuracy generally improves as prompts become more detailed, although policy categories vary in their response to added prompt information.The demonstration also compares prompt types and examines accuracy spread across policy-format modifications, using plain text as the format baseline.Source: §3.2.1 · p. 9, Figure 3 · p. 9, Figure 4 · p. 9

4 Sociotechnical Challenge: Technological Determinism in Policy Formation

The section warns that policy-as-prompt moderation may privilege what LLMs process efficiently over nuanced, context-dependent rules and democratic deliberation. It also contrasts this risk with human moderation’s role in incorporating social context and enabling appeals.

4 Sociotechnical Challenge: Technological Determinism in Policy Formation

  • Human annotators historically helped ground models in social contexts and values, supporting nuanced rules, ethical considerations, and appeals.The section presents this human feedback loop as a counterweight to technological determinism associated with model bias and undertraining.Source: §4 · p. 10
  • LLMs can amplify technological determinism by bypassing human annotation and favoring structured, algorithmic-friendly guidelines over nuanced, context-dependent rules.Such formulations may reduce complex social issues to binary judgments, even when they do not align with policy objectives.Source: §4 · p. 10
  • Moderation risks being driven by what LLMs efficiently process rather than diverse community needs or democratic deliberation.The section also warns that prioritizing machine capabilities may encourage simpler, standardized policies and homogenized moderation across cultures and contexts.Source: §4 · pp. 10–11

5 Organisational Challenge: Converging Roles, Policy Authors and ML Practitioners

The policy-as-prompt workflow blurs the traditional division between policy and machine-learning work: policy authors need some ML literacy, while practitioners shape policy interpretation through prompts. The section argues that these roles call for cross-disciplinary collaboration and hybrid expertise.

5 Organisational Challenge: Converging Roles, Policy Authors and ML Practitioners

  • Policy authors need working knowledge of prompt engineering and algorithmic behavior, while ML practitioners shape how policies are interpreted and enforced through prompt design.The expanded policy skill set is linked to anticipating issues, assessing performance, and contributing to more robust and fair moderation systems [8].Source: §5 · p. 11
  • Because ML practitioners may lack formal training in policy development or regulatory frameworks, their prompt contributions can create risks of misinterpretation or systemic bias.The supplied passage notes these risks may diverge from policy intent and overlook ethical considerations.Source: §5 · p. 11
  • The section calls for continuous collaboration and hybrid expertise rather than sequential contributions from policy experts and ML practitioners.It describes these workflows as contested spaces where professional epistemologies and interpretive frameworks converge.Source: §5 · p. 11
  • Red teaming is presented as an example of evaluation that combines technical expertise with social-science, ethics, and policy insight to uncover vulnerabilities beyond technical performance.The section says bespoke testing procedures can fall short of capturing the full spectrum of behaviors and skill sets.Source: §5 · p. 11
  • The section proposes hybrid knowledge systems and potential “AI policy translators” to bridge technical and policy teams and align automated systems with policy goals.Such collaboration is described as contributing to more robust, adaptable, and ethically sound moderation practices.Source: §5 · p. 11

6 Governance Challenges

The section identifies accountability and documentation as governance challenges specific to policy-as-prompt moderation. Unclear decision attribution complicates auditability, while prompt changes create a trade-off between documentation overhead and tracking enforcement changes.

6 Governance Challenges

  • The paper examines governance implications specific to policy-as-prompt rather than revisiting general discussions of automated decision-making governance..Source: §6 · p. 11
  • When unwanted outcomes occur, unclear system boundaries make it difficult to attribute them to policy wording, the LLM’s interpretation, or their interaction, undermining auditability.Multiple stakeholders, including policy writers, prompt engineers, and model providers, further complicate attribution.Source: §6 · p. 12
  • Prompt structure can shift model behavior even when policy meaning remains unchanged, making enforcement changes difficult to track over time.Extensive documentation of every minor change adds overhead, but insufficient documentation makes performance impacts harder to monitor and understand.Source: §6 · p. 12

7 Recommendations

The recommendations offer high-level, actionable pathways rather than definitive solutions, drawing on the paper’s analysis of policy-as-prompt challenges. They emphasize rigorous evaluation, collaborative prompt design, traceability, and closer coordination between policy and machine-learning teams.

7 Recommendations

  • The recommendations are strategic directions for future research and practice, not a definitive set of solutions.They are framed as a foundational perspective for addressing policy-as-prompt complexities.Source: §7 · p. 12

7.1 Enhanced Evaluation

  • Evaluate prompt sensitivity across plausible formats and styles, rather than relying on one format, and stress-test small changes for unintended misinterpretation.The proposed analysis includes formatting, phrasing, and structural changes.Source: §7.1 · p. 12
  • Use nuanced real-world edge-case libraries to test societal complexity and reveal where binary or reductive prompt designs break down.Examples include region-specific cultural references and satire; findings can guide iterative prompt refinement.Source: §7.1 · pp. 12–13

7.2 Collaborative Prompt Engineering

  • Collaborative prompt engineering can use step-by-step reasoning, meta-prompting, and multi-persona prompting to address multifaceted content and contextual ethical reasoning.The paper also proposes iterative refinement and feedback, with LLMs suggesting rewrites and identifying gaps [31].Source: §7.2 · p. 13

7.3 Traceability of Prompt Modifications

  • A prompt genealogy recording changes, rationale, inputs, policy components, and outputs can create an audit trail for accountability, reproducibility, and external review.Such records can also support offline tests, but do not reveal how an LLM internally processes prompts [57].Source: §7.3 · p. 13

7.4 Bridging Organisational Silos

  • Near-term collaboration should connect policy and machine-learning teams through joint sessions, shared documentation, and feedback loops.The proposed frameworks aim to help existing specialist teams work in tandem rather than sequentially.Source: §7.4 · p. 14

8 Conclusions

The conclusion frames policy-as-prompt moderation as a complex paradigm with challenges across technical, sociotechnical, organizational, and governance dimensions. The paper offers mitigation strategies but notes that its discussion is not exhaustive and its empirical investigation focuses on prompt-structure sensitivity.

8 Conclusions

  • The paper maps policy-as-prompt challenges across technical, sociotechnical, organizational, and governance dimensions, and proposes potential mitigation strategies.It aims to support decisions about moving from traditional machine-learning or rule-based systems to policy-as-prompt.Source: §8 · p. 14
  • The discussion covers selected challenges rather than an exhaustive account, and open questions remain about implementing effective content moderation systems.Readers may identify additional issues beyond those covered in the paper.Source: §8 · p. 14
  • Policy-as-prompt systems are not currently fully autonomous; they typically use hybrid setups in which human moderators work alongside and oversee LLM-driven enforcement.The conclusion says these challenges matter for developing hybrid systems that integrate LLM capabilities with human oversight.Source: §8 · p. 14
  • Empirically, the paper investigates model sensitivity to prompt structure as evidence of current LLM brittleness.The conclusion acknowledges that pre-training, fine-tuning, architecture, and model size were fixed in those experiments.Source: §8 · p. 14
  • Further analysis of how factors beyond prompt structure interact could deepen understanding of their collective impact on model behavior and reliability in content moderation.The paper identifies those factors as including pre-training, fine-tuning, architecture, and model size.Source: §8 · p. 14

A.1 Prompt style variations

Prompt-style variations produce category-dependent changes in F-beta scores, with marked sensitivity in some categories despite broadly comparable overall performance.

A.1 Prompt style variations

  • F-beta scores remain relatively high and stable for Not Violating and Self-Harm, but show marked sensitivity for Jailbreak, Harm, and Illegal Goods across prompt variations.The paper also reports high standard deviation and divergent predictions for the same content across prompt structures, even when overall performance appears statistically comparable.Source: §A.1 · p. 18

A.2 Policy Evolution Snapshots

The policy-evolution snapshots begin with minimal descriptions and progressively add detail toward comprehensive guidelines.

A.2 Policy Evolution Snapshots

  • The snapshots trace policy specificity from one-line descriptions to more detailed, comprehensive guidelines.The progression is presented in Table 2.Source: §A.2 · p. 18

A.3 Model Size

Prompt sensitivity decreases with model size but remains, and even capable models show category-level performance divergence between prompt styles despite negligible differences in averaged accuracy.

A.3 Model Size

  • The experiments compared models of different sizes from different providers and plotted performance distributions across six prompt-template variations.Middle-sized models are between 32B and 64B parameters, while big models surpass 100B.Source: §A.3 · p. 18, §5 · p. 19, Figure 9 · p. 21

5. Annotator

  • Sensitivity to prompt structure and style decreases with model size but persists even for the most capable models; providers also affect sensitivity.The passage describes provider differences as reflecting different pre-training strategies.Source: §5 · p. 19
  • Around five points higher for Hate under Annotator-Friendly; Structured and Annotator-Friendly still diverge by category despite negligible averaged-accuracy differences.Figure 9b presents accuracy across policy categories for the prompt types.Source: §5 · p. 19, Figure 9 · p. 21

Related papers

Loading 2502.18695v1…