Source-linked AI summary
The Ethics of Prompt Engineering
Federico Germani, Giovanni Spitale
TL;DR
Prompt engineering is often treated as a technical way to elicit useful outputs, but prompts also shape what information and assumptions enter model responses. This paper contrasts knowledge generation with knowledge injection and argues for responsible prompting principles, while emphasizing that prompting cannot by itself ensure truthful or unbiased outputs.
Problem
Prompting is often treated as technical optimization, although it also shapes the information, assumptions, and perspectives in model outputs.
Source: Prompting as value injection · p. 4Method
The paper distinguishes knowledge generation from knowledge injection and proposes five principles for responsible prompting.
Source: Knowledge injection versus knowledge… · p. 3, Core principles of responsible prompting · p. 5Results
The paper argues that knowledge injection is a more appropriate model for responsible use and identifies epistemic humility, uncertainty handling, source sensitivity, non-deceptive framing, and recognition of model limits and task allocation as core principles.
Source: Knowledge injection versus knowledge… · p. 3, Core principles of responsible prompting · p. 5Takeaways & Limitations
Responsibility for AI-generated outputs is distributed across actors, and responsible prompting is one layer of epistemic responsibility rather than a complete solution.
Source: The limits of responsible prompting:… · pp. 8, 10Prompts cannot correct biased or incomplete training data or guarantee truth, and careful prompting cannot fully compensate for deficiencies in models, retrieval, governance, or users' capacity to assess outputs.
Source: The limits of responsible prompting:… · pp. 8, 10
Abstract
from arXiv · showhide
Prompt engineering is commonly understood as a technical practice aimed at eliciting useful, accurate, and appropriately structured outputs from large language models (LLMs). Yet prompts do more than optimize model performance: they shape what information is elicited, which assumptions are introduced, and which perspectives are privileged. In this paper, we argue that prompting should therefore be understood as an epistemic and normative act. We distinguish between two conceptions of prompting: knowledge generation, in which the model is implicitly treated as an epistemic authority, and knowledge injection, in which users provide relevant knowledge, evidence, and context that the model can organize, articulate, and extend. We contend that the latter provides a more appropriate model for responsible LLM use, since prompting cannot substitute for subject-matter expertise or the critical evaluation of model outputs. At the same time, prompts inevitably inject values, intentions, and assumptions into the generation process. Building on these premises, we propose five principles for responsible prompting: epistemic humility, explicit engagement with uncertainty and epistemic underdetermination, source sensitivity, avoidance of deceptive framing, and recognition of model limitations and appropriate task allocation. We further argue that responsibility for AI-generated outputs is distributed across users, developers, institutions, and societies, and that some applications cannot be rendered ethical through better prompting alone.
Knowledge injection versus knowledge generation
The paper contrasts treating prompts as requests for knowledge generated by an authoritative-seeming model with using them to ground generation in explicit, evaluable information. It argues that expertise helps users supply context and assess outputs, while responsible prompting also requires safeguards and system designs that support users without domain expertise.
Knowledge injection versus knowledge generation
- Knowledge injection grounds generation in explicit information that the model can process, structure, and articulate, rather than treating it as an epistemic authority.The information may come from user expertise or external sources and tools; the distinction is whether its evidentiary basis is made explicit, constrained, and open to evaluation.Source: Knowledge injection versus knowledge… · p. 3
- Users with little topic knowledge have limited ability to determine whether a model's account is accurate, balanced, or complete.Retrieval can help identify primary sources and distinguish stronger from weaker evidence, but users still need to evaluate the resulting account.Source: Knowledge injection versus knowledge… · p. 3
- Subject-matter expertise helps users provide context, spot missing information and errors, and critically evaluate model outputs, but it is not required to formulate productive prompts.Non-experts can ask for competing arguments, primary sources, evidentiary distinctions, and areas without consensus.Source: Knowledge injection versus knowledge… · p. 3
- Users with lower AI and information-retrieval literacy may struggle to detect inaccurate citations, poor source selection, omitted evidence, or misleading conclusions.This vulnerability can be reinforced by automation bias.Source: Knowledge injection versus knowledge… · p. 4
- When users lack subject knowledge, the epistemic burden shifts toward retrieval constraints, source transparency, verification, and systems that make evidence easier to inspect.The paper frames prompting as involving decisions about sources, uncertainty, and output evaluation, not just question formulation.Source: Knowledge injection versus knowledge… · p. 4
Prompting as value injection
Prompts convey values and assumptions as well as information, shaping what counts as a good output and whose perspectives are represented. The paper warns that predetermined or one-sided framing can make an output appear objective while reinforcing the user's prior beliefs.
Prompting as value injection
- Prompts embed judgments about the desired output, acceptable trade-offs, represented perspectives, and intended audience.Even a prompt designed to produce persuasive content has a particular rhetorical goal.Source: Prompting as value injection · p. 4
- Asking a model to confirm a predetermined conclusion can manufacture the appearance of evidence rather than seek information.The paper gives prompts presupposing conclusions about climate lockdowns or migration controls as examples of loaded framing.Source: Prompting as value injection · p. 4
- When users present belief-confirming output as objective analysis, the paper characterizes this as epistemic deception, whether intentional or not.Because models may mirror embedded assumptions persuasively, such outputs can contribute to confirmation bias and reinforce false beliefs.Source: Prompting as value injection · pp. 4–5
Core principles of responsible prompting
Responsible prompting requires epistemic commitments that shape how users supply and evaluate information, while recognizing that prompting cannot by itself resolve model, institutional, or user-capacity limitations. Responsibility for outputs is distributed among actors according to their influence over the process.
Core principles of responsible prompting
- Epistemic humility requires users to elicit caution rather than assume models will surface uncertainty, and to supply scrutinized knowledge or explore competing positions.Prompts should resist turning assumptions into apparently justified conclusions.Source: Core principles of responsible prompting · p. 5
- When evidence is incomplete or supports multiple interpretations, prompts should distinguish established findings from speculation and invite the model to surface uncertainty and disagreement.Otherwise, fluent outputs can obscure uncertainty or underdetermination.Source: Core principles of responsible prompting · p. 6
- Because claim provenance may be opaque and citations may be fabricated or misrepresented, users should request evidentiary distinctions and verify references against primary sources.Models can assist with locating and organizing information but cannot replace independent source verification.Source: Core principles of responsible prompting · pp. 6–7
- Role prompts can change model behavior, but using them to manufacture expertise or credibility or steer misleading content constitutes deceptive framing.The effects of role prompting vary across models, tasks, and contexts; reported jailbreak strategies include impersonation and hypothetical framing.Source: Core principles of responsible prompting · p. 7
- Users should match tasks to model capabilities, using LLMs for support while retaining human responsibility for verification, normative evaluation, and consequential decisions.Careful prompts cannot correct biased or incomplete training data, guarantee truth, eliminate hallucinations, or compensate fully for deficiencies in design, retrieval, governance, or users' assessment capacity.Source: Core principles of responsible prompting · pp. 7–8, The limits of responsible prompting:… · pp. 8, 10
- Responsibility is distributed across developers, institutions, users, and societies, with each actor's obligations corresponding to its influence over AI generation and use.Distributed responsibility does not mean that all actors bear it equally.Source: The limits of responsible prompting:… · pp. 8–9
- In a deceptive medical or policy argument, model design and selective evidence, institutional verification practices, and the user's framing and presentation can each contribute to epistemic failure.The example shows why the failure cannot be attributed to prompting alone.Source: The limits of responsible prompting:… · p. 9
Ethics as effectiveness
The paper distinguishes epistemic effectiveness from merely achieving a user's objective: responsible prompting supports evidence-representative, uncertainty-aware outputs suited to knowledge work. Its climate-mobility example contrasts a predetermined conclusion with an open request for perspectives and evidential limits.
Ethics as effectiveness
- Responsible prompting can contribute to epistemic effectiveness when it supports accurate representation of evidence, communicates uncertainty, and avoids misleading framing.Effectiveness here concerns knowledge-seeking, evaluation, or communication, not simply achieving the user's objective.Source: Ethics as effectiveness · p. 10
- An open request to summarize perspectives on climate-related mobility restrictions is more likely to yield rigorous, useful analysis than a request to show they are necessary and inevitable.The alternative asks the model to distinguish evidential support from political or ethical assumptions.Source: Ethics as effectiveness · p. 10
- The paper presents ethical improvement and epistemic improvement as aligned when prompting seeks to understand evidence and disagreement rather than confirm assumptions.It characterizes responsible prompting in these contexts as prompting well.Source: Ethics as effectiveness · p. 10