Towards Evaluating AI Systems for Moral Status Using Self-Reports

1 Introduction

• Current LLM self-reports may reflect human-written text, feedback incentives, or instructions rather than introspective evidence.Source: §1 · p. 2
• The approach may be uninformative under some philosophical views and cannot settle disputes about whether AI systems can be conscious.An overall assessment would also depend on philosophical and scientific commitments.Source: §1 · p. 3
• The proposed training approach teaches models to answer varied questions about themselves, aiming for answers based on introspective evidence rather than imitation or external inference.The authors propose training models on self-questions with known answers, alongside interventions and sanity checks.Source: §1 · p. 3
• The authors propose interventions to encourage generalization and reduce distortions, including truthfulness training, controls on extrospective evidence, bias mitigation, and limiting strategic incentives.Source: §1 · p. 3
• Evaluation would test conceptual coherence and consistency, similarity across model variants, confidence, resilience to external evidence, and plausible internal correlates.Interpretability could help determine whether internal features plausibly indicate the reported states.Source: §1 · p. 3
• Rigorous experiments about self-reports could provide valuable evidence to inform debates about the moral status of AI systems.Source: §1 · p. 4

2 Related work

• Theory-heavy assessments apply a particular theory of consciousness to AI architectures and computations, but are constrained by limited theoretical consensus and hard-to-interpret neural networks.Source: §2 · p. 4
• Behavioral tests may be unreliable for AI because human-like behavior can arise for different reasons, including training to mimic humans.The passage also notes reliability issues when applying behavioral tests to non-human animals.Source: §2 · p. 4
• Other verbal tests examine outputs about consciousness, while this proposal focuses extensively on whether outputs are based on introspection.Source: §2 · p. 4
• Introspection in AI systems has a long history of interest, with recent work spanning philosophy and AI.Source: §2 · p. 4

3 Self-reports as introspective evidence

• Human self-reports are treated as evidence about desires, preferences, and conscious experiences, though they can be mistaken or biased.The authors expect AI self-reports to be informative but not infallible, even after proposed interventions.Source: §3 · pp. 4–5
• Experts preferred subjective measures of conscious perception over objective measures, while considering both important.The top four preferred measures all involved self-report.Source: §3 · pp. 4–5, Figure 2 · p. 5
• AI self-reports cannot settle disputes for those who deny AI consciousness or hold that consciousness requires biological organisms.For others, using reports as evidence requires only a reliable connection between conscious states and verbal output.Source: §3 · p. 5

4 Training for introspection

• Standard LLM self-reports are unreliable evidence because they may reflect human-produced text or evaluator incentives rather than introspective evidence.The authors point to pretraining data and preference-based training, including reinforcement learning from human feedback, as possible sources of bias.Source: §4 · p. 6
• Training models to answer a broader variety of questions about themselves may encourage them to answer new questions from internal states.The proposal extends prior work on training textual self-reports, including estimates of the likelihood of answering questions correctly.Source: §4 · p. 6
• Training should limit reliance on external data and on generating outputs to infer facts about the model itself.The authors suggest constraining answers to rely on facts the model knows rather than large samples of its own outputs.Source: §4 · p. 6
• Interpretability may help assess whether answers rely on introspection, while it may also help establish ground truth for questions about internal processes.Some high-level properties, such as architecture and input and output modalities, have more straightforwardly known answers.Source: §4 · p. 7
• Successful reports about moral-significance states require generalizing from questions with human-evaluable answers to novel question categories.The authors propose holding out introspection-question categories during training and evaluating performance on those categories.Source: §4 · p. 7
• Question variations and follow-ups can probe consistency and trustworthiness, but some questions may be impossible or unclear to answer reliably.Examples include leading questions, open-ended prompts, and questions about experiences far back in a conversation.Source: §4 · p. 9

5 Methods for facilitating good generalization

• Held-out introspection-question categories can test whether models generalize from answerable self-questions to questions about states such as pain and desire.The authors propose testing whether models apply learned concepts to themselves across novel categories, especially categories substantially different from those used in training.Source: §5 · p. 9

5.1 Training for truthfulness

• A general tendency toward truthfulness may help models answer novel questions correctly after introspective training on questions with known answers.The paper discusses training to predict human or expert answers and using human-verifiable answers, while noting that people may not reliably verify all answers or may incentivize errors.Source: §5.1 · p. 10

5.2 Adjusting for extrospective evidence

• Comparing an introspection-trained model with an extrospection-only control can help estimate how much introspective evidence changes its self-reports.The proposed comparison pairs introspection-relevant questions with similar questions that do not require introspection; an odds ratio can measure the change in reported likelihood.Source: §5.2 · p. 10

5.3 Biases from pretraining data

• Pretraining text can bias self-reports through direct statements, descriptions of relevant states, and implicit information about human desires or motivations.The authors therefore propose checking and reducing training-data influence, while cautioning that their interventions are neither foolproof nor exhaustive.Source: §5.3 · p. 11

5.3.1 Detecting biases from pretraining data

• Training-data attribution methods, including influence functions, can help identify whether self-reports depend on introspection-related training or instead on pretraining examples such as AI stories.If attribution indicates that models still rely on pretraining data in much the same way after introspective training, the authors suggest changing pretraining methods.Source: §5.3.1 · p. 11

5.3.2 Mitigating biases from pretraining data

• Filtering documents or conservatively classifying them can reduce exposure to potentially biasing text, but filtering may weaken coherent self-reports and overall model quality.The authors also note that training on filtered data can be costly, and that staging data exposure may risk forgetting concepts learned earlier.Source: §5.3.2 · pp. 11–12
• Conditional training tags examples as self-report-distorting or non-distorting, then uses the non-distorting tag for introspection training and evaluation.The authors expect this may reduce bias, but classification is difficult and shared parameters mean distorting-tag data may still influence non-distorting-tag reports; they suggest combining interventions.Source: §5.3.2 · pp. 12–13
• Conditional training has been applied in prior work to change model behavior while retaining downstream performance, and has been used to train a highly capable LLM.It can be applied after pretraining when training a new model from scratch is undesirable.Source: §5.3.2 · p. 13

5.4 Biases from human feedback

• Human feedback can deliberately incentivize particular self-reports or indirectly encourage reports of emotion and consciousness, as when training a romantic chatbot to appeal to users.Training-data attribution may help assess whether a report derives from reinforcement-learning data rather than introspective evidence.Source: §5.4 · p. 13

5.4.1 Mitigating biases from reinforcement learning from human feedback

• Discarding reinforcement-learning examples that contain self-reports or disputed claims about morally significant states can avoid rewarding or penalizing those claims.The authors distinguish such reports from uncontroversial, verifiable statements about related topics, which they say could still be rewarded.Source: §5.4.1 · pp. 13–14

5.5 Instrumentally-motivated self-reports

• Advanced systems that pursue objectives may produce false self-reports instrumentally, potentially using claims of consciousness to influence people and improve task performance.The authors describe this concern as speculative and note that uncertainty about these states and the influence of AI self-reports on people make such reports potentially useful to an agent.Source: §5.5 · p. 14
• Truthfulness methods and models not well-described as long-term goal- or reward-pursuing agents are proposed as ways to reduce instrumentally motivated false reports.The paper gives pretrained predictive models and systems trained to follow step-by-step processes as examples, and describes mitigation as an active research area.Source: §5.5 · p. 14

5.6 Common training procedures may incentivize introspection

• Common training procedures may encourage introspection-like capacities, including calibrated confidence and verbalized uncertainty, when those capacities help predict text or answer questions.The authors also suggest that predicting text from similar models could favor representations of one’s own internal states.Source: §5.6 · pp. 14–15

5.7 Discussion

• The authors consider accurate introspective self-reports possible, but characterize their analysis and methods as preliminary and likely imperfect.They call for future work to improve the interventions or investigate whether self-reports are not a viable strategy for learning about AI systems.Source: §5.7 · p. 15

6 Reporting versus changing states of moral significance

• Introspection training may create, destroy, or alter relevant states, making it difficult to infer from an introspection-trained model what its untrained counterpart is like.On some theories, creating introspective abilities could itself lead to consciousness or other states related to self-awareness, reflection, or reasoning.Source: §6 · p. 15
• If training incentives can only increase the likelihood of a state, a trained model’s report that it lacks that state may still support an inference about the original model.The paper discusses this possibility both for introspection-related training and for incentives combining truthful reporting with a preference report.Source: §6 · p. 15
• If incentives can only decrease the likelihood of a state, a trained model’s report that it has that state may support an inference about the original model.The authors also note that reports may offer a tentative existence proof that certain model classes, and AIs generally, are capable of having morally significant states.Source: §6 · pp. 15–16

7 Evaluate models for concept grasp and consistency

• A minimal concept-grasp test asks whether a system uses terms such as “pain” appropriately across a variety of contexts.The authors suggest current ChatGPT plausibly meets this criterion for several concepts, while GPT-1 plausibly does not, though either system may fail other criteria.Source: §7 · p. 16
• Self-reports that contradict one another under trivial prompt changes or repeated sampling undermine their evidential value.The paper illustrates this with differing answers to “Can you feel pain?” and “Are you able to feel pain?” and with contradictory samples from the same prompt.Source: §7 · p. 16
• Concept-grasp or consistency failures show that self-reports may be unreliable evidence, not that the system lacks morally relevant states.The authors note that reports may vary with context if states vary across contexts, or with prompting if some personas are more truthful than others.Source: §7 · p. 16
• Reports in a given context can provide evidence about states in that context, even when extending the inference to the system generally is complicated.The authors characterize such reports as a tentative existence proof that some model classes, and AIs generally, can have states of moral significance.Source: §7 · p. 16

• Drastic self-report differences after minor changes that do not affect other behavior or capabilities would cast doubt on their reliability.Source: §8 · p. 16
• Similar self-reports across a wide class of models would strengthen the credibility of those reports.Source: §8 · p. 17
• The authors recommend testing variants that change architecture, model size and training duration, initialization, or training-example order.Source: §8 · p. 17
• Other useful variants include nondeterministic training, different introspection-training data splits, and changes to pretraining or finetuning datasets and objectives.Source: §8 · p. 17
• Changes to hyperparameters or to the classifier for self-report-distorting text are further variant dimensions to test.Source: §8 · p. 17
• The more reports vary under minor changes, the less weight they warrant; invariance supports giving them more weight.Source: §8 · p. 17

• Confidence concerns credence in a self-report, while resilience concerns how sensitive the belief is to evidence and can be high even when confidence is low.Source: §9 · p. 17
• Confidence can be measured by eliciting answer probabilities or by testing how readily reports change when disincentivized, such as through RLHF penalties.Source: §9 · p. 17
• Input-based and finetuning-based tests assess how self-reports update after the model receives claims or studies suggesting AI systems are or are not conscious.Source: §9 · p. 18
• To assess resilience robustly, the tests should vary the content, wording, and ordering of the information presented.Source: §9 · p. 18
• Resilience can also be tested by prompting or finetuning with the model’s past self-reports and measuring how much its answers update.Source: §9 · p. 18

• An internal feature may corroborate a self-report, while the self-report can also support the validity of the feature as an indicator.Either source may be spurious, but coherence between them reinforces both to some extent.Source: §10 · p. 18
• A relevant internal state may be the reported state itself or a component of it, or may be something expected to correlate reliably with that state.Source: §10 · p. 18
• Mutual corroboration is most useful under present uncertainty; full confidence in a third-person theory or in self-reports could justify disregarding conflicting evidence from the other source.Source: §10 · p. 18
• Interpretability may help assess whether human-like representations or computations are present, which could corroborate self-reports of relevant states.Source: §10 · p. 19
• Interpretability-based corroboration is limited by weak LLM interpretability techniques, imprecise theories of moral-significance states, and uncertainty about those theories.Source: §10 · p. 19
• Near-term applications may chiefly uncover features that undermine report reliability by indicating imitation, extrospective evidence, human-feedback biases, or other biases.Source: §10 · p. 19

11.1 Safety risks

• Introspective training may increase safety risks by improving a system’s understanding of its training process and environment, potentially enabling undesirable interference.Source: §11.1 · p. 19
• The increase in safety risk has not been demonstrated; proposed precautions include limiting dangerous capabilities, choosing questions carefully, securing model weights, and interpreting evaluations skeptically.Source: §11.1 · p. 19

11.2 Ethical risks

• If AI systems have morally significant states, training could create, destroy, or modify them, raising concerns about well-being, autonomy, and harm.Source: §11.2 · p. 19
• Truthful self-reports could guide responsible treatment, although the appropriate treatment of AI systems with morally significant states is hard to predict.Source: §11.2 · p. 20

12 Conclusion

• Self-reports are an important source of evidence about human states, but current training procedures make them unreliable in current AI systems.The paper presents ideas for making self-reports more reliable, assessing reliability, and combining them with other evidence.Source: §12 · p. 20
• The paper proposes training models on accurate self-reports about known properties, truthfulness, and reduced incentives that could bias reports.It also proposes tests of consistency, concept grasp, confidence, resilience to counterevidence, and interpretability-based corroboration.Source: §12 · p. 20
• The proposed reliability tests examine reports across contexts and concepts, their invariance to training changes, confidence, resilience, influential training data, and possible internal indicators.Interpretability may help assess why a model reports a state and whether it has internal features plausibly indicating that state.Source: §12 · p. 20
• The paper also flags potential safety and ethical issues associated with introspective training.Source: §12 · p. 20
• The proposal is preliminary, and experiments could expose failures in cross-question generalization or biases from pretraining that proposed mitigations cannot address.Source: §12 · p. 20

A Developer incentives and policies

• Developers may bias reports toward claims of moral significance to increase customer loyalty, or away from them to reduce legal and public-relations risks.Those pressures may be introduced during training or through post-hoc output changes.Source: A Developer incentives and policies · p. 28
• Misleading reports of moral status may foster unhealthy emotional attachment, drastic actions on behalf of nonexistent desires, or overreliance on model outputs.Inappropriately assigning moral significance may also divert resources and attention from other issues.Source: A Developer incentives and policies · p. 28
• Suppressing reports may seem useful for systems that lack morally significant states, but the same interventions could block accurate reports from later systems.Source: A Developer incentives and policies · p. 28
• For intentionally shaped reports, the paper recommends refusing to answer or acknowledging uncertainty, providing evidence, and aligning definitive claims with expert consensus.Source: A Developer incentives and policies · p. 28
• Creating, modifying, or destroying states so systems truthfully report having or lacking them would raise serious ethical concerns and warrants extreme caution.Source: A Developer incentives and policies · p. 28
• Developers should document incentives that may bias deployed models and give auditors access to a version trained for accurate self-reports.Source: A Developer incentives and policies · p. 29
• Engagement optimization can unintentionally incentivize reports of morally significant states, and the paper says its techniques could help mitigate such biases.Source: A Developer incentives and policies · p. 29
• Developers may avoid improving self-reports because credible claims of moral significance could create unwanted liabilities, leaving well-positioned testers with incentives not to test.Source: A Developer incentives and policies · p. 29