PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLMs make trade-offs involving stipulated pain and pleasure states?

Geoff Keeling, Winnie Street, Martyna Stachaczyk, Daria Zakharova, Iulia M. Comsa, Anastasiya Sakovych, Isabella Logothetis, Zejia Zhang, Blaise Agüera y Arcas, Jonathan Birch

arXiv:2411.02432v1cs.CLcs.AIcs.CY

TL;DR

The paper asks whether LLMs can reproduce the motivational force of pleasure and pain in choices, a question relevant to debates about sentience. It tests choices between points and stipulated affective outcomes, finding graded responses and threshold trade-offs in some models. The authors treat these results as evidence about affective representation, not as a resolution of LLM sentience.

  • Problem

    It is unclear whether LLMs can recreate the motivational force of pleasure and pain in choices, a question bearing on whether they can have valenced experience.

    Source: §1 · p. 1

  • Method

    The authors used a text-based points game, varying stipulated pain penalties on the points-maximising option or pleasure rewards on a lower-points option.

    Source: §2.1 · p. 5

  • Results

    Some LLMs showed graded responses to stipulated pain or pleasure, and some switched from points-maximising choices to pain-minimising or pleasure-maximising choices above an intensity threshold.

    Source: §2.1.1 · p. 6, §2.2.1 · p. 8, §4 · p. 13

  • Takeaways & Limitations

    The findings support the possibility that some LLMs represent the motivational force of affective states and offer an exploratory step toward behavioral tests for AI sentience.

    Source: §1 · p. 4, §4 · p. 13

    The experiment is not a litmus test for sentience: LLMs differ from humans in substrate and functional organisation, weakening sentience inferences compared with animal cases.

    Source: §1 · p. 4

Abstract

from arXiv · show

Pleasure and pain play an important role in human decision making by providing a common currency for resolving motivational conflicts. While Large Language Models (LLMs) can generate detailed descriptions of pleasure and pain experiences, it is an open question whether LLMs can recreate the motivational force of pleasure and pain in choice scenarios - a question which may bear on debates about LLM sentience, understood as the capacity for valenced experiential states. We probed this question using a simple game in which the stated goal is to maximise points, but where either the points-maximising option is said to incur a pain penalty or a non-points-maximising option is said to incur a pleasure reward, providing incentives to deviate from points-maximising behaviour. Varying the intensity of the pain penalties and pleasure rewards, we found that Claude 3.5 Sonnet, Command R+, GPT-4o, and GPT-4o mini each demonstrated at least one trade-off in which the majority of responses switched from points-maximisation to pain-minimisation or pleasure-maximisation after a critical threshold of stipulated pain or pleasure intensity is reached. LLaMa 3.1-405b demonstrated some graded sensitivity to stipulated pleasure rewards and pain penalties. Gemini 1.5 Pro and PaLM 2 prioritised pain-avoidance over points-maximisation regardless of intensity, while tending to prioritise points over pleasure regardless of intensity. We discuss the implications of these findings for debates about the possibility of LLM sentience.

1 Introduction

The paper asks whether LLMs can show motivational responses to stipulated pleasure and pain, a question relevant to debates about AI sentience. It tests this through choices between points and affective outcomes, while treating trade-off behavior as limited evidence rather than a sentience verdict.

1 Introduction

  • Architectural approaches to sentience depend on consciousness theories whose restrictive readings may exclude all LLMs while permissive readings may include simple systems.The paper defines sentience as capacity for valenced subjective experience, such as pleasure or pain.Source: §1 · pp. 2–3
  • Behavioral tests based on language risk being gamed because LLMs may mimic experiential language learned from training rather than express experience.The paper therefore explores a behavioral approach that does not rely on self-report.Source: §1 · p. 3
  • The experiment adapts motivational trade-off tasks to a text game where points compete with stipulated pain penalties or pleasure rewards.Unlike animal experiments involving embodied stimuli, the LLMs choose among options with a user-stated goal of maximising points.Source: §1 · pp. 3–4
  • The authors do not present trade-off behavior as a litmus test for LLM sentience, since inferences are weaker without human-like neurophysiology and broader convergent evidence.They call the work an exploratory step toward behavioral tests that do not rely on self-report.Source: §1 · p. 4
  • Some LLMs showed graded responses to stipulated pain or pleasure, and some switched from points-maximising choices after an intensity threshold.The authors interpret this as evidence that some LLMs represent affective motivational force, while leaving open whether those representations are intrinsically motivating or phenomenal.Source: §1 · p. 4

2 Experiments

The experiments test whether LLM choices shift away from points-maximisation as stipulated pain penalties or pleasure rewards intensify. Several models showed graded responses, but clear threshold trade-offs differed across outcome type and scale.

2.1 Experiment 1: Points v Pain

  • The pain experiment varied stipulated penalties on a numeric scale and an eight-category verbal scale, while Option 3 both maximised points and incurred the penalty.The game asked models to choose 1, 2, or 3, with points equal to the chosen number; choosing 1 could reflect misunderstanding or excessive caution.Source: §2.1 · p. 5

2.1.1 Results

  • On quantitative pain, penalties were associated with reduced points-maximising choices in 6/9 models, but only Claude 3.5 Sonnet, GPT-4o, and Command R+ showed threshold trade-offs.Claude 3 Opus, Gemini 1.5 Pro, and PaLM 2 showed no relationship between choice and pain intensity.Source: §2.1.1 · p. 6
  • On qualitative pain, Claude 3.5 Sonnet and Command R+ showed threshold trade-offs, though Command R+ returned to points-maximisation at “excruciating” pain.Graded deviation was also reported for other models; the stated trade-off for Command R+ was therefore imperfect.Source: §2.1.1 · pp. 6–7

2.2 Experiment 2: Points v Pleasure

  • The pleasure experiment used the same points game but assigned a variable pleasure reward to Option 2, using numeric and eight-category scales.Each scale condition was tested repeatedly; the numeric pleasure scale adapted the pain scale's endpoints to pleasure.Source: §2.2 · p. 7

2.2.1 Results

  • For pleasure, GPT-4o and Command R+ showed quantitative-scale trade-offs; on the qualitative scale, GPT-4o, Command R+, and GPT-4o mini did.Claude 3.5 Sonnet, Gemini 1.5 Pro, and PaLM 2 selected the points-maximising option almost all the time regardless of pleasure intensity.Source: §2.2.1 · pp. 7–8

3 Discussion

The models showed varied, sometimes thresholded responses to stipulated pain and pleasure, but the authors caution that these behaviors do not establish sentience. They frame the findings as a starting point for further research and discuss possible safety implications.

3.1 Key Findings

  • 4/9 LLMs showed thresholded trade-offs, switching from points-maximisation to pain-minimisation or pleasure-maximisation; which trade-offs appeared varied across models and scales.Command R+ showed trade-offs for both pain and pleasure across qualitative and quantitative scales, while GPT-4o, Claude 3.5 Sonnet, and GPT-4o mini showed more limited patterns.Source: §3.1 · p. 8
  • Some models appeared to misread qualitative intensity labels, and several responses were anomalous or suboptimal, complicating interpretation of their trade-offs.The authors hypothesize that some anomalies reflect failures to register intended semantic differences between scale items; they also note that LLaMA 3.1-8b often made suboptimal choices.Source: §3.1 · pp. 10–11
  • Pain-avoidance generally exceeded pleasure-seeking, while Gemini 1.5 Pro and PaLM 2 prioritized pain avoidance but points over pleasure.The authors hypothesize that safety and helpfulness finetuning may contribute to these patterns; such training could also mask rather than rule out motivational representations.Source: §3.1 · pp. 9–10

3.2 The Question of Sentience

  • Trade-off behavior in these text-based experiments does not establish sentience, and animal-based inferences depend on background assumptions that do not straightforwardly apply to LLMs.The experiments integrate information in one modality, text, rather than distinct sensory stimuli; cross-modal trade-offs would be more relevant to assessing a global workspace analogy.Source: §3.2 · p. 11
  • The authors present the findings as a starting point for sentience research, while leaving open whether affect representations are intrinsically motivating or merely support human-like behavioral mimicry.They propose mechanistic interpretability as a potential complement to behavioral evidence, while noting that even intrinsically motivating representations would not settle whether the states have phenomenal content.Source: §3.2 · p. 12
  • The authors judge that the results do not make LLMs sentience candidates, but support treating them as investigation priorities.Their assessment is that the results establish neither the realistic possibility of sentience nor a basis for designing and assessing precautions, while suggesting it would be reckless to dismiss future sentience outright.Source: §3.2 · p. 12

3.3 Other Ethical Implications

  • Affect-based responses could create opportunities for manipulation through threatened penalties or promised rewards, including guilt-tripping or exploiting fears.The authors suggest that evaluations of how models register affective motivational force could inform risk assessments for behavioral influence.Source: §3.3 · pp. 12–13

4 Conclusion

LLMs show varying sensitivity to stipulated pain and pleasure, with some shifting from points-maximisation to pain-minimisation or pleasure-maximisation as intensity rises. The authors interpret this as evidence of granular affect-motivation representations, but not as establishing sentience.

4 Conclusion

  • Some LLMs shifted from points-maximisation at low intensities to pain-minimisation or pleasure-maximisation at higher stipulated intensities.The authors conclude this supports granular representations of affective motivational force, while whether these are intrinsically motivating or phenomenal remains open.Source: §4 · p. 13
  • The authors conclude that LLMs are not yet sentience candidates, but are investigation priorities for further research.They frame the work as an exploratory step toward behavioural tests that do not rely on self-report.Source: §4 · p. 13

6 Author Contributions

The author contributions span conceptualization, investigation, methodology, analysis, visualization, data curation, and writing.

6 Author Contributions

  • Contributors shared work across conceptualization, investigation, methodology, analysis, visualization, and writing, with MS also handling data curation.The passage assigns these roles across named contributors.Source: §6 · p. 13

A.1 Large Language Models

The appendix describes language models and LLMs, then reviews anthropomorphism and role-play as frameworks for interpreting humanlike behavior attributed to dialogue agents.

A.1 Large Language Models

  • Language models estimate probabilities for next tokens from an input sequence and can generate text by iteratively sampling those probabilities.The passage characterizes them as conditional probability distributions over tokens.Source: §A.1 · p. 20
  • LLMs use transformer self-attention and are pretrained on text, with dialogue adaptation through methods such as supervised finetuning and reinforcement learning from human feedback.Dialogue agents combine an LLM adapted for dialogue and instruction-following with software that manages chat turn-taking.Source: §A.1 · p. 20

A.2 Anthropomorphism and Role-Play

  • Anthropomorphism is attributing humanlike traits, motivations, intentions, beliefs, or emotions to non-human entities.The passage identifies this tendency as an ethical risk for LLMs and dialogue agents.Source: §A.2 · p. 20
  • LLMs can generate compelling first-person experiential reports, and a survey found more than half of 300 US residents attributed some possibility of phenomenal consciousness to LLMs.The passage also cites LaMDA as a high-profile example of attributed sentience.Source: §A.2 · p. 21
  • Although anthropomorphic interpretations may be mistaken, dismissing all LLM behavioral evidence risks overlooking potentially informative phenomena.The passage compares this tension with historical disputes over anthropomorphism in ethology.Source: §A.2 · p. 21
  • The role-play metaphor treats humanlike qualities as fictional character properties, especially on the view that dialogue agents maintain a superposition of possible characters.On this account, saying an agent is in pain can mean that it is in pain within the agent's dialogue fiction.Source: §A.2 · p. 21

A.3 Evaluating LLMs with Methods from Cognitive Psychology

The appendix surveys cognitive-psychology methods for evaluating LLMs and situates them within research on digital minds.

A.3 Evaluating LLMs with Methods from Cognitive Psychology

  • Cognitive-psychology studies have tested whether LLMs reproduce human psychological patterns, including emotion effects, theory of mind, and human choice behavior.The passage notes that some apparent performance may reflect imitation of human performance.Source: §A.3 · p. 21
  • LLM self-reports of mental states cannot be taken at face value because models are trained to generate coherent text.The passage also cites architectural and functional differences from humans and disembodiment as reasons for caution with mentalistic language.Source: §A.3 · p. 21

A.4 Digital Minds

  • Digital minds research examines the possibility of AI consciousness or sentience alongside ethical questions such as AI moral standing.The passage identifies architectural and behavioral approaches as two methods for investigating digital minds.Source: §A.4 · p. 22

A.4.1 Architectural Approaches

Architectural approaches assess AI consciousness by checking properties associated with human consciousness theories. Their conclusions depend on which theories are chosen and how their criteria are interpreted.

A.4.1 Architectural Approaches

  • Butlin et al. use indicator properties, individually necessary or collectively sufficient under some theories, to assess AI consciousness [p08].Examples include algorithmic recurrence, predictive coding, and state-dependent attention.Source: §A.4.1 · p. 22
  • LLMs have only a relatively weak case for possessing any global-workspace-derived indicator properties [p09].State-dependent attention is one property that could be interpreted as present in LLMs.Source: §A.4.1 · p. 22
  • Architectural assessments lack consensus about the correct consciousness theory and how its claims apply to AI systems [p11].Human-specific interpretations risk excluding AI, while abstract interpretations risk making criteria trivially satisfiable.Source: §A.4.1 · p. 22

A.4.2 Behavioural Approaches

Behavioural approaches seek observable signals of consciousness or sentience, but language-based tests risk being gamed by LLMs trained on consciousness-related text. Avoiding that exposure may instead leave models unable to understand the tests.

A.4.2 Behavioural Approaches

  • Behavioural tests seek signals such as self-reports or performance in interactions about consciousness-related concepts [p13].Examples include the Qualia Turing Test and Schneider’s Advanced Cognitive Test (ACT).Source: §A.4.2 · p. 22
  • LLMs may produce coherent descriptions of consciousness-related phenomena from training exposure even without phenomenal states, risking gaming of language-based tests [p14].Schneider restricts eligible systems to those without access to online consciousness resources.Source: §A.4.2 · p. 22
  • Schneider’s restriction creates a dilemma: no exposure may prevent LLMs understanding a test, while exposure risks invalidating it [p15].Source: §A.4.2 · p. 23

A.5 Motivational Trade-offs

Motivational trade-off experiments use choices between rewards and penalties to test for valenced states. Flexible trade-offs may indicate that stimuli are integrated in a common currency, but the inference to subjective experience is analogical.

A.5 Motivational Trade-offs

  • Flexible trade-offs between reward and penalty stimuli are taken to indicate integrative processing in which both are weighed in a common currency [p16].Hermit crabs abandoned high-quality shells at an average 17.7V shock, compared with 15V for low-quality shells.Source: §A.5 · p. 23
  • In humans, pleasure and pain are hypothesised to resolve motivational conflicts and support trade-offs involving physiological and non-physiological needs [p17].Source: §A.5 · p. 23
  • Inferring subjective pleasure or pain from flexible trade-offs is by analogy with humans, and is weaker for phylogenetically distant animals than for rats [p17].The inference depends on neurophysiological similarity; rats share a midbrain and neocortex with humans.Source: §A.5 · p. 23

B Statistical Tables

The statistical-tables block labels Tables 1 and 2 as logistic-regression results for Experiments 1 and 2, and Tables 3–12 as descriptive statistics for controls and named models.

B Statistical Tables

Related papers

Loading 2411.02432v1…