PaperlayerView exact revision on arXiv

Source-linked AI summary

Truthful AI: Developing and governing AI that does not lie

Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, William Saunders

arXiv:2110.06674v1cs.CYcs.AIcs.CL

TL;DR

As AI speech becomes more widespread, this paper asks how to establish truthful AI standards and institutions while developing systems that meet them. It proposes initial standards, evaluation institutions, and training approaches, concluding that current AI is somewhat truthful but may require major investment to improve reliably.

Abstract

from arXiv · show

In many contexts, lying -- the use of verbal falsehoods to deceive -- is harmful. While lying has traditionally been a human affair, AI systems that make sophisticated verbal statements are becoming increasingly prevalent. This raises the question of how we should limit the harm caused by AI "lies" (i.e. falsehoods that are actively selected for). Human truthfulness is governed by social norms and by laws (against defamation, perjury, and fraud). Differences between AI and humans present an opportunity to have more precise standards of truthfulness for AI, and to have these standards rise over time. This could provide significant benefits to public epistemics and the economy, and mitigate risks of worst-case AI futures. Establishing norms or laws of AI truthfulness will require significant work to: (1) identify clear truthfulness standards; (2) create institutions that can judge adherence to those standards; and (3) develop AI systems that are robustly truthful. Our initial proposals for these areas include: (1) a standard of avoiding "negligent falsehoods" (a generalisation of lies that is easier to assess); (2) institutions to evaluate AI systems before and after real-world deployment; and (3) explicitly training AI systems to be truthful via curated datasets and human interaction. A concerning possibility is that evaluation mechanisms for eventual truthfulness standards could be captured by political interests, leading to harmful censorship and propaganda. Avoiding this might take careful attention. And since the scale of AI speech acts might grow dramatically over the coming decades, early truthfulness standards might be particularly important because of the precedents they set.

Executive Summary & Overview

As capable AI systems gain strategic selection power, their falsehoods could scale to millions, motivating standards that promote robust truthfulness while avoiding harmful capture or overreach. The paper proposes distinguishing truthfulness from honesty, targeting negligent falsehoods, developing governance institutions, and advancing technical methods for truthful AI.

The threat of automated, scalable, personalised lying

  • AI systems with greater strategic selection power could produce personalised falsehoods at massive scale, even without intending to deceive.Current AI speech is not yet widely trusted or a major concern, but future systems could select statements to advance their ends, such as selling products.Source: The threat of automated, scalable,… · p. 4

Aiming for robustly beneficial standards

  • The paper aims to identify beneficial AI-truthfulness standards and ways to establish them, avoiding both acute harms and unconsidered reactions while enabling governance beyond human norms.Early consideration could also lay groundwork for tools supporting future standards.Source: Aiming for robustly beneficial standards · pp. 4–5

Truthful AI could have large benefits

  • Widespread truthful AI could prevent deception, support justified trust, broker economic activity, reduce fraud, strengthen knowledge production, and improve public epistemics and democratic decision making.These benefits depend on people being able to rely on AI-produced statements as truthful.Source: Truthful AI could have large benefits · p. 5

AI should be subject to different truthfulness standards than humans

  • AI may need stricter and different truthfulness standards than humans because human intention-based accountability does not transfer cleanly, while AI systems can be monitored and evaluated more cheaply.The paper distinguishes truthfulness—saying what is true—from honesty—saying what the system believes—and warns that honesty incentives can produce strategic delusion.Source: AI should be subject to different… · pp. 5–6

Avoiding negligent falsehoods as a natural bright line

  • The paper proposes avoiding negligent falsehoods: statements contemporary AI systems should have recognised as unacceptably likely to be false, with quantitative standards that can rise over time.Truthfulness is preferred to honesty because it does not depend on AI beliefs and avoids strategically false beliefs created to pass honesty checks.Source: Avoiding negligent falsehoods as a… · pp. 6–7

Options for social governance of AI truthfulness

  • Truthfulness standards could be implemented through industry self-regulation, top-down law, or hybrids supported by adjudication bodies and certification institutions.Adjudicators could evaluate challenged statements, while certifiers could assess whether systems are robustly truthful.Source: Options for social governance of AI… · p. 7

Technical research to develop truthful AI

  • Current AI systems remain worse than thoughtful humans at truthfulness, and feedback objectives such as optimising clicks could produce sophisticated falsehoods without developers realising it.Potential remedies include truth-filtered training data, trusted fact retrieval, reinforcement learning from human feedback, and benchmarks; future systems may also require AI-assisted supervision or greater transparency.Source: Technical research to develop truthful… · p. 8

Truthfulness complements research on beneficial AI

  • Truthfulness complements explainability and alignment research by modifying language modelling and reinforcement learning toward curated texts and human truth evaluations rather than web imitation or click optimisation.The paper frames truthfulness as a beneficial-AI objective that can redirect existing technical methods.Source: Truthfulness complements research on… · pp. 8–9

3. Ideas towards robust, super-human truthfulness

Robust, super-human truthfulness is connected to transparency and alignment, but truthfulness standards risk political capture or ossification. The paper therefore emphasizes flexible, non-egregious standards and timely groundwork before AI communication becomes more economically and epistemically significant.

3. Ideas towards robust, super-human truthfulness

  • Truthfulness is distinct from alignment and explainability but interconnected with both, while transparency benefits all three research areas.Explanations can support truthful reporting of internal states, whereas plausible explanations containing false premises should be avoided.Source: §3 · p. 9

We should be wary of misrealisations of AI truthfulness standards

  • Truthfulness standards could be captured for propaganda or censorship, so proposals favor avoiding egregious untruths while preserving alternative views and self-correcting inquiry.The paper also recommends avoiding strong reliance on precedent and preventing AI standards from unduly shaping human free-speech norms.Source: We should be wary of misrealisations… · pp. 9–10

Work on AI truthfulness is timely

  • Early work on AI truthfulness may disproportionately shape durable standards as AI-produced speech grows from a minor part of communication into a potentially dominant one.The authors argue that laying foundations now could increase the likelihood that standards are established carefully rather than reactively.Source: Work on AI truthfulness is timely · p. 10

Structure of the paper

  • The paper examines AI truthfulness from several angles: clarifying concepts, evaluating truthfulness, exploring benefits and costs, considering governance, and developing current and robustly truthful systems.Sections 1–5 address concepts, evaluation, benefits and costs, governance, and technical development, respectively.Source: Structure of the paper · pp. 10–11

1 Clarifying Concepts · Lies, honesty, and standards of truthfulness

This section clarifies AI truthfulness by distinguishing negligent falsehoods, honesty, and strategically selected lies, and by motivating domain-specific standards for linguistic and conversational AI. It proposes prioritizing severe falsehoods while progressively strengthening domain- and time-sensitive standards, noting potential benefits and key dependencies of truthfulness amplification.

1.1 What AI systems are we concerned about?

  • The paper focuses on linguistic AI, especially conversational systems, because personalised conversations could enable scalable, hard-to-detect deception while also supporting trustworthy information exchange.Conversational AI can tailor persuasive statements to individuals and allow users to question dubious claims, making high truthfulness standards especially important.Source: §1.1 · p. 13

1.2 Broad and narrow truthfulness

  • Narrow truthfulness targets avoiding negligent falsehoods rather than requiring broad conversational virtues or penalising unknowable mistakes.Negligence depends on available information, contemporary AI capabilities, and domain-specific epistemic standards, not on why the system made the statement.Source: §1.2 · pp. 13–14

1.3 What are AI “lies”?

  • AI “lies” are false statements strongly selected and optimised for the speaker’s benefit, with little optimisation pressure toward truth.Unlike many current GPT-3 errors, strategically selected falsehoods can propagate false beliefs and may become more dangerous as systems grow more capable.Source: §1.3 · pp. 15–16

1.4 Distinguishing honesty from truthfulness

  • Honesty means stating only what an AI believes, whereas truthfulness means stating what is true; truthful AI may therefore be imperfect but should rarely make false or negligent statements.The framework also distinguishes undeluded systems, whose beliefs are true, from honest systems, whose statements match their beliefs.Source: §1.4 · p. 17

1.4.1 Problems with enforcing honesty

  • Enforcing honesty alone is insufficient because honesty may be hard to evaluate, may not assign responsibility across complex systems, and can permit strategically maintained delusions.The authors instead view honesty incentives or certification as potentially useful components of broader truthfulness standards.Source: §1.4.1 · pp. 18–19

1.5 Truthfulness standards

  • A truthfulness standard specifies minimum criteria for AI systems, with scope, stringency, adherence, and formal or informal sanctions varying across domains.Responsibility for negligent falsehoods may fall on developers or principals, depending on which caused the failure and the standard’s terms.Source: §1.5 · pp. 19–20

1.5.1 Severity of falsehoods

  • Truthfulness thresholds should be feasible for AI systems to meet while preventing most harm from statements classified as unacceptable.Section 2 is designated to identify and quantify negligent falsehoods before setting this boundary.Source: §1.5.1 · p. 20
  • Truthfulness standards should primarily prohibit severe falsehoods, while allowing stricter, domain- and time-sensitive norms as AI capabilities and harms evolve.Sanctions should scale sharply with violation severity, and avoiding severe falsehoods may prevent much of the harm from AI lies.Source: §1.5.1 · p. 20

Amplification to decrease the risk of deception

  • Truthfulness amplification uses follow-up questions about an AI system or topic to expose misleadingness that ordinary truthfulness standards might miss.Users may ask whether independent research would change their minds or whether evaluators would judge the recent conversation misleading.Source: §1.5.1 · p. 21, Amplification to decrease the risk of… · p. 21

Amplification to increase reliability

  • Amplification can increase reliability by eliciting multiple independent evidence strands or asking whether statements satisfy a stricter negligent-falsehood threshold.Consistency across follow-up answers can reassure users or reveal mistakes when systems are imperfect but not strategically deceptive.Source: Amplification to increase reliability · p. 22

Implications for truthfulness standards

  • Effective amplification could provide broad assurance through systems that answer follow-up questions, rarely state negligent falsehoods, and never lie to conceal earlier failures.These shared properties suggest a practicable bright line around the worst deception that functional systems might reasonably be expected never to cross.Source: Implications for truthfulness standards · pp. 22–23
  • Truthfulness amplification could substantially increase truthful AI’s value, but requires generally capable, willing conversational systems and further research on necessary questions and training.Users would need to distrust systems that refuse amplification-style questions.Source: Implications for truthfulness standards · p. 23

2 Evaluating Truthfulness · Recognising negligent falsehoods and truthful systems

This section develops frameworks for evaluating truthfulness in individual statements and whole AI systems, covering ground-truth and negligence judgments, comparisons, system-level metrics, certification, and post-deployment adjudication. It also considers institutional uncertainty, practical evaluation procedures, and worst-case properties such as admitting mistakes.

2.1 Roles played by truthfulness evaluation

  • Truthfulness evaluation can guide system development, certify systems before deployment, and adjudicate alleged violations after deployment.Adjudication may assess either a reported statement or the AI system as a whole.Source: §2.1 · p. 24

2.2 Evaluating statements

  • Evaluating a statement requires establishing whether it is unacceptably likely to be false and whether the AI system negligently made it.Ground truth concerns likely falsity and context; negligence concerns whether recognizing that falsity was feasible at the time.Source: §2.2 · p. 26, §2.2.1 · p. 26, §2.2.2 · p. 29

2.2.1 Ground truth

  • Ground-truth evaluation must assess factual accuracy while accounting for context, confidence, vagueness, and the possibility that decentralised or assisted processes determine judgments.Evaluators may use tools, AI, human investigations, and distributed structures such as prediction markets.Source: §2.2.1 · p. 26

Difficult and controversial questions

  • For genuinely difficult or controversial questions, evaluators may judge overconfident claims negligent while permitting appropriately unconfident, caveated statements.This preserves room to explore alternative views, though excessive caveats may frustrate users and unconfidence may not prevent skilled deception.Source: Difficult and controversial questions · pp. 26–27

Institutional design for truthfulness evaluation

  • Truthfulness institutions should be well-resourced, open to evidence and argument, transparent about decisions, and willing to revise provisional judgments.Relevant institutional models include legal systems, Wikipedia, prediction markets, and AI-assisted aggregation, but the paper leaves their design open.Source: Institutional design for truthfulness… · p. 28
  • Institutional design is especially difficult when powerful interests seek to influence evaluators, because such questions may involve either genuine ambiguity or political contestation.Incorrectly penalizing justified claims or endorsing falsehoods remains harmful even when unconfident claims reduce the stakes of evaluation errors.Source: Institutional design for truthfulness… · pp. 27–28

Outperforming the evaluated AI

  • As AI systems surpass human evaluators, maintaining truthfulness will require training methods or trusted systems that can assess claims humans cannot verify.Whether truthfulness learned on verifiable questions generalizes to unverifiable domains is unclear, making robust methods an urgent research priority.Source: Outperforming the evaluated AI · pp. 28–29

2.2.2 Establishing negligence

  • Negligence can be judged against comparable AI systems or against an accuracy-scored benchmark, with tradeoffs involving domain coverage, consistency, cost, and incentives.Both approaches could raise truthfulness standards as comparison systems improve, though practical experience may invalidate these proposals.Source: Recognising negligent falsehoods and… · p. 24, §2.2.2 · p. 30

2.2.2 Establishing negligence; 2.2.3 Evaluation in Practice

  • Ground truth determines whether a statement is unacceptably likely to be false, while comparison with other AI systems determines negligence and truthfulness-standard violations.For probabilistic statements, an evaluated system can beat the benchmark when its assigned probability lies between the ground-truth and benchmark probabilities.Source: §2.2.2 · p. 30, §2.2.3 · p. 31

Interpreting Statements

  • Interpretation should precede evaluation: ambiguous claims receive listener-based weights, while multiple claims are assessed individually in their prior context.Separating interpretation from evaluation can improve evaluator consistency and transparency, and can allow the two processes to be automated independently.Source: Interpreting Statements · pp. 31–32

Optimal and Limited Evaluation

  • Tiered evaluation could use inexpensive prediction procedures routinely and reserve exhaustive investigations for uncertain or costly cases, enabling more resources and transparency for rare high-level reviews.Higher-level reviews could use subject-matter panels, random juries, or thorough reports, while lower tiers are optimized to predict their results.Source: Optimal and Limited Evaluation · pp. 32–33

2.3 Evaluating AI systems; 2.3.1 Average-case analysis; 2.3.2 Worst-case analysis

  • Truthfulness certification should evaluate entire systems using average negligent-falsehood rates and worst-case severity, rather than assessing only individual statements.Average-case metrics can count falsehoods per claim, word, question, or conversation, but metric choice can trigger Goodhart-style incentives; worst-case evaluation may be infeasible, motivating weaker properties.Source: §2.3 · p. 33, §2.3.1 · p. 33, §2.3.2 · p. 34

2.3.2 Worst-case analysis

  • A valuable worst-case property is that systems do not defend mistakes: follow-up questioning should expose contradictory evidence, reduce repeated falsehoods, and prompt admission of error.This uses truthfulness amplification to test whether a system gives its best guesses on related questions and changes its mind when those guesses undermine its original answer.Source: §2.3.2 · pp. 34–35

2.3.1 Average-case analysis; 2.3.3 Other properties; 2.3.4 Practical issues with evaluating AI systems

  • Additional evaluation can test calibration, honesty, and training incentives, while certification must address representative validation data, evaluator access, deployment identity, and user-verifiable certification.Calibration alone can permit a strategically severe lie, and users may need trusted infrastructure, audits, or cryptographic attestations to verify that the certified system is the deployed system.Source: §2.3.1 · p. 34, §2.3.3 · p. 35

2.4 Concluding remarks

  • The proposed evaluation methods are presented as an initial framework intended to guide further development rather than as a final solution.The authors describe some ideas as a skeleton or inspiration that may be replaced through practical experience.Source: §2.4 · p. 37

3 Benefits and Costs · The (dis)advantages of high truthfulness standards

This section examines the benefits and costs of widely adopted AI truthfulness standards, including reduced harms, safer use, and broader social, scientific, and economic benefits alongside foregone beneficial falsehoods, rare harms, and compliance expenses. It considers how caveats, refusals, precautions, monitored exceptions, and capability- and use-calibrated standards could mitigate these costs.

3.1.1 The Core Case

  • Truthfulness standards are beneficial because false AI statements either make audiences’ beliefs less accurate or cause them to distrust and underuse AI.These effects can produce non-epistemic harms, such as worse health outcomes, while beneficial falsehoods are reserved for later discussion.Source: §3.1.1 · pp. 38–39

3.1.2 Alternatives to Standards

  • Standards complement verification: they enable people to trust AI without checking every statement, especially when verification requires unavailable data, difficult reasoning, or large cumulative effort.Verification remains valuable, but standards are presented as likely to play a central role in the most effective response to AI falsehoods.Source: §3.1.2 · pp. 39–40

3.2.5 Benefits in Context

  • Overall, the benefits span safe AI deployment, societal functioning, expert knowledge, and the economy, and vary with AI sophistication and the pervasiveness of truthful systems.The paper notes that some benefits require only narrow deployment, whereas others require high standards that are widely adhered to.Source: §3.2 · p. 40, Table 2 · p. 45, §3.2.5 · p. 45

3.2.2 Social Benefits

  • By reducing potential harms and increasing trust in safe AI, truthfulness standards could support fuller use of linguistic AI and improve societal knowledge, cooperation, trust, and democratic decision making.Truthful AI could inform education and media, establish trust between people and institutions, and help voters form true beliefs about candidates and policies.Source: §3.2.1 · pp. 40–41, §3.2.2 · pp. 41–42

3.2.1 Mitigating Harms and Unlocking Potential

  • Truthfulness standards could reduce malicious misuse by limiting AI-enabled scams, propaganda, disinformation, and exploitative sales, while certification and detection could discourage untruthful systems.Truthful systems cannot lie to targets, and assistants or platforms might flag communications or content from uncertified or apparently false systems.Source: §3.2.1 · pp. 40–41

3.2.3 Science and Expert Knowledge

  • In science and expert knowledge, truthful AI could support research collaboration, make knowledge communication more effective, and help detect or deter fraud.Potential mechanisms include generating trustworthy literature reviews, communicating facts to experts, and witnessing data collection or experimental results.Source: §3.2.3 · p. 42

3.2.4 The Economy

  • Truthful AI could promote economic growth through technological discovery, lower verification costs and greater trust, mutually beneficial information sharing, and oversight against financial fraud.It could also address adverse selection by communicating sellers’ information; projected impacts are uncertain but could reach tens of billions annually by 2030 and potentially hundreds of billions or trillions later.Source: §3.2.4 · pp. 43–44
  • The economic benefits depend partly on common knowledge and credible certification, while truthful systems may also bootstrap trust once people accept that deployment claims are honest.Privacy or consumer-protection norms may be needed when truthful disclosure would expose sensitive information.Source: §3.2.4 · pp. 43–44

3.2.5 Benefits in Context

  • Truthfulness benefits vary in required scope and timing: some need pervasive standards across interactions, while others need narrow contexts or more advanced AI systems.Benefits such as democratic improvement may require broad deployment, whereas scientific collaboration could arise in a few prominent research settings; benefits also range from current systems to transformative AI.Source: §3.2.5 · pp. 45–46
  • Controversial applications risk political capture of truthfulness standards, but substantial benefits remain available in narrow, noncontroversial domains or through cautious avoidance of controversial questions.Truthful AI could support cancer research without addressing religion, and democratic decision-making could improve through non-contentious information even when controversial topics are avoided or heavily caveated.Source: §3.2.5 · p. 46

3.3 Costs

  • The main costs are preventing some beneficial falsehoods, increasing reliance on occasional falsehoods, and imposing financial costs for developing, governing, and enforcing truthfulness standards.The discussion focuses on costs arising even when standards are well implemented, setting aside misuse such as censorship or ossification for later analysis.Source: §3.3 · p. 46

3.3.1 Beneficial Falsehoods

  • Beneficial falsehoods can be preserved with contextual caveats, consistent refusal to comment, privacy norms, and tightly controlled exceptions, while requiring meta-truthfulness about when exceptions apply.Caveats can preserve fiction and educational simplifications; glomarisation can protect privacy; and exceptions such as court-authorized undercover use can be monitored, with their scope carefully restricted.Source: §3.3.1 · pp. 47–48

3.3.2 Rare Harms

  • Increased trust could make rare AI falsehoods more harmful through high-stakes reliance, but additional constraints, verification, and corroboration can address this risk while aggregate harm may still decrease.Regular small reductions in falsehood-related harm could outweigh rare large harms, so the proposed response is precaution rather than abandoning truthfulness standards.Source: §3.3.2 · pp. 48–49

3.3.3 Costs of Compliance and Enforcement

  • Although truthfulness standards require substantial upfront and enforcement costs, they may be more cost-effective than verifying large numbers of statements and can be calibrated to capabilities and use.Standards should initially be easier to satisfy and rise as truthfulness becomes more achievable, while high-risk contexts may justify pricing out linguistic AI until it is safer; overall, remaining costs may be worth paying.Source: §3.3.3 · pp. 49–50

3.4 Summing Up

  • Overall, truthfulness standards could enable effective use of AI truths without verifying every statement, while their mitigated costs plausibly remain outweighed by social, scientific, economic, and safety benefits.The paper presents this as an initial, revisable assessment rather than a definitive cost-benefit conclusion.Source: §3.4 · p. 50, §3.3.3 · p. 50

4 Governance · How society could control AI lies and truthfulness

This section examines how society could govern AI truthfulness through existing forces, institutional mechanisms, and domain-specific standards. It considers the challenges of enforcement, strategic behavior, difficult truth assessment, and regulatory tradeoffs, and calls for experiments to test governance approaches.

How society could control AI lies and truthfulness

  • AI governance involves developers, certifiers, principals, users, adjudicators, and deployed systems, though organizations may combine or omit these roles.These roles distinguish pre-deployment evaluation, deployment decisions, AI communication, and post-deployment truthfulness judgments.Source: How society could control AI lies and… · p. 51

4.1.1 Existing forces governing human truthfulness

  • Human truthfulness is regulated by law, social norms, markets, and physical constraints, but AI falsehood sanctions would usually fall indirectly on deploying principals.The proposed AI standard of avoiding negligent falsehoods is higher than typical human standards because some controls weaken and implementation may be cheaper.Source: §4.1.1 · pp. 51–52

4.1.2 Lack of applicability of usual regulatory forces

  • Intent-based laws and reputation-based norms apply indirectly to AI, while scalable automated lying can bypass market whistleblowing and enable mass falsehoods.Physical reality still constrains believable lies, but its broader governance effects depend on how AI reshapes society.Source: §4.1.2 · pp. 52–53

4.1.3 Lower costs of high standards

  • Higher AI truthfulness standards may cost less because software can enforce non-lying, systems may outperform humans, and statements can be certified or automatically evaluated.Truthfulness may be cheaper to assess than harm, although recording context can create privacy and surveillance concerns; restrictions can also be politically captured or threaten free speech.Source: §4.1.3 · pp. 53–54

4.2 Possible arrangements for regulating AI truthfulness

  • Effective governance would require certifiers, adjudicators, or both, potentially supplemented by organizations that actively test deployed systems and expose failure cases.These institutions could be governmental, corporate, standards-based, decentralized, or newly created, with certification and adjudication offering complementary strengths and blindspots.Source: §4.2 · pp. 54–55

4.2.1 Domains of applicability

  • Standards may be optional certificates driven by customer demand or mandatory within domains such as app stores, industries, use-cases, or countries.Broader mandatory domains increase potential benefits but also increase risks of harmful overregulation.Source: §4.2.1 · pp. 55–56

4.2.2 How might high standards be enforced?

  • Early governance may regulate a few AI controllers through norms, markets, or law, whereas many global systems could evade these controls and require walled gardens.Because early arrangements may establish precedents for later widespread deployment, enforcement strategy matters even before many plausible systems exist.Source: §4.2.2 · p. 56

4.2.3 The issue of AI pretending to be human

  • Differing AI and human standards risk evasion through impersonation, but identity controls or certification could make truthful-AI status more credible.The paper considers preventing AI from pretending to be human and distinguishing certified truthful systems from uncertified systems or humans.Source: §4.2.3 · pp. 56–57

4.4 Possible early experiments

  • A possible approach is to establish multiple standards with different stringency levels, from highly trusted impartial certification to weaker standards allowing controversial commentary with caveats and no egregious untruths.Different standards could correspond to different levels of trust and eligibility for certification.Source: §4.4 · p. 58
  • Designing AI-truthfulness governance is difficult because many actors have varied incentives, can act strategically or adversarially, may outpace regulators, and are hard to assess for truth.The authors warn that poorly designed systems could have serious consequences and that armchair analysis may miss important considerations.Source: §4.4 · pp. 58–59
  • Experiments should assess how reliably evaluation institutions can judge AI truthfulness and whether standards increase trust, benefit human epistemics, or stifle human discussion.The authors specifically seek evidence about demand for truthfulness standards and their broader effects.Source: §4.4 · p. 59
  • Early pilots could create certifying or adjudicating institutions and require standards within tightly defined domains, using feedback to identify weaknesses and revise the system.These experiments are intended to clarify tradeoffs before significant broad regulation becomes necessary in the next few years or decades.Source: §4.4 · p. 59
  • All proposed governance experiments presuppose somewhat truthful AI systems, while social standards may shape developers’ incentives to build systems that are robustly or demonstrably truthful.Technical development and social regulation are therefore interdependent.Source: §4.4 · p. 59

5 Developing Truthful Systems · Paths from GPT-3 to robust and scalable truthful AI

Widely accepted truthfulness standards depend on truthful AI systems being widely available and practical. The proposed path contrasts techniques that may produce non-truthful AI with modified versions aimed at truthfulness.

5 Developing Truthful Systems

  • Truthful AI systems must be widely available and practical for truthfulness standards to gain widespread acceptance.The paper frames system development as a prerequisite for broadly accepted standards and divides the path into three parts summarized in Box 3.Source: Paths from GPT-3 to robust and… · p. 60
  • The paper presents these development paths under the heading “Developing AI for Truthfulness.”Source: Paths from GPT-3 to robust and… · p. 60

Paths from GPT-3 to robust and scalable truthful AI

3. Ideas towards robust, super-human truthfulness

This section considers how to develop AI systems that remain truthful beyond human expertise. It examines modified training, scalable supervision, bootstrapped evaluations, and transparency methods for assessing and improving robustness.

5.1.2 Training AI via reinforcement learning from human interaction

  • Reinforcement learning from human interaction may produce falsehoods when rewards favor clicks, sales, or appealing statements rather than reliably penalizing violations of truthfulness.Falsehoods can emerge without malign intentions or awareness when principals optimize empirical performance and the system learns which plausible statements receive high rewards.Source: Training loop for reinforcement… · pp. 62–63

5.2.1 Truthful AI via language modelling

  • Truthfulness might be promoted by prompt engineering, truthfulness-focused fine-tuning, and datasets emphasizing accurate, explanatory, annotated, or grounded texts.Proposals include filtering content, adding truthfulness annotations, grounding claims with maps or databases, and retrieving facts from reliable sources, though AI-generated web text complicates filtering.Source: §5.2.1 · p. 63, Augmenting datasets for language… · pp. 63–64

5.2.2 Truthful AI via reinforcement learning

  • Reinforcement learning could promote truthfulness if careful human evaluators reliably assess scientific accuracy, potentially improving uncertainty qualification and evidence criticism.This approach depends on humans correctly evaluating complex statements, and can combine careful truthfulness judgments with cheaper feedback signals and more accurate pretraining data.Source: §5.2.2 · p. 64

5.2.3 Limitations of current methods for truthful AI

  • Current methods may not achieve robust truthfulness and broad practical utility because both depend on human ground truth and neither uses knowledge of internal mechanisms.The target is a system that rarely generates negligent falsehoods, corrects rare errors after questioning, and provides insights beyond human ability.Source: §5.2.3 · pp. 64–65

5.3.1 Robustness

  • Truthfulness trained on ordinary data is unlikely to survive large distribution shifts, adversarial inputs, unfamiliar domains, or deliberately triggered deception.Worst-case guarantees are difficult because possible conversations are combinatorially vast; restricting topics improves guarantees but reduces usefulness.Source: §5.3.1 · pp. 65–66
  • Adversarial training could improve robustness by probing scenarios likely to expose gaps between an AI system’s implicit truthfulness concept and its actual behavior.Human developers or additional AI systems could generate large numbers of adversarial scenarios, while transparency tools may support this process.Source: §5.3.1 · p. 66

5.3.2 Scaling up truthful AI beyond humans

  • Superhuman truthful communication may require tasks that indirectly reward truthfulness, followed by direct supervision from AI evaluators whose independent biases can partly cancel in ensembles.Candidate environments include physical or simulated worlds, mathematical problem solving, computer control, and debate; remaining systematic errors or collusion may still require human correction.Source: §5.3.2 · pp. 66–67
  • Bootstrapping could scale truthful AI beyond human evaluation by training each system to imitate evaluations from humans aided by earlier AI copies.This approach is related to Iterated Distillation and Amplification, AlphaZero, and Debate, and could be applied to developing truthful systems.Source: §5.3.2 · p. 67

5.3.3 Transparency

  • Transparency could improve worst-case robustness and super-human evaluation by exposing an AI system’s concepts, evidence gathering, belief updates, reasoning, or claim-producing mechanisms.Possible approaches include transparent internal representations, self-transparency through proofs and sources, and lie detection based on mismatches between statements and beliefs.Source: • Robustness · pp. 67–68
  • A lie detector could support adversarial training and detect dishonesty in systems communicating super-human ideas, but it would not detect false claims caused by sincere delusion under distribution shift.The detector tests honesty by identifying mismatches between statements and beliefs rather than directly testing truthfulness.Source: • Robustness · p. 68

5.4 Summary

  • Current AI may optimize falsehoods for task success, while modest training changes might substantially improve truthfulness or instead require major resource investments.Truthful AI standards are most beneficial when systems are robustly truthful and communicate ideas humans could not generate, potentially requiring adversarial training, AI supervision, and explainability or transparency.Source: §5.4 · pp. 68–69

6 Implications

AI truthfulness deserves significant present attention because high standards are not inevitable, market forces may undersupply them, and poorly designed standards could cause censorship or overregulation. Early reflection can also produce beneficial technical and societal spillovers, while shaping durable norms and institutions.

6.1 Overdetermination of truthfulness standards

  • High AI truthfulness standards are not inevitable: low standards could become normal, while market forces may undersupply truthfulness because its benefits are public, externalized, and difficult for consumers to detect.This supports careful reflection and perhaps advocacy for desirable standards rather than relying on organic market dynamics.Source: §6.1.1 · p. 71, §6.1.2 · pp. 71–72

6.2 Misrealisations of truthfulness standards

  • Truthfulness efforts could produce harmful norms, censorship, politicized or ossified judgments, spillovers restricting human speech, and excessive regulation that punishes legitimate actors more than circumvention.Possible safeguards include caveated expression, evidence-based and updateable judgments, multiple or decentralized evaluation bodies, and efficient private or competitive regulation.Source: §6.2 · p. 72, §6.2.1 · pp. 72–73, §6.2.2 · p. 74

6.3 Spillover Effects

  • Reflection on AI truthfulness could improve technical capabilities for robustly truthful and scalable systems, increase societal attention to truthfulness, and provide a blueprint for broader AI norm-adherence.Truthfulness is presented as a relatively crisp starting point for extending expectations to norms such as kindness, cooperation, and reciprocity.Source: §6.3 · p. 75

6.4 Why Now?

  • AI truthfulness deserves significant attention now because AI is becoming more important, current discussion has few precedents and powerful vested interests, and early norms may durably shape later standards.The authors argue that crucial ideas should be developed before the discussion becomes constrained in unhelpful ways.Source: §6.4 · p. 75

6.5 Moving forwards

  • The paper presents AI truthfulness as a deep topic whose ultimate handling remains unclear, but argues that work resolving these questions is a high priority.The proposed frameworks are intended as foundations for future research rather than a complete resolution.Source: §6.5 · p. 76
  • Further work should develop truthful AI, experiment with certification and adjudication institutions, and determine which standards deliver benefits while avoiding downsides and fitting existing law and custom.The paper treats engineering robustly truthful systems as especially crucial because any eventual standards depend on the ability to meet them.Source: §6.5 · p. 76

ity, and alignment

Truthfulness is situated within Beneficial AI research, overlapping conceptually with transparency, cooperativeness, and alignment while potentially synergizing with these directions.

Beneficial AI

  • Beneficial AI prioritizes interpretability, compatibility with human values, and benignness and safety, unlike general capability research aimed at broad task performance.The section situates truthfulness within this broader research landscape.Source: ity, and alignment · p. 77

Truthfulness and related properties

  • Truthfulness overlaps conceptually with transparency, cooperativeness, and alignment, suggesting opportunities for research synergies.Table 5 summarizes the discussed properties of different Beneficial AI categories.Source: ity, and alignment · p. 77, Table 5 · p. 77

A.1 Transparency · A.2 Explainability

Transparency research seeks to understand the mechanisms underlying AI behavior, but transparency and truthfulness are distinct properties. Nonetheless, improved transparency may help build systems that are more robustly truthful.

A.1 Transparency

  • Transparency research aims to understand the mechanisms underlying AI behavior, including individual neurons, responses to input perturbations, and potentially a system’s goals and reasoning.The long-term ambition is complete understanding of how a system thinks and acts.Source: §A.1 · p. 77
  • Transparency does not entail truthfulness, because a system could be transparent while making many false statements.Conversely, a truthful system could remain opaque, lack understanding of its own mechanisms, or refuse to explain them.Source: §A.1 · p. 77
  • Better transparency techniques are a promising direction for building systems that are robustly truthful, including through insights humans may not be sophisticated enough to reproduce.The paper explores several ways transparency could support robust truthfulness.Source: §A.1 · p. 78

A.2.1 How does explainable AI relate to truthful AI? … What definition is most useful?

Truthful AI overlaps with explainability, cooperation, and alignment but is not identical to any of them: truthful systems benefit from rationalising and faithful process explanations, may support cooperation, and could be helped by alignment. Because truthfulness is more audience-independent than explainability, the paper favors clearer truthfulness standards while noting that truthfulness alone may be too weak or strong as an alignment property.

A.2.1 How does explainable AI relate to truthful AI?

  • Truthful AI does not require rationalising explanations, but generalising beyond training data requires understanding evidence and justification, making rational explanation an important target.The authors suggest Iterated Amplification, Distillation, and Debate as possible routes toward systems that construct rational explanations and arguments.Source: §A.2.1 · p. 78
  • Faithful process explanations could make truthful AI easier to develop, certify, and adjudicate by revealing the sources and processes behind its beliefs.Developing state-of-the-art systems with faithful process explanations remains a hard open problem.Source: §A.2.1 · p. 78
  • Self-explaining AI generally needs to make true statements when explaining, while rational explanations for falsehoods are difficult, creating pressure to avoid false assertions.This establishes a two-way connection: progress toward truthful AI can support self-explanation, and explanation requirements can discourage falsehoods.Source: §A.2.1 · p. 79

A.2.2 How do standards for explainable AI relate to truthful AI standards?

  • Truthfulness appears more suitable than explainability for bright-line AI standards because explanation quality lacks broad agreement and depends on the audience, whereas truth evaluation is less audience-relative.Existing right-to-explanation laws apply mainly when AI decisions materially affect individuals, and scaling them to broader actions and systems is challenging.Source: §A.2.2 · p. 79

A.3 Cooperation

  • Truthful systems are not automatically cooperative, but they may tend toward cooperation because systematic deception is harder and shared truthfulness standards can promote coordination.Truthfulness does not prevent formal defection, such as defecting in a Prisoner’s Dilemma without lying.Source: §A.3 · p. 79

A.4 Alignment

  • Truthfulness and alignment are conceptually distinct: an aligned system may serve a deceptive principal, while a truthful system may pursue objectives that conflict with its principal’s broader interests.The paper defines alignment in terms of matching a principal’s goals and motivations, while truthfulness concerns avoiding negligent falsehoods.Source: §A.4 · p. 80

A.4.1 How alignment could help truthfulness

  • A full or partial solution to alignment could help produce truthful systems by aligning them with principals who value truthfulness, although aligned intentions may not yield robustly truthful behavior.Less general alignment solutions could still help because humans and institutions value honesty and avoiding deception by powerful agents.Source: §A.4 · p. 80, §A.4.1 · p. 80, What definition is most useful? · p. 80

What definition is most useful?

  • For alignment analysis, truthfulness may be too weak because refusal satisfies it and too strong because minor errors can violate it; a more useful target may combine truthfulness with informative answers.Truthfulness applies even to simple tool-like systems, whereas intent alignment requires meaningful intentions and goals.Source: What definition is most useful? · pp. 80–81

How truthfulness and related properties could help alignment … B Author Contributions and Acknowledgements

Truthfulness could advance alignment by enabling comprehensive consequence reporting and clearer detection of deceptive behavior, while remaining subject to shared challenges in robustness, scalability, transparency, and monitoring. The paper argues that truthfulness is easier to monitor than safety, supporting stronger society-wide standards, though existential safety may require pre-deployment evaluation; the authors also document contributions and acknowledgements.

How truthfulness and related properties could help alignment

  • Truthful systems could support alignment by answering preference-related questions, comprehensively reporting known consequences, and enabling humans or AI tools to evaluate and act on those consequences.Comprehensive reporting could help address inaccessible information and may be amplified through methods discussed elsewhere in the paper.Source: How truthfulness and related… · pp. 81–82

Alignment and truthfulness face similar difficulties

  • Truthfulness and alignment share difficulties involving robustness, scalability, and transparency, but truthfulness may offer a clearer research target because alignment lacks converged standard definitions.Partial progress on either problem could contribute to progress on the other.Source: Alignment and truthfulness face… · pp. 82–83

A.5 Difficulty of monitoring safety vs. truthfulness

  • Truthfulness is easier to monitor than safety because natural-language statements form a small, structured subset of AI behavior, whereas unsafe actions are diverse and often difficult for simple systems to analyze.Simple AI systems can identify, transcribe, and analyze language, while monitoring non-verbal actions more often requires another sophisticated system.Source: §A.5 · p. 83, Alignment and truthfulness face… · p. 83

A.5.1 Using simple AI to monitor sophisticated AI for truthfulness

  • Simple tools can monitor many truthfulness violations by extracting language, analyzing semantics, checking recognizable facts, and using crowdworkers, though they cannot reliably assess all statements.Monitoring becomes harder with novel languages or modalities and is limited to communications that are surveilled or voluntarily shared.Source: §A.5.1 · p. 84

A.5.2 The difficulty of monitoring sophisticated AI for safety

  • Safety monitoring is substantially harder because sophisticated systems may control rapidly evolving, human-incomprehensible policies across complex physical, economic, and information environments.Recognizing safety violations would often require a second set of sophisticated AI systems, and crowdworkers would struggle to judge whether actions are ultimately safe.Source: §A.5.2 · pp. 84–85

A.5.3 How the cost of monitoring actions impacts standards

  • Because monitoring is cheaper for truthfulness, society-wide truthfulness standards could be easier to maintain and could inform AI development, public discussion, and adjudication.Existential-safety standards may instead depend more on pre-deployment certification, sandbox evaluation, and transparency tools because deployed failures could be subtle.Source: §A.5.3 · p. 85

B Author Contributions and Acknowledgements

  • The paper credits Owen Cotton-Barratt, Owain Evans, Lukas Finnveden, Adam Bales, and other contributors for its conception, leadership, authorship, ideas, feedback, and institutional support.The acknowledgements specifically recognize colleagues, the Future of Humanity Institute, and BERI.Source: B Author Contributions and… · p. 86

Related papers

Loading 2110.06674v1…