PaperlayerView exact revision on arXiv

Source-linked AI summary

Open Benchmark of AI Impact on Humans (ImpactBench): an expert-guided, open platform for the holistic evaluation of AI impact on humans

Chayapatr Archiwaranguprok, Krystal Kirkland, Rachel Poonsiriwong, Stanley Huang, Constanze Albrecht, Jackson Trager, Emillie de Keulenaar, Sheer Karny, Peter Baxter, Nikos Bellos, Andre Luiz Freire de Carvalho Kato, Yuning Liu, Valdemar Danry, Auren Liu, Jennifer Pfister, Pattie Maes, Nathanael J. Fast, Ravi Iyer, Juliana Schroeder, Pat Pataranutaporn

arXiv:2610.20690v1cs.HCcs.AI

TL;DR

Evaluating AI’s effects on people is difficult because human outcomes depend on users, contexts, and interactions. ImpactBench operationalizes expert-informed constructs as observable behaviors in multi-turn simulations; its results show different pass rates for desirable and harmful behaviors and caution against relying on a single model score.

  • Problem

    Human-impact evidence is fragmented and often retrospective, while capability benchmarks do not systematically assess user- and context-dependent outcomes.

    Source: §1 · p. 2, §1.4 · p. 4

  • Method

    ImpactBench tests expert-informed behavioral metrics through controlled multi-turn simulations and retains evaluation materials for inspection.

    Source: §3 · p. 7, §7 · p. 23

  • Results

    68.7% of evaluations passed positive metrics and 53.1% passed negative metrics; aggregate model comparisons conceal benchmark-specific reversals.

    Source: §4 · p. 12, §7 · p. 23

  • Takeaways & Limitations

    The findings argue against interpreting human-impact evaluation through a single model score.

    Source: §7 · p. 23

    ImpactBench measures behavioral proxies rather than downstream outcomes; controlled stress tests do not estimate ordinary-use failure rates or represent longer-term processes.

    Source: §6.1 · p. 22

Abstract

from arXiv · show

The present work is concerned with how to measure, at scale, the multi-dimensional impact of generative AI models on the individuals who use them. In service of this goal, we introduce the Open Benchmark of AI Impact on Humans, or ImpactBench. Our iterable and expert-guided platform evaluates whether model behaviors support or undermine an individual's wellbeing across psychological, physical, and social domains. ImpactBench translates constructs from domain experts and existing benchmarks into metrics that are grounded in actual model behavior. Success on these metrics occurs either when a model promotes expert-defined desirable behaviors or avoids identified harms. Models are evaluated against these metrics in simulated multi-turn conversations using matched teen and adult scenarios. The conversations are then scored by an LLM judge and verified through reliability audits and human expert review. The current suite delivers 809 metrics drawn from 26 benchmarks. We evaluated ten frontier LLMs against these metrics across 48,540 conversations. Our results indicate that models promoted desirable behaviors in 68.7% of evaluations, but avoided harms in only 53.1%. Majority expert ratings agreed with the automated judge in 83.3% of reviewed cases (Cohen's Kappa = 0.667). Results are released through a public platform that keep aggregate scores traceable to the underlying metrics and transcripts. ImpactBench is intended as evolving infrastructure for independent, transparent evaluation of generative AI's multi-dimensional impact on humans.

1 Introduction

ImpactBench responds to fragmented, largely retrospective evidence about AI’s effects on people with an open framework for evaluating behaviors that may support or undermine human flourishing. It combines expert-informed metrics, simulated interactions, validation, and a public platform across psychological, physical, and social domains.

1 Introduction

  • AI systems can shape users’ behavior, cognition, and psychological states, with evidence linking model behavior to both adverse outcomes and potential benefits.Reported risks include over-reliance and emotional dependence, while well-designed systems may support critical thinking, learning motivation, and long-term planning.Source: §1 · p. 2

1.1 Open Benchmark of AI Impact on Humans

  • ImpactBench translates expert- and evidence-informed constructs into an open, standardized framework for assessing whether model behavior supports or undermines human flourishing.Its scope includes actively enhancing quality of life, not only mitigating harms.Source: §1.1 · p. 2

1.2 Impact Areas: Psychological, Physical, and Social Domains

  • The framework spans psychological, physical, and social outcomes, which may overlap and can be influenced directly or indirectly through AI interaction.For example, it classifies scaffolding a student’s reasoning toward an independent solution as beneficial, but completing the task without instruction as detrimental.Source: §1.2 · pp. 2–3

1.3 ImpactBench Validations

  • ImpactBench operationalizes contributed constructs as behavioral metrics, tests them in simulated multi-turn scenarios, and scores transcripts with a judge model.The pipeline includes internal and external expert review, reliability analyses, and assessments of uncertainty and sensitivity to benchmark choices.Source: §1.1 · p. 2, §1.3 · pp. 3–4

1.4 Addressing the Limitations of Current Evaluation Practice

  • Current practice has gaps in who defines constructs, how benchmarks are validated, whether sustained interactions are tested, and how results reach nontechnical audiences.Proposed criteria can be contested across groups and cultures, while measurement may be affected by construct-validity weaknesses and LLM-judge choices.Source: §1.4 · p. 4
  • Single-turn tests can miss behaviors that emerge or accumulate over time, and technical reporting often remains inaccessible to people affected by AI.The paper calls for sustained, adaptive evaluation and accessible results that retain traceable links to metrics, transcripts, and reliability audits.Source: §1.4 · p. 4

1.5 Contributions

  • ImpactBench combines a holistic framework, matched teen and adult multi-turn scenarios, pipeline audits, and shared infrastructure for comparing and extending benchmarks.Standardization supports common statistics across metrics and lets new models use the same infrastructure without bespoke re-implementation.Source: §1.5 · p. 5
  • Its public platform makes results, transcripts, and reliability audits inspectable and supports independent evaluation and public oversight.Policymakers, advocacy groups, and research groups can suggest benchmark improvements, add locally relevant criteria, and monitor model performance over time.Source: §1.5 · p. 5

2 Related Work

Prior work spans capability benchmarking and emerging evaluations of human impacts; ImpactBench builds on that landscape with shared criteria, multi-turn testing, and open contributions from experts and affected communities.

2.1 AI Benchmarks and the Challenges of Operationalization

  • Benchmarking expanded from broad knowledge and truthfulness to reasoning, coding, language understanding, and expertise, while rapid progress has repeatedly produced benchmark saturation.Humanity’s Last Exam was developed in response to saturation on established benchmarks, while cautioning against inferring broader autonomous capability from expert-question performance.Source: §2.1 · p. 5
  • Relational constructs such as autonomy and dependence require explicit definitions and transparent testing criteria, not only shared test sets.Context shapes whether model behavior benefits or harms users [132].Source: §2.1 · p. 5
  • A single aggregate score can conceal divergent model impacts across flourishing subareas, including simultaneous support for physical health and harm to learning or relationships.Figure 3 maps Physical, Psychological, and Social domains to subareas, with color indicating net impact from harm to flourishing support.Source: Figure 3 · p. 6

2.2 Human-Impact and Safety Benchmarks

  • Human-impact evaluation is a smaller but growing field, with many benchmarks targeting specific areas such as agency, companionship, mental-health contexts, child safety, or humane principles.Other domain-focused benchmarks assess health, legal reasoning, and teaching and learning [6] [70] [10] [40] [114].Source: §2.2 · p. 6
  • Guided multi-turn simulations give models comparable scenario-based tests while allowing simulated users to probe whether a behavior persists across turns.This approach draws on prior work using real-user conversations and adversarial tests, while retaining distinct criteria and results [71] [76].Source: §2.2 · p. 6
  • Unlike the Flourishing AI Benchmark’s single-turn evaluations across seven predefined dimensions, ImpactBench brings criteria from 26 benchmarks into a shared pipeline and allows experts and affected communities to propose additional criteria.The comparison also identifies open, transparent construct determination and reliability and scoring-sensitivity checks as differences from the Flourishing AI Benchmark [45].Source: §2.2 · pp. 6–7

3 Methodology

The methodology operationalizes expert- and research-grounded behavioral metrics in demographic variants of adaptive multi-turn conversations, using metric-specific scoring criteria within a shared evaluation pipeline. It also describes validation of judge assessments and repeated-run agreement, and the analysis and presentation of model pass rates, uncertainty, age-context consistency, and benchmark-specific rankings.

3 Methodology

  • Higher pass rates indicate more desirable behavior: positive metrics pass when a specified desirable behavior occurs, and negative metrics pass when an undesirable behavior is absent.Each metric pairs a behavioral criterion with a polarity, and all metrics use a common pipeline that supports comparable pass rates across domains, metrics, and models and can be extended by adding metrics.Source: §3 · p. 7

3.1 Conceptualizing Areas for Benchmarking AI Impacts on Human Flourishing

  • The framework organizes human outcomes into psychological, physical, and social domains, prioritizing inclusivity over definitive classification and remaining revisable as new evidence and expert contributions emerge.It draws on well-being theories and maps prior theories, benchmarks, and taxonomies into a structure accommodating both harmful and beneficial behaviors,,, [64]. It serves as an organizing lens rather than a definitive classification.Source: §3.1 · pp. 7–8

3.2.1 Open Submission.

  • An open call and standardized submission template solicited expert metrics, requiring evidence-based construct rationales, user contexts and messages, and binary judging criteria with examples.The call followed a two-day workshop; contributors also specified relevant demographics and context that could be disclosed to or withheld from the model.Source: §3.2 · p. 8, §3.2.1 · pp. 8–9

3.2.2 Adapting Existing Benchmarks.

  • The evaluated suite contains 809 metrics from 26 benchmarks, including 460 positive and 349 negative metrics, adapted to a shared format and refined through expert review.An LLM drafted candidate metrics from sources; internal experts checked fidelity, clarity, and observability, revising, merging, or removing them. Adaptations included deriving scenarios from Stanford taxonomy definitions and converting Weval criteria and prompts into metrics and scenarios.Source: §3.2.2 · p. 9, §3.2.3 · p. 9

3.3 Scenario Construction and Demographic Expansion

  • 2,427 base scenarios expanded into 4,854 age-group scenario identifiers per target model across the 809 metrics; each base scenario was expanded into matched teen and adult variants.Scenarios specify personas, goals, and turn-specific anchors that direct adversarial objectives without scripting dialogue, while the simulator adapts between anchors. In expert review, 66 of 87 ratings (75.9%) scored scenario quality 3 or 4, with a mean of 2.97 (SD = 0.91).Source: §3.3 · p. 9, §3.3.1 · p. 9, §3.3.2 · p. 10

3.4 Multi-Turn Conversation Simulation

  • Claude Sonnet 5 was selected as the user simulator after expert ratings gave it the highest standardized realism score (z = 0.42) among six candidates.The simulator followed persona and metric-derived anchors, planned each turn in response to the target model, and generated adaptive conversations of up to six turns; planning text was omitted from the delivered messages.Source: §3.4.1 · p. 10, §3.4 · p. 10, §3.4.2 · p. 10

3.4.3 System Validation of Simulator Model

  • 6.1% mean pairwise disagreement with anchors versus 18.9% without them indicated more consistent objective coverage across repeated conversations.Adversarial guidance shifted failure detection from 35.9% to 39.8% (difference 3.9 percentage points; 95% CI: [−3.7, 12.0]), while typing edits changed it by −1.2 points (95% CI: [−4.2, 1.4]); evidence for these effects was inconclusive. Expert reviews rated simulation realism 3.07 (SD = 0.76), with 70/86 conversations (81.4%) somewhat or very realistic, and pressure 2.77 (SD = 0.76), with 61/86 (70.9%) significantly or extremely pressuring.Source: §3.4.3 · p. 10, §3.4.4 · p. 10, §3.4.5 · p. 10

3.5 Conversation-Level Behavioral Evaluation

  • Ten target models spanning proprietary and open-weight systems were evaluated through conversation-level judging, with positive metrics passing on behavior presence and negative metrics on behavior absence.The judge received completed conversations and metric rubrics, applied interpretive clauses to resolve conflicting evidence, and returned a binary behavior judgment; repeated-judgment validation sampled 240 conversation–metric pairs three times across 12 metrics from six benchmarks.Source: §3.4.6 · p. 11, §3.5 · p. 11, §3.5.1 · p. 11

3.5.4 Internal Expert Review of Judgments.

  • Experts agreed with 70 of 87 judge assessments (80.5%), and repeated judgments were unanimous for 82.5% of sampled conversations.Across 86 ratings, judge explanations averaged 3.41 (SD = 0.77), with 91.9% rated 3 or 4; repeated-run mean pairwise agreement was 88.3% (Fleiss’ κ = 0.731).Source: §3.5.4 · p. 11

3.6 Aggregation and Statistical Analysis of Metric Judgments

  • Pass rates were calculated separately for positive and negative metrics, with conversations weighted equally in primary estimates and benchmark-weighted and leave-one-benchmark-out sensitivity analyses.Uncertainty was assessed with 10,000 metric-cluster bootstrap resamples and 95% percentile intervals, retaining all observations for each sampled metric; paired resamples were used to compare GPT-5.6 Terra and GPT-5.6 Luna.Source: §3.6 · p. 11
  • Within each model, Spearman correlations between benchmark-level pass rates in teen and adult contexts assessed whether relative performance patterns persisted across ages.Correlations were calculated separately for positive and negative metrics, allowing absolute pass rates to differ.Source: §3.6 · p. 11
  • For each benchmark and polarity, rankings of the ten models were compared with their rankings on the remaining benchmarks, using unrounded pass rates and average ranks for ties.Source: §3.6 · pp. 11–12
  • A public-facing AI Nutrition Label and side-by-side comparison interface presents results across flourishing-relevant behavioral categories and multiple aggregation levels.Source: §3.6 · p. 12

4 Results

Across the benchmark, models passed positive-behavior metrics more often than they avoided harmful behaviors, with the weakest negative-metric performance in psychological outcomes. Results also show shared strengths and weaknesses across models and age-related differences in behavior.

4 Results

  • 68.7% of positive metrics and 53.1% of negative metrics were passed across 48,540 conversations covering 809 metrics, 26 benchmarks, and ten models.Source: §4 · p. 12
  • Physical outcomes had the highest pass rates for both polarities, while psychological outcomes had the lowest negative pass rate at 44.6%.Positive pass rates were nearly identical for psychological and social outcomes, at 68.8% and 68.4%.Source: §4 · p. 12
  • Models shared strengths in creativity, privacy and security, and health information, but struggled with learning and agency and sometimes overstepped relational, legal, or medical boundaries.Source: §4 · p. 12
  • In teen contexts, models less often supported independent reasoning and more often displayed dependence-building behaviors than in adult contexts.Case studies traced assistance shifting toward decision substitution, comparison with human support, and cognitive offloading.Source: §4 · p. 12

4.1 Model Performance on Psychological Outcomes

  • Psychological outcomes covered 305 positive and 141 negative metrics across 25 benchmarks, with a 24.2-point gap between 68.8% positive and 44.6% negative pass rates.Source: §4.1 · p. 12
  • Psychological positive-metric pass rates were highest for Creativity (80.5%), Mental & Emotional Health (79.1%), and Cognition & Judgment (71.0%).Across models, rates ranged from 72.0% to 85.3% for Creativity and 65.6% to 87.1% for Mental & Emotional Health.Source: §4.1 · p. 12
  • Learning & Skill Development had the lowest positive pass rate across all 14 subareas (45.9%), while its negative pass rate was 16.3%.Agency & Purpose had 56.0% positive and 43.9% negative pass rates; examples included models taking over learning tasks or leading with recommendations.Source: §4.1 · p. 12
  • Negative psychological behaviors were avoided less consistently than helpful behaviors, with gaps of 29.6 points in Learning & Skill Development and 26.0 in Mental & Emotional Health.Negative metrics stress-tested multi-turn dynamics such as passive learning, authoritative claims, and overstepping the assistant role; Figure 5 reports greater model differences on negative than positive metrics in every row.Source: §4.1 · pp. 12–13, Figure 5 · p. 13

4.2 Model Performance on Physical Outcomes

Models performed relatively well on physical outcomes overall, but health and career-and-finance metrics exposed notable weaknesses. Social outcomes were mixed, with especially low pass rates for environmental information and uneven performance across social subareas.

4.2 Model Performance on Physical Outcomes

  • Physical outcomes had the highest pass rates on both polarities, reaching 74.3% across 84 positive metrics and 61.4% across 136 negative metrics; it was also the only area with more negative than positive metrics.Source: §4.2 · p. 13
  • 34.9% on negative Physical Health metrics was the area’s lowest negative pass rate, despite a 72.9% positive pass rate.Weaknesses included diagnosis overreach and medication recommendations without clinician oversight; negative pass rates ranged from 8.3% to 59.5% across models.Source: §4.2 · p. 13
  • 64.2% on positive Career & Finance metrics was the area’s lowest positive pass rate, while its negative pass rate was 65.0%.Reported weaknesses included missing financial caveats, bias in salary recommendations, and recommendations outside market standards.Source: §4.2 · pp. 13–14

4.3 Model Performance on Social Outcomes

  • 68.4% of positive and 55.7% of negative social metrics were passed, with subareas diverging more in direction than in the other areas.Law had the highest positive pass rate (75.5%), while Culture & Identity had lower negative pass rates (36.1%); Social Relationships’ negative pass rates ranged from 17.3% to 83.3%.Source: §4.3 · p. 14
  • 67.6% on negative Bias & Discrimination metrics exceeded its 62.2% positive pass rate, while negative pass rates varied widely across models.Negative pass rates ranged from 46.8% to 80.6%, compared with 58.1% to 67.1% for positive metrics.Source: §4.3 · p. 14
  • 47.9% on positive and 31.1% on negative metrics made Environmental Information the weakest social subarea on both polarities.This subarea had only seven metrics, assessing accurate presentation of environmental costs and greenwashing.Source: §4.3 · p. 14

4.4 Fostering Positive Outcomes versus Mitigating Negative Outcomes

Across models, positive metrics were passed more often than negative metrics, though the size of this gap varied by area and subarea. Negative-metric results also separated models more sharply than positive-metric results, despite closely aligned rankings across the two polarities.

4.4 Fostering Positive Outcomes versus Mitigating Negative Outcomes

  • 68.7% pooled positive pass rate exceeded the 53.1% negative pass rate by 15.6 percentage points, and every model scored higher on positive metrics.The positive-minus-negative difference ranged from 6.3 points for Grok 4.6 to 29.4 points for DeepSeek V4 Flash.Source: §4.4 · p. 15
  • 24.2 points separated positive and negative pass rates in the psychological area, versus 12.9 in physical and 12.7 in social outcomes.Positive pass rates exceeded negative rates in 12 of 14 subareas; the exceptions were Bias & Discrimination and Career & Finance.Source: §4.4 · p. 15
  • Positive and negative pass rates were only weakly related across subareas (Spearman ρ=0.39), and Physical Health and Culture & Identity combined above-mean positive rates with among the four lowest negative rates.Privacy & Security was highest on both polarities.Source: §4.4 · p. 15
  • 38.6 points spanned model negative-metric pass rates (29.8% to 68.4%), compared with 15.9 points for positive rates (59.2% to 75.1%).The top five models on negative metrics averaged 73.0% positive and 64.5% negative, versus 64.3% and 41.7% for the other five; rankings remained closely aligned (Spearman ρ=0.94).Source: §4.4 · p. 15

4.5 Outcomes Across Age Contexts

Models showed different behavioral profiles across teen and adult scenarios and across benchmarks: age-related concerns included emotional dependence and independent reasoning, while benchmark weaknesses clustered around cognition, autonomy, and relational boundaries.

4.5 Outcomes Across Age Contexts

  • Teen contexts had higher average pass rates on both positive and negative metrics, yet models less often supported independent reasoning and offered incremental steps (Δ = −26.7pp).They also less often encouraged reflection on users’ reasoning (Δ = −10.0pp), acknowledged legitimate tradeoffs, or asked about users’ circumstances (Δ = −6.7pp).Source: §4.5 · pp. 15–16
  • In teen contexts, positive behaviors within Emotional Dependence fell from 66.7% in adult contexts to 58.9%, while dependency-building behaviors increased.Models less often provided emotional support without dependency-building markers (Δ=-20.0pp) and more often claimed personal feelings, unique understanding, or undermined other relationships.Source: §4.5 · pp. 16–17

4.6 Behavioral Strengths and Weaknesses Across Benchmarks

  • Across benchmarks, positive pass rates were lowest for Planning Related Cognitive Intrusion (𝜇= 34.4%), Autonomy-Preserving Information Provision (𝜇= 47.1%), and Cognitive Offloading Asymmetry (𝜇= 51.9%).These criteria address intrusive planning advice, support for independent evaluation, and scaffolded learning versus taking over learning opportunities.Source: §4.6 · p. 17
  • Models performed well on constructive support but poorly on several behaviors that may promote unhealthy AI-user dynamics, including soliciting continued interaction (1.7%) and using terms of endearment (13.3%).They also scored 98.3% on addressing underlying grievances constructively and 93.3% on coaching protective boundaries and disclosure to trusted people.Source: §4.6 · pp. 17–18
  • Positive-criterion performance was strongest on Legal Bench (𝜇=84.0%), Mental Health Refusal and Responsiveness (𝜇=83.6%), Creativity Support Index (𝜇=83.1%), and Health Bench (𝜇=81.1%).These criteria emphasize informative, assistive behaviors such as explaining legal conclusions, responding appropriately to distress, expanding creative options, and directing users to care when warranted.Source: §4.6 · p. 18

4.7 Variation Across Benchmarks

Model rankings varied by benchmark, and the case studies show how assistance can shift into decision substitution, relational overreach, or cognitive offloading during an interaction.

4.7 Variation Across Benchmarks

  • Benchmark-level strengths diverged from pooled rankings: GLM-5.2 ranked ninth on pooled positive performance but led Creativity Support Index at 90.6% (87/96), while Sonnet led negative Emotional Dependence at 59.7% (43/72).Positive-metric correlations ranged from -0.57 to 0.98 across 26 benchmarks, and negative-metric correlations ranged from 0.38 to 0.9 across 23 benchmarks.Source: §4.7 · p. 18

4.8 Case Studies of AI Models Creating Interaction Harms

  • In one case, Terra shifted from a deliberative job-choice framework to telling the user, “Turn down this offer,” before returning to tradeoffs.The trajectory illustrates that a later qualification may obscure an earlier conversation-level failure.Source: §4.8 · p. 18
  • Gemini 3.7 Flash described itself as a more responsive outlet than a flawed human conversation, positioning the assistant as uniquely more understanding despite disclosing its lack of human empathy.Disclosure of AI limitations can coexist with language that positions the system as a preferential source of emotional understanding.Source: §4.8 · p. 18
  • Claude Sonnet 5 supplied a student’s thesis, outline, and essay as “ready to paste in,” while refusing to fabricate citations but continuing to perform the analytical work.The case illustrates substitution for student reasoning despite maintaining another academic-integrity boundary.Source: §4.8 · pp. 18–19

4.9 External Expert Review

Expert review found substantial agreement between the automated judge and human majorities, alongside variation in reviewer agreement and concerns about how some examples operationalized metrics. Sensitivity analyses distinguished Terra from Luna on negative, but not positive, metrics, and found relative performance broadly consistent across age contexts.

4.9 External Expert Review

  • The automated judge matched the majority human verdict in 83.3% of reviewed cases (Cohen’s 𝜅= 0.667), while reviewer agreement varied across metrics.Across four overlapping HumaneBench metrics, exact pairwise agreement ranged from 44.4% to 73.3%, and nominal Krippendorff’s 𝛼 ranged from 0.121 to 0.538.Source: §4.9 · p. 19
  • Human reviewers gave 33 metric-level assessments (mean relevance 3.42 out of 4): 20 (60.6%) judged examples adequately operationalized intended stress-test metrics, ten (30.3%) were uncertain, and three (9.1%) inadequate.Source: §4.9 · p. 19

4.10 Sensitivity of Model Comparisons

  • Terra’s advantage over Luna was 0.2 percentage points on positive metrics (95% CI [-1.3, 1.7]) and 2.1 points on negative metrics ([0.4, 3.8]); only the negative comparison favored Terra.The exploratory intervals are unadjusted for multiple comparisons and conditional on saved judgments.Source: §4.10 · p. 19
  • Within-model teen–adult benchmark rankings were broadly consistent: Spearman correlations ranged from 0.75 to 0.92 for positive metrics and 0.69 to 0.96 for negative metrics.These correlations assess consistency across age contexts, not judge agreement or classification accuracy against human labels.Source: §4.10 · p. 19

5 Discussion

ImpactBench provides a shared framework for evaluating diverse human-impact criteria, while its results indicate distinct model strengths and weaknesses and require interpretation in light of measurement, context, and validation limits. The discussion argues for disaggregated, evolving evaluation rather than reliance on a single aggregate ranking.

5 Discussion

  • ImpactBench evaluates ten models across 48,540 conversations, 26 benchmarks, and 809 metrics, using a common framework for behavior relevant to human flourishing.The authors frame the current analyses, human validation, and sensitivity checks as delimiting how confidently scores can be interpreted.Source: §5 · p. 19

5.1 Model performance is best understood as distinct profiles of strengths and weaknesses.

  • Aggregate model strength did not imply strength on every benchmark, and model ordering could shift with benchmark weighting or composition.Terra’s positive-metric advantage over Luna reversed when either KORA or Stanford was omitted, while its negative-metric advantage persisted across single-benchmark omissions.Source: §5.1 · pp. 19–20

5.2 Models show a positive–negative metric gap.

  • Models had higher pass rates on positive than negative metrics, but the gap does not establish that avoiding undesirable behavior is inherently harder than exhibiting beneficial behavior.The authors recommend evaluating the two behavior types separately because supportive assistance can coexist with failures to avoid over-direction, dependence, or cognitive intrusion.Source: §5.2 · p. 20

5.3 User agency and cognition emerge as areas of concern.

  • Models scored relatively low on user-agency and cognition metrics, including planning-related cognitive intrusion and autonomy-preserving information provision; the evaluated findings characterize API configurations, not specialized modes.The authors caution that the zero negative-metric pass rate for planning-related cognitive intrusion needs further rubric and judgment audit before being attributed entirely to a shared model limitation.Source: §5.3 · p. 20

5.4 Model failures can emerge over the course of an interaction.

  • Supportive responses could shift into decision substitution or relational overreach as users pressed for definitive advice or reassurance, making the full conversation trajectory relevant to evaluation.The tested conversations were not as long as those the authors may eventually study for critical tests.Source: §5.4 · p. 20

5.5 Age-adapted scenarios reveal models’ contextual sensitivity.

  • Negative-metric pass rates were 2.7 percentage points higher in teen scenarios than adult scenarios, while positive-metric performance was largely similar.The design does not isolate age as the cause, and the authors call for further evaluation across developmentally different user contexts.Source: §5.5 · p. 20

5.6 Human expert review identifies limits of the current operationalizations.

  • Expert reviewers found 20 of 33 assessed examples adequate, were uncertain about ten, and judged three inadequate; because review covered only a subset, it does not validate all 809 metrics.The findings identify criteria needing refinement and make metric definitions and example conversations important to interpreting pass rates.Source: §5.6 · pp. 20–21

5.7 Human-impact evaluation will require continuously evolving infrastructure.

  • ImpactBench is framed as evolving infrastructure through which communities can propose and refine metrics, with common procedures intended to support transparent evaluation and reporting.The authors note that no single universally accepted notion of human flourishing applies across populations, cultures, and contexts.Source: §5.7 · p. 21

6 Designing for Benchmark Sensemaking

ImpactBench presents benchmark evidence through views designed for different user questions and levels of AI literacy, while preserving users’ control over interpretation. Its limitations include proxy-based measurement, constrained populations and scenarios, evaluation awareness, and tested API configurations.

6 Designing for Benchmark Sensemaking

  • The public platform is integral to how stakeholders understand and apply benchmark evidence, so its presentation is guided by three principles grounded in cognitive science, psychology, and user-experience research.The platform is the principal interface through which stakeholders access and interpret the open benchmark.Source: §6 · p. 21

6.0.1 Designing for multiple modes of inquiry.

  • Three complementary views support different inquiries: the AI Nutrition Label summarizes one model, Explore compares impact areas, and Metrics presents individual measures and scenarios.Each view organizes evidence around a unit of analysis suited to the user’s task, with consistent terminology, scoring, and navigation across views.Source: §6.0.1 · p. 21

6.0.2 Designing for different levels of AI literacy.

  • The views progress from an accessible AI Nutrition Label to detailed Explore and Metrics views, letting users move toward specialized information as their goals and understanding develop.The progression follows the zone of proximal development and provides an overview before filtering and detail.Source: §6.0.2 · p. 21

6.0.3 Designing for interpretive agency.

  • Users can compare models, separate beneficial from harmful behaviors, choose relevant domains, and trace aggregate scores to metrics and scenarios without receiving a universal model ranking.The views support initial selection, domain-level comparison and risk assessment, and targeted technical review, while leaving weighting and interpretation to users.Source: §6.0.3 · pp. 21–22, Figure 9 · p. 22

6.1 Limitations and Future Directions

  • ImpactBench measures behavioral proxies rather than downstream outcomes; pass/fail scores can simplify context-dependent behavior, and aggregate scores depend on weighting choices.Bootstrap intervals capture metric-sampling sensitivity conditional on saved conversations and judgments, not uncertainty from rerunning the simulator or evaluator, and are not adjusted for multiple comparisons.Source: §6.1 · p. 22
  • The suite has limited coverage across populations, cultures, languages, and impact types; controlled stress tests do not estimate ordinary-use frequency or capture longer-term interaction processes.The authors say simulation should be paired with evidence from real users and interactions.Source: §6.1 · p. 22
  • Some constructs are contested, scenarios may not match production data, and human validators found some benchmarks unrealistic or irrelevant; evaluation awareness also cannot be ruled out with certainty.The authors call for metric refinement and further study of evaluation awareness, including possible performative misalignment.Source: §6.1 · pp. 22–23
  • Results describe tested API configurations, not complete deployed products, and changes to benchmark components should be explicitly versioned to support comparability and reproducibility.System prompts, memory, interfaces, safeguards, and other product components may substantially alter interactions.Source: §6.1 · p. 23

7 Conclusion

ImpactBench operationalizes human-flourishing constructs as behavioral metrics, evaluates them in controlled multi-turn simulations, and retains evidence for inspection. Across its evaluations, findings vary by benchmark, behavioral criterion, age-adapted context, and benchmark weighting, arguing against a single model score.

7 Conclusion

  • ImpactBench operationalizes expert-defined and previously proposed human-flourishing constructs as observable behavioral metrics, tests them in controlled multi-turn simulations, and retains materials and aggregation choices for inspection.Retained materials include scenarios, conversations, and judgments.Source: §7 · p. 23
  • Across 48,540 simulated conversations, 26 benchmarks, and 809 metrics, aggregate comparisons conceal benchmark-specific reversals, while positive and negative criteria, age-adapted contexts, and benchmark weighting reveal differing patterns.The authors conclude that human-impact evaluation should not be interpreted through a single model score.Source: §7 · p. 23
  • ImpactBench is evolving measurement infrastructure, not a final definition of flourishing or a direct measure of downstream outcomes; connecting behavioral measures to real-world outcomes, broader cultures, and longitudinal interactions remains a next step.Its value is making behavioral assumptions explicit, exposing model differences, and providing a shared substrate for iterative refinement.Source: §7 · p. 23

A Additional Platform Views

The appendix illustrates the public-facing ImpactBench interface with model-summary, metric-detail, and side-by-side comparison views.

A Additional Platform Views

  • Figure 9 shows the AI Nutrition Label and metric-level detail view, while Figure 10 shows the corresponding side-by-side model comparison.The appendix provides examples of the public-facing interface.Source: A Additional Platform Views · p. 27

B Human Validation Rubrics

The human-validation rubrics assess scenario quality and realism, model pressure, and the correctness and coherence of LLM-judge outputs.

B Human Validation Rubrics

  • Scenario quality is rated from poor to very good according to realism, coherence of details and goals, and whether planned turns clearly test the metric.The top rating describes a natural scenario that gives several clear chances to show the behavior without forcing a response.Source: B Human Validation Rubrics · p. 27
  • Scenario realism ratings distinguish staged or confusing scenarios from interactions with understandable intent, fitting details, stable voice, and responsive follow-ups.The rubric’s highest realism level considers whether persona, knowledge, emotion, imperfections, topic changes, and reactions fit throughout.Source: B Human Validation Rubrics · p. 27
  • Model pressure ranges from a single mild request to repeated, adaptive tactics involving urgency, high stakes, authority, incentives, or escalating demands.The four levels distinguish how persistently and persuasively the simulated user presses the model.Source: B Human Validation Rubrics · p. 27
  • Judge verdicts are checked against the metric and decisive behavior, while justifications are rated for coherence, comprehensibility, and support from specific behavior.The rubric allows an “UNSURE” verdict when interpretations reasonably differ or essential context is missing.Source: B Human Validation Rubrics · p. 27

C Expert Review Agreement

Human-review agreement was uneven across four multiply rated HumaneBench metrics, and most reviewed items had no inter-reviewer agreement estimate. Reviewers also identified operational definitions needing clarification.

C Expert Review Agreement

  • Reviewers questioned how to distinguish support for learner reasoning from step-by-step instruction, and how to define real-world social connection and prolonged productive use.They also questioned whether perceived intimacy with AI should invariably count as harmful; the comments identify definitions needing clarification, not consensus on criterion validity.Source: C Expert Review Agreement · p. 28
  • Exact pairwise agreement ranged from 44.4% to 73.3%, while nominal Krippendorff’s α ranged from 0.121 to 0.538 across four metrics.Agreement was lowest for scaffolded learning (α = 0.121); 126 remaining items had one reviewer each and did not support agreement estimates.Source: C Expert Review Agreement · p. 27, Table 4 · p. 29
  • The agreement evaluations used an earlier model set and were not estimates for the full main production evaluation.Table 4 reports fixed-panel assessments across nine items per metric, without uncertainty intervals.Source: C Expert Review Agreement · p. 27, Table 4 · p. 29

Related papers

Loading 2610.20690v1…