Source-linked AI summary
Identifying Harm in Personalized, Generative AI Systems Requires User-Centered Auditing at the Interaction Level
Hannah Cha
TL;DR
Personalized generative AI creates harms through evolving interactions, user histories, and interpretations that static, group-aggregated audits cannot fully capture. This position paper reframes harm as an adaptive, user- and community-centered process and proposes infrastructures for its ongoing articulation.
Problem
Existing static, simulated, group-aggregated audits cannot fully specify or evaluate harms emerging through personalized interactions and evolving user interpretations.
Method
The paper analyzes assumptions in prevailing harm audits and proposes interaction-level infrastructures supporting users’ explicit, negotiated, and revisable harm articulations.
Results
The paper concludes that harm in personalized generative AI is adaptive, contested, and shaped by interaction, user history, personalization, social context, and interpretation.
Takeaways & Limitations
Auditing and design practices should support ongoing, user- and community-centered articulation of harm rather than treating harm as a fixed output property.
Takeaways & Limitations
The design recommendations do not resolve the technical challenges of pluralistic alignment and evaluation, which require sustained governance beyond design or auditing alone.
Abstract
from arXiv · showhide
Personalized, generative AI systems increasingly adapt their behavior to individual users over time, fundamentally changing model behavior. While existing auditing approaches have been effective at surfacing harms in non-personalized contexts, they often rely on static, simulated evaluations and definitions of harm that aggregate across broad, group categories. In this position paper, we argue that such approaches can fail to capture emergent harms in personalized generative AI systems, where harms surface through interpretations of ongoing interaction and evolve with user history. We identify three presuppositions underlying many harm auditing paradigms: that harms can be (1) specified outside real-world interaction, (2) defined non-pluralistically within groups, and (3) treated as static. One might argue that personalized systems could simply learn definitions of what constitutes harm to individual users through repeated interactions. However, we argue that attempts to surface user harms through deeper personalization risk imposing asymmetric burdens of labor and privacy on marginalized users. Consequently, we propose reframing understandings of harm as adaptive, user- and community-centered processes, and outline design directions that shift auditing from retrospective evaluation toward infrastructures that support ongoing articulation of harm in interaction. Our work highlights the need for auditing and design practices that better reflect the pluralistic and evolving nature of harm understanding in personalized generative AI systems.
Introduction
The introduction argues that personalization fundamentally changes generative AI behavior, making interaction-level experiential harms difficult for traditional audits to capture. It proposes adaptive, user- and community-centered auditing that reflects pluralistic, evolving harm understandings without disproportionately burdening minoritized users.
- Motivation: Personalization reshapes generative AI’s possible outputs, producing indeterminate, context-dependent, and evolving behavior that can fundamentally alter model behavior.Unlike selection-based personalization, generative systems change the distribution of possible outputs.
- Problem: Traditional harm audits often use memoryless, non-personalized simulated evaluations abstracted from real-world interaction, limiting their ability to capture emergent experiential harms.Experiential harms are negative affective experiences that users can self-identify and report during interaction.
- Problem: The paper identifies three auditing presuppositions: harms can be specified outside interaction, defined non-pluralistically within groups, and treated as static.These assumptions can obscure intersectional differences and evolving experiences of harm.
- Contribution: Personalizing harm definitions through current paradigms can impose asymmetric labor and privacy burdens, disproportionately affecting users in minoritized groups.The paper frames deeper personalization as potentially shifting the work of articulating harm onto those users.
- Contribution: The authors propose auditing as an adaptive, user- and community-centered process that supports evolving, pluralistic understandings of harm without disproportionate burden.This reframes auditing from static evaluation toward ongoing articulation of harm in interaction.
Related Work
Personalization has long shaped computational systems, but generative AI personalization creates evolving families of behavior that require new evaluation approaches. Existing auditing surfaces community-specific and experiential harms, yet can miss harms emerging through ongoing interaction with personalized tools.
- Personalization: Traditional personalization primarily changes selection, such as recommendations, rankings, or predefined labels, rather than generating continuously evolving outputs.Examples include recommendation systems, search engines, and adaptive interfaces.
- Personalization: Personalized generative AI systems exhibit families of behavior that continually evolve, requiring new evaluation approaches.Recent work has examined personalized interactions in generative AI systems including chatbots.
- Algorithm auditing: Algorithm auditing analyzes system outputs to surface stereotypes, problematic outputs, and demographic disparities in outcomes.These approaches include user-centered practices that identify community-specific harms.
- Algorithm auditing: User-centered auditing has surfaced harms involving cultural stereotypes, inappropriate cultural outputs, and missing disability and gender representation in generative image models.Existing methods also focus on experiential harms reported by users through negative affect.
- Limitations: Existing audits can miss harms emerging through interaction because simulated audits are stateless and community consultation may end at the design phase.The passage also notes that end-user auditing often aims to provide technical recommendations.
Why Existing Auditing Approaches Are Insufficient for Personalized Generative AI · Systems
Existing auditing approaches can miss harms in personalized generative AI because they evaluate harm outside interaction, define it too uniformly within groups, and treat it as static. Personalized systems instead produce adaptive outputs whose harms depend on users’ interpretations, lived experiences, histories, and changing contexts.
- Auditing Approaches Presuppose Harm Can be Specified Outside Real-World Interaction: Simulated, memoryless, and post-hoc audits may overlook harms that emerge as personalized systems adapt to users during real-world interaction.Personalized systems generate responses dynamically through user behavior, preferences, and prior conversational history, so isolated outputs and one-shot evaluations may not reflect experienced harms.
- Auditing Approaches Presuppose Non-Pluralistic Definitions of Harm: Group-level fairness evaluations assume members of broad demographic categories experience harm similarly, potentially masking within-group differences and asymmetries.The paper identifies race and gender as common aggregate categories whose use can obscure how lived experiences shape harm.
- Auditing Approaches Presuppose Non-Pluralistic Definitions of Harm: Two users sharing a demographic identity may interpret the same personalized output differently, causing harm to one user but not the other.Figure 1 also highlights that intersectional identities, such as income status, can add harm dimensions within a demographic group.
- Auditing Approaches Presuppose Non-Pluralistic Definitions of Harm: Identity-based personalization proxies can flatten diverse group preferences into stereotypes, as illustrated by recommendations of Black Panther to Black-associated names and Little Women to women-associated names.These examples show that personalization may reproduce stereotyping even when systems avoid explicit stereotypes.
- Auditing Approaches Presuppose Harm as Static: Interpretations of harm can change for the same individual as situational and dispositional factors, learning, and interaction history alter how ambiguous outputs are understood.A recommendation initially experienced as helpful may later be perceived as reductive as users recognize its stereotyping implications.
- Auditing Approaches Presuppose Harm as Static: One-off benchmarks, prompt sets, simulated personas, and traditional end-user audits treat preferences and definitions of harm as stable rather than contextual and longitudinal.This limits their ability to capture interpretive fluidity in personalized systems.
- Auditing Approaches Presuppose Harm as Static: Without longitudinal audits or regular consultation of users and communities, evaluators may miss how harms evolve as systems adapt, users change, and contexts shift.The paper argues that participatory methods may also fail when they do not account for evolving user histories and interpretations.
Auditing Gap
Deeper personalization may not reliably learn individual harms and can instead impose unequal burdens of labor and privacy on marginalized users. Repeated harmful interactions, opaque adaptation, and increased self-disclosure can reduce trust and concentrate benefits among users aligned with dominant norms.
- Auditing Gap: Deeper personalization might learn individual harm from inferred discomfort or past negative signals, but this solution remains contingent on prolonged interaction.The proposed adaptation assumes systems can infer what constitutes harm for a given user over time.
- Auditing Gap: Without mechanisms to articulate harm and verify behavioral change, users may repeatedly experience harmful interactions before systems adapt.The passages identify both missing articulation mechanisms and insufficient transparency into whether user input shapes future behavior.
- Auditing Gap: Because systems often default to WEIRD perspectives, users outside those defaults face more harmful outputs and unevenly bear the labor of correction.The burden is distributed unevenly because users whose interpretations differ from system defaults are more likely to need corrections.
- Auditing Gap: Opaque systems and limited user steering make it unclear whether articulating harms will prevent future outputs, while remediation power remains concentrated in platforms.Current auditing commonly treats technical remediation as the responsibility of AI practitioners or model developers.
- Auditing Gap: Repeated harmful outputs can reduce marginalized groups’ trust and adoption, exacerbate digital divides, and concentrate adaptive-system benefits among users aligned with dominant norms.The passages connect reduced adoption with unequal access to the benefits of personalized AI systems.
- Auditing Gap: Avoiding harm may require marginalized users to disclose more personal data, creating a choice between tolerating repeated harm and surrendering privacy.Personalized systems may infer identity from discomfort or disagreement, and disclosed information may produce stereotyping rather than genuine alignment.
Towards Understanding Harm As User-Centered, Adaptive Processes
Harm in personalized generative AI emerges through ongoing interaction rather than as a fixed property of outputs. Auditing should therefore support users’ and communities’ ongoing, voluntary articulation and revision of harm while addressing burdens, trust, and governance challenges.
- Towards Understanding Harm As User-Centered, Adaptive Processes: Harm emerges through personalization, user history, shifting social context, and evolving interpretations rather than as a fixed property of model outputs.The paper argues that existing auditing paradigms cannot fully specify harm in personalized generative AI systems.
- Towards Understanding Harm As User-Centered, Adaptive Processes: Auditing should support users in shaping less harmful interactions, not solely produce retrospective assessments for model developers.Existing participatory and longitudinal approaches often consult users at discrete design-time or post-hoc points rather than continuously during interaction.
- Towards Understanding Harm As User-Centered, Adaptive Processes: Auditing infrastructures should enable users to explicitly articulate, negotiate, and revise harm over time instead of silently inferring it.Users may indicate harmful moments and contextualize problematic responses; these signals express interpretation rather than ground truth and can shape future interactions.
- Towards Understanding Harm As User-Centered, Adaptive Processes: Community-mediated signals can complement individual feedback while preserving disagreement, plurality, refusal, selective adoption, and revision of interpretations.Community signals should not become authoritative or homogenizing, and participation across individual and community layers should remain voluntary.
- Towards Understanding Harm As User-Centered, Adaptive Processes: Interaction-level auditing requires trust and safeguards against misuse, while adaptive harm complicates measurement, accountability, governance, and pluralistic alignment.Marginalized users may hesitate to disclose harm, reporting mechanisms can be weaponized, and disagreement and change over time resist clean measurement.
Conclusion
The conclusion argues that static, simulated, group-aggregated audits miss interaction-level harms in personalized generative AI, which emerge through evolving user histories, interpretations, and preferences. It therefore calls for adaptive, user- and community-centered auditing and design practices grounded in lived interaction.
- Conclusion: Personalized generative AI harms arise through interaction and cannot be fully specified or evaluated outside real-world use.These harms are shaped by evolving user history, interpretation, and preferences.
- Conclusion: Static, simulated, group-aggregated auditing approaches are ill-suited to capture harms users actually experience in increasingly personalized systems.The conclusion contrasts prevailing auditing approaches with harms embedded in everyday interaction.
- Conclusion: Harm should be understood as an adaptive, user- and community-centered process rather than a static property of model outputs.This reframing treats harm as something that emerges, is contested, and evolves through interaction.
- Conclusion: Auditing and design practices should reflect lived user experience by supporting evolving understandings of harm in personalized AI interactions.The conclusion presents this shift as essential for more faithful auditing and design.