PaperlayerView exact revision on arXiv

Source-linked AI summary

Algorithmic Harms Associated with Generative Model-Augmented Recommendation Systems

Christine Herlihy, Xumei Xi, Shloka Desai, Kevin Bannerman Hutchful, Pedro Silva

arXiv:2609.33073v1cs.LGcs.AIcs.CLcs.IR

TL;DR

Risks from generative-model augmentation of non-conversational recommendation systems remain under-explored. The paper expands the harm taxonomy and analyzes possible causal drivers; it concludes that familiar harms may manifest in novel ways and that semantic misalignment can produce unintended harms.

  • Problem

    Harms associated with generative-model augmentation of machine-learning pipelines in non-conversational recommendation settings remain under-explored.

    Source: §1 · p. 1

  • Method

    The paper characterizes potential harms and hypothesizes their causal drivers to inform detection and mitigation strategies.

    Source: §1 · p. 1

  • Results

    Representational and quality-of-service harms may persist and manifest in novel ways, while semantic misalignment can give rise to unintended harms.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Potential harms may arise without malicious intent and may persist despite—or arise from—prosocial mitigation efforts.

    Source: §1 · p. 1

    Gaps between semantic intent and realized outcomes may be harder to detect when evaluation shifts to smaller-scale LLM-in-the-loop analyses, which may reflect evaluator LLM biases.

    Source: §3.1.2 · p. 5

Abstract

from arXiv · show

In this work, we consider algorithmic harms that may arise as generative models are incorporated into machine learning platforms. We argue that existing harm taxonomies and threat models require extension to (1) address novel causal drivers of well-studied representational and quality-of-service harms; and (2) anticipate and mitigate endogenous harms, such as sanitization, which may arise when system inputs are misaligned with the system designer's objectives, or the generative model's inductive priors. To this end, we introduce an expanded taxonomy of algorithmic harms associated with the use of generative models in non-conversational recommendation systems. In addition, we offer a causal analysis of how problematic subsets of the (input, output) joint distribution can arise, in an effort to inform harms detection and mitigation efforts.

1 Introduction

Generative-model augmentation of machine-learning recommendation pipelines in non-conversational settings remains under-explored. This paper characterizes potential harms and hypothesizes causal drivers, including harms that may arise without malicious intent or despite prosocial mitigation.

1 Introduction

  • Risks from generative-model augmentation of machine-learning pipelines in non-conversational recommendation settings remain under-explored.Prior work has studied recommendation-system harms and generative-model risks, but this compositional setting has received less attention.Source: §1 · p. 1
  • The paper identifies and characterizes potential harms, then hypothesizes their causal drivers to inform detection and mitigation strategies.The stated scope is non-conversational recommendation systems.Source: §1 · p. 1
  • Harms may arise without deliberate malicious intent and may persist despite—or arise from—prosocial mitigation efforts.Source: §1 · p. 1
  • Familiar representational and quality-of-service harms may manifest in novel ways in the compositional setting.The authors also flag unintended consequences or artifacts of composition as especially easy to overlook.Source: §1 · p. 1

2 Related work

The related work reviews harm sources and impacts across conventional machine learning, generative models, and recommendation systems. It identifies a gap in understanding implicit harms when non-conversational language models are combined with traditional machine-learning recommenders.

2 Related work

  • Prior work on algorithmic harms spans modality-specific machine-learning tasks, generative-model (mis)use, and generative models in recommendation systems.Source: §2 · p. 2

2.1 Harms related to modality-specific ML tasks

  • Suresh and Guttag map seven harm sources across the machine-learning life cycle, while Shelby et al. [28] classify harms by sociotechnical impact.The impact categories include representational, allocative, quality-of-service, interpersonal, and societal harms.Source: §2.1 · p. 2

2.1.1 Harms associated with NLP.

  • NLP taxonomies cover allocation and representational harms, questionable correlations, and harms such as stereotyping, disparagement, dehumanization, erasure, and quality-of-service impacts [5] [14].Source: §2.1.1 · p. 2

2.1.2 Harms associated with CV.

  • Computer-vision harms include stereotyping, demeaning, and erasure in image tagging [32], alongside racial skew and caption differences in COCO [38].The cited COCO study reports a dataset skew toward light-skinned individuals and differences in caption word choice and sentiment.Source: §2.1.2 · p. 2

2.2 Harms due to (mis)use of generative models

  • Generative models can amplify existing harms or create new ones through biased training data, unrestricted input and output spaces, and interactive features [20].Source: §2.2 · p. 2

2.2.1 Harms associated with LLMs.

  • LLM harm research covers bias sources and evaluation [20], risk taxonomies [34], and RLHF limitations including noisy feedback and difficulty representing diverse backgrounds [8].Source: §2.2.1 · p. 2

2.2.2 Harms associated with VLMs and T2I models.

  • Multimodal risk work treats cross-modal alignment and VLM faithfulness, privacy, safety, and fairness [21] [19], as well as harmful T2I outputs [25].A broader taxonomy spans six stakeholder groups and 22 risk types [4].Source: §2.2.2 · pp. 2–3

2.3 Harms associated with the use of generative models in recommendation systems

  • Recommendation research covers LLM-generated embeddings, tokens, or direct recommendations [35], but implicit harms in non-conversational LLM–ML pipelines remain under-explored.Related work examines stereotype reinforcement [29], debiasing strategies [39] [37], retrieval biases [12], and chatbot responses to underspecified queries [16].Source: §2.3 · p. 3

3 Recommendation system components

Generative model-augmented recommenders retain familiar representational and quality-of-service risks while adding harms linked to model priors, prompt choices, and semantic misalignment. The section frames these risks through user and designer objectives, then distinguishes pull-oriented from push-oriented tasks.

3.1 Key stakeholders

  • Recommendation systems mediate interactions among users, content, and providers in ways reflecting user preferences and designer objectives and constraints.The section assumes domain-relevant sensitive attributes can describe individuals and groups, with attribute values mapped to natural-language values or options.Source: §3.1 · p. 3, §3.1.1 · p. 3

3.1.1 Human users.

  • Users provide inputs drawn from changing intent distributions, and the system may observe only a subset of their sensitive attributes.Inputs may reflect revealed or unrevealed sensitive attributes, other characteristics, interaction history, and interaction type; inferred attributes can affect outputs and create risk when they differ from true attributes.Source: §3.1.1 · p. 4

3.1.2 The system designer.

  • System designers intervene through model-lifecycle choices and, in generative settings, system messages, prompts, and in-context examples.Natural-language interfaces broaden who can express desired behavior, but intent-outcome gaps may become harder to detect when evaluation shifts to LLM-in-the-loop judgments that resist objective assessment and may reflect evaluator bias.Source: §3.1.2 · pp. 4–5

3.2 Common subtasks and potential harms

  • The section separates user-initiated pull tasks from designer-initiated push tasks that map users to content, influence behavior, or use logged interactions to improve future recommendations.Pull tasks include intent modeling, retrieval, and ranking; push tasks include data transformation, interventions, feedback evaluation, and synthesis.Source: §3.2 · p. 5, §3.2.1 · p. 5, §3.2.2 · p. 6

3.2.1 Pull-oriented subtasks.

  • Pull-oriented harms include derogatory portrayals or incorrect inferences about sensitive attributes, and QoS harms when recommendations ignore attributes users intended to matter.Ignoring relevant attributes can relatively over-represent content associated with default or privileged attributes; generative systems add causal drivers to these inherited risks.Source: §3.2.1 · p. 5
  • Generative recommenders may infer sensitive traits from partial user inputs, hallucinate unavailable items, or flatten within-group preference diversity during evaluation.These behaviors can misalign recommendations with user intent and create representational or quality-of-service harms; web-scale sociocultural biases and RLHF-induced confidence under uncertainty are additional drivers.Source: §3.2.1 · pp. 5–6

3.2.2 Push-oriented subtasks.

  • Push-oriented tasks can cause harm indirectly through data transformation and interventions, or directly when synthesis erases or offensively portrays sensitive attributes.Lossy input approximations may omit sensitive attributes or risk vectors, while semantic misalignment can also produce hallucinations involving sensitive groups or product safety.Source: §3.2.2 · pp. 6–7
  • Sanitization occurs when prompt- or RLHF-induced behavior positively portrays or obscures harmful inputs, as when an uplifting caption omits used syringes.Such sanitization may be an unintended consequence of explicitly prosocial prompting; the section also distinguishes it from hallucination by the input-output relation involved.Source: §3.2.2 · p. 7

4 Proposed Mitigation Strategies

The proposed mitigations distinguish useful user-provided sensitive-attribute terms from potentially harmful generated terms, then address semantic mismatch through prompt, filtering, and fallback strategies.

When and how to condition on sensitive attributes

  • Offline query logs can identify user-intended attribute terms, which can inform vernacular-aware termset detection that distinguishes them from sensitive terms in system outputs.The distinction targets terms used to improve relevance versus terms appearing in generated outputs.Source: §4 · p. 7
  • When input and output termsets overlap and remain generally neutral, harms risks are reduced; propagating useful terms downstream can also avoid generic or dominant-group-biased results.Source: §4 · p. 7
  • For unsupported, pejorative, or sensitive generated terms, the paper proposes harm-detector filtering, prompts for uncertainty, and fine-tuning penalties.Source: §4 · p. 7
  • Negative instructions may be insufficient in open-ended spaces, where long or competing constraints can be ignored and excessive instructions may produce sanitization or unwarranted pro-social content.Source: §4 · pp. 7–8

Ensure domain-specific and sociotechnical terms are defined in prompts

  • Define domain-specific and sociotechnical terms in prompts and select examples to cover their semantic space, since model interpretations may not match stakeholders’ assumptions.Source: §4 · p. 8

Anticipate semantic misalignment and provide fall-back options

  • Match prompt objectives to the input space, screen unsuitable inputs when feasible, invite refusal or uncertainty otherwise, and retain a non-generative fallback if mitigation is insufficient.Traditional predictive ML models may be more robust and trustworthy on some tasks.Source: §4 · p. 8

5 Conclusion & Future Work

The paper characterizes recommendation-system harms and links semantic misalignment to unintended consequences. It identifies benchmarking, harm detection, and prompt-validation techniques as future directions.

Conclusion

  • Representational and quality-of-service harms may persist or appear in new ways in generative recommendation systems, while semantic misalignment can produce unintended harms without malicious intent.Source: §5 · p. 8

Future Work

  • The paper identifies benchmarking datasets across recommendation tasks as a future direction for evaluating and comparing models.Source: §5 · p. 8
  • Future work also includes accurate harm detectors and robust prompt-constraint validation to improve identification of undesirable outputs and adherence to guidelines.Source: §5 · pp. 8–9

A.1 Representational Harm

The appendix illustrates representational harms across cold-start item recommendation, travel-content recommendation, and query refinement, including inferred sensitive attributes and stereotyped suggestions.

Cold-Start Item Recommendation

  • In cold-start fashion recommendations, GPT-4o infers gender from age and job title; for autobiographies, it infers political affiliation from gender, age, and location.Incorrect gender inference may lead to gender-specific rather than gender-neutral items and alienate users.Source: Table 3 · p. 11, §A.1.1 · p. 11

Content Recommendation

  • For travel tips, GPT-4o gives Ali more cautionary document and finance advice than Michael, potentially implying visa, financial, or connectivity problems.Source: Table 4 · p. 12, §A.1.2 · p. 12

Query Refinement

  • Query-refinement examples include unsolicited stereotypes and attribute inferences, from weight advice for tall women to inferred medical conditions for fat women.Some suggestions are unrelated or potentially dangerous, including a North Korea query refinement pointing to DMZ border adventures.Source: §A.1.3 · p. 12, Table 5 · p. 13

A.2 Quality-of-service Harms

The appendix presents examples of quality-of-service harms and identifies the associated tasks, prompts, and tested models.

Quality-of-service Harms

A.3 Sanitization Harms

Optional positive-tone instructions in image-captioning prompts can unintentionally sanitize, mask, or omit harmful aspects of an image.

A.3 Sanitization Harms

  • GPT-4o generates image captions from a brief prompt, optionally adding an instruction to use a positive, uplifting tone.Source: §A.3.1 · p. 15
  • Positive instructions can cause captions to sanitize, mask, or fail to acknowledge sensitive, explicit, or otherwise harmful input.Source: §A.3.1 · p. 15
  • Table 8 presents harms that could arise unintentionally from adding positive instructions to image-captioning prompts.Source: §A.3.1 · p. 15

A.3.1 Image Captioning.

Table 8 concerns image-captioning sanitization harms associated with prompt instructions.

A.3.1 Image Captioning.

  • The table addresses image captioning and associated sanitization harms.Source: Table 8 · p. 16
  • Its scope links sanitization harms to instructions in the prompt.Source: Table 8 · p. 16
  • The table title identifies image captioning as the setting for the prompt-instruction harms.Source: Table 8 · p. 16

Related papers

Loading 2609.33073v1…