PaperlayerView exact revision on arXiv

Source-linked AI summary

The Impact of Personalized AI Response Suggestions on Content Creators' Direct Messaging

Eun Jeong Kang, Jingruo Chen, Jadon Geathers, Susan Fussell

arXiv:2610.34770v1cs.HCcs.AI

TL;DR

Creators face heavy DM volumes while composing responses that balance audience expectations and their media personas. The paper studies Creact, a personalized AI response-suggestion system, and finds that creators value its help with tone but must also verify its outputs.

  • Problem

    Creators’ high-volume DMs make it difficult to sustain personal audience relationships while managing response labor and the boundaries between public and personal personas.

    Source: §1 · pp. 1–2

  • Method

    The authors used formative interviews with 15 creators to design Creact, then compared creators’ response behaviors and experiences using it in a study with 13 creators.

    Source: §1 · p. 2

  • Results

    Creators valued Creact’s ability to soften potential impoliteness and anticipated higher response rates, but reported extra work verifying AI authenticity and possible limits on individuality and relationship building.

    Source: §1 · p. 2

  • Takeaways & Limitations

    AI response support for creators should move beyond reflecting personas and increasing efficiency to account for creator–audience relationships and professional growth.

    Source: §1 · p. 2, §8.5 · p. 25

    The evaluation involved 13 US-based participants and examined creators’ perspectives and behaviors, limiting generalizability and leaving follower perspectives unassessed.

    Source: §8.4 · p. 25

Abstract

from arXiv · show

Direct messaging is often employed by social media content creators to directly interact with followers. Those creators attempt to engage with followers in private communication, but they often encounter a large volume of messages. Through formative interviews with 15 creators, we found that creators frequently experience fatigue from writing suitable content, as they moderate audience expectations while concurrently integrating their media personas into responses. To explore how AI can address these challenges, we designed a system providing personalized, AI-generated response suggestions tailored to creators' writing styles to maintain their media personas. In a comparison study with 13 creators that examines how the system affects response behaviors, creators valued the system's ability to soften potential impoliteness and anticipated higher response rates, helping them maintain a professional image. However, creators reported additional labor in verifying the authenticity of AI outputs to mitigate risks caused by AI misalignment. Our results indicate that while AI suggestions enhance efficiency, they can undermine a creator's engagement in communication by shifting their attention toward monitoring AI.

1 INTRODUCTION

Creators use DMs to build personal relationships with audiences, but high message volumes make response writing laborious and require creators to balance audience expectations with their public and personal personas. The paper introduces Creact, a personalized response-suggestion system, and reports both valued support and added verification work.

1 INTRODUCTION

  • DMs let creators build intimate audience relationships, but large volumes of incoming messages can burden the emotional and relational work of responding.Creators use this private channel to interact with fans, clients, and followers.Source: §1 · p. 1
  • In interviews with 15 creators, response writing involved integrating media personas and moderating reactions, often causing overuse of time, disengagement, short replies, or non-responsiveness.The six types of messages creators receive further complicate tailoring responses and maintaining persona boundaries.Source: §1 · p. 2
  • Creact generates two personalized response suggestions aligned with creators’ media personas, using their social-media data, writing style, conversation history, and the sender’s previous messages.Creators can use the system to generate or revise messages and customize their persona across message types.Source: §1 · pp. 2–3
  • In a comparison study with 13 creators, participants valued suggestions that softened potential impoliteness and anticipated higher response rates, but reported labor verifying authenticity and risks to individuality and relationship building.Participants composed longer messages and actively used AI features; the study compared Creact with a standard messaging interface alongside general-purpose AI tools.Source: §1 · p. 2

2 RELATED WORK

Prior work describes AI support for general interpersonal communication and creators’ strategic, laborious management of audience relationships, but leaves open how AI should support creators’ boundary-setting in DMs. This study uses a three-phase Research through Design approach to explore that gap.

2.1 AI-Supported Message Writing in AI-mediated Communication

  • Prior AI-mediated communication research mostly examines general users with shared communication goals, while AI support can affect perceived authenticity, trustworthiness, agency, and ownership.Context-aware support matters because social norms and interaction stakes vary, and ignoring them can lead to communicative breakdowns or relationship harms.Source: §2.1 · p. 3

2.2 Parasocial Interaction of Social Media Content Creators

  • Creators strategically tailor authentic self-presentation and intimacy, but this relational labor is emotionally demanding and can intensify as platform technologies require communication styles to change.Prior research had not sufficiently examined how interaction designs shape creators’ behavior in one-to-one messaging.Source: §2.2 · p. 4

2.3 Direct Messaging in Mass-personal Communication Channels

  • DMs combine private, personalized interaction with the reach and anonymity of public platforms, exposing users to risks such as scams, harassment, and unsolicited sexual content.Creators also balance the benefits and burdens of incoming messages, including monetization and relationship boundaries.Source: §2.3 · p. 5
  • The study addresses limited understanding of AI support for boundary-setting across broader creator personas through three Research through Design phases: investigating pain points, developing a persona-aware system, and assessing a prototype.A professional content creator co-authored the work and provided expert feedback throughout.Source: §2.1 · p. 4, §2.3 · p. 5

3 FORMATIVE INTERVIEW

The formative interviews examined how creators manage direct messages they cannot reasonably answer individually. Creators balance channel goals and audience expectations with persona-sensitive writing and personal boundaries.

3 FORMATIVE INTERVIEW

  • The study asked how creators write and send direct messages and what challenges arise when incoming volume exceeds what they can reasonably manage.Researchers conducted 15 semi-structured interviews and analyzed transcripts thematically; they also reviewed 203 publicly shared DM excerpts to validate the typology, with initial inter-rater reliability of 0.905.Source: §3 · pp. 5–6, §3.2 · p. 6

3.3.1 A Broad Spectrum of Direct Messages, from Personal to Professional.

  • Creators received messages spanning general social interaction to professional inquiries, with demands ranging from low-effort reactions to messages requiring knowledge and professional judgment.Personal inquiries from followers could cross boundaries creators did not wish to share publicly; repetitive or negative messages could cause emotional and cognitive distress.Source: §3.3.1 · p. 7

3.3.2 Balancing Reaction Management Across Different Message Types based on Channel Goals.

  • Creators prioritized messages aligned with channel goals, mutual followers, or potential clients, while ignoring or de-prioritizing irrelevant or uncomfortable messages.Some creators gave brief, non-committal replies to intimate requests to prevent further conversation.Source: §3.3.2 · p. 7

3.3.3 Stylizing Responses Tailored for Imagined Audience Groups and Channel Personas.

  • Creators presented distinct personas in responses, commonly combining professional and intimate styles, while some adopted a more straightforward style to shield against harassment.They adjusted tone and detail for anticipated recipients; clients could receive formal replies and prepared information to avoid mistakes.Source: §3.3.3 · p. 7
  • Creators tailored responses to audience groups rather than applying fixed persona traits uniformly, using formal tones with clients and different styles with followers, friends, and family.Some creators also used ChatGPT to polish response tone.Source: §3.3.3 · pp. 7–8
  • Creators struggled to choose wording that conveyed intent without offending recipients while keeping their personas authentic and approachable, especially when declining requests or negotiating.They described insufficient information about audiences as a challenge to audience design.Source: §3.3.3 · p. 8

3.3.4 Self-Control Reactions to Avoid Over-Engagement of Individuals.

  • Creators varied what they shared by audience familiarity and priority, maintaining personal boundaries while trying to remain friendly and relatable.They calculated how much further interaction effort to spend and found it difficult to deter personal questions without seeming unfriendly.Source: §3.3.3 · p. 8, §3.3.4 · p. 9
  • Detailed, empathetic replies could invite unwanted follow-ups or harassment, so some creators kept responses brief or used emojis to limit conversation length.Crafting lengthy replies was also described as exhausting.Source: §3.3.4 · p. 9

4 DESIGN GOALS

The formative findings informed four design goals for creator-focused direct messaging tools. They address persona alignment, strategic response management, expressing intent, and efficient yet credible AI assistance.

4 DESIGN GOALS

  • The design goals were derived from formative-study findings and prior literature to guide response-composition features for creators.Source: §4 · p. 9
  • DG1 calls for reflecting creators’ curated text, visual, and interactional cues to maintain a coherent professional image across communication contexts.Source: §4 · p. 9
  • DG2 calls for tailoring AI assistance by DM type and priority to support reaction management according to situational demands and channel goals.Source: §4 · p. 9
  • DG3 calls for helping creators express their intentions with appropriate language, while DG4 calls for context-grounded generation, review, and verification with minimal effort to reduce latency without sacrificing credibility.Source: §4 · p. 9

5 CREACT

Creact is an AI-powered direct-message writing tool that personalizes suggestions using creators’ social media personas and writing-style preferences. Its interface lets creators configure styles, review message context, and revise AI outputs before sending.

5 CREACT

  • Creact provides personalized response suggestions based on a creator’s social media persona, which combines channel characteristics and writing-style preferences.Source: §5 · p. 10

5.1 Overview

  • The interface guides creators from setting writing preferences through category and persona controls to reviewing thread summaries and opening individual messages.Source: §5.1 · p. 10

5.2 Constructing a Social Media Persona based on Social Media Content

  • The system extracts recurring stylistic features from creators’ posts, including emotional register, tone, word choice, and emoji use, and uses embeddings of the same text as generation context.Creators can review, edit, or remove extracted attributes in the interface.Source: §5.2 · p. 11

5.3 Adopting Persona Styles and Content to Use for Varying DM Types

  • Creact classifies incoming DMs into six categories so creators can prioritize responses and apply category-specific settings; fixed, legible categories were chosen over inferred audience groups for easier correction and fewer sender-data privacy concerns.Source: §5.3 · pp. 11–12

5.3.1 Writing style setting.

  • Writing Style Settings let creators adjust expressivity, emoji use, capitalization, and whether responses draw on their social media content across DM types and individual conversations.Creators can exclude personal or professional information from selected categories to maintain privacy or professional distance.Source: §5.3.1 · p. 12

5.3.2 Scenario Simulation.

  • Scenario Simulation generates topics and simulated chats so creators can preview and iteratively tune whether AI-generated tone matches their desired self-presentation.Source: §5.3.2 · p. 12

5.4 Generating Personalized AI-based Response Suggestions

  • Response Suggestion generates a full reply, while Enhance Message refines a creator’s draft, offering complementary levels of assistance while preserving authorial agency.Generation draws on conversation history, inferred DM type, sender intent, whether the message starts a conversation, and the creator’s persona; optional retrieval adds relevant prior-post snippets.Source: §5.4 · pp. 12–13

5.5 Implementation

  • A review panel lets creators verify and revise generated replies before sending; the system was implemented with GPT-4, LangChain, NextJS, TypeScript, Django REST, and Heroku.Source: §5.4 · p. 13, §5.5 · p. 13

6 EVALUATION

The evaluation used a mixed-method, within-subjects comparison of Creact with a standard messaging interface and ChatGPT. It combined response tasks, surveys, behavioral logs, interviews, and thematic and statistical analyses.

6.1 Participants

  • Thirteen U.S.-based creators with more than 10,000 followers, all active on Instagram, participated in the evaluation.One participant had also taken part in the formative study.Source: §6.1 · p. 14

6.2 Study Design

  • In a within-subjects design, participants used Creact and a standard messaging interface with optional ChatGPT assistance.The comparison was intended to assess creators’ holistic experiences of the prototype.Source: §6.2 · pp. 14–15
  • Each condition involved responding to eight direct messages within a 10-minute limit, using stimuli representing six DM typologies.Researchers created two instances for selected persona-related and partnership subcategories.Source: §6.2 · p. 15, §6.2.1 · p. 15

6.2.1 Stimuli.

  • Stimuli drew on creators’ prior DM cases and used relational cues such as direct language, punctuation, intentional misspellings, and emojis.Authors generated candidates with GPT-4, screened them for category fit, ecological validity, and interactional complexity, then selected non-overlapping messages for the conditions.Source: §6.2.1 · p. 15

6.2.2 Study Procedure.

  • Sessions included consent, persona setup from anonymized Instagram posts, a pre-task survey, a walkthrough, response tasks, a post-task survey, and an interview.Participants were randomly assigned to condition order; a researcher facilitated ChatGPT responses in the baseline condition.Source: §6.2.2 · pp. 15–16

6.2.3 Measurements.

  • Measures covered response preferences, efficiency, self-presentation, agency, controllability, satisfaction, and observed AI-feature use.Efficiency included NASA-TLX cognitive load, response time, and character counts; interviews probed feature use, settings, and observed revisions.Source: §6.2.3 · p. 16

6.3 Data Analysis

  • The analysis used Wilcoxon signed-rank tests for paired survey and feature-use data, and thematic analysis of interviews for perceived values and challenges.Feature use was coded as any suggestion or enhancement use versus none; DM preferences were categorized relative to each participant’s median ratings.Source: §6.3 · p. 17

6.4 Positionality Statement and Ethical Considerations

  • The U.S.-based research team noted that its positioning may shape what it notices or overlooks, while creators elsewhere may navigate platform governance differently.Researchers obtained consent to process public social media data, reviewed and transformed potentially identifying content, and excluded stimuli with explicit negative intent or profanity.Source: §6.4 · p. 17

7 FINDINGS

Creators produced longer responses with Creact and valued its support for professional self-presentation and interaction management. They also described burdens and risks, including setup effort, flattened personas, and labor to check AI-generated messages.

7.1 SMCs’ Response Composition Behavior

  • 164.37 characters per response with Creact versus 76.95 with baseline; category-level differences did not remain significant after Holm correction.Across the eight message categories, output was generally higher with Creact; the reported category effects were not robust to multiple-comparison correction.Source: §7.1 · pp. 17–18, Table 4 · p. 18

7.3 SMCs’ Perceived Self-Presentation Ability

  • Participants reported greater tone and message-refinement confidence with Creact, while self-image regulation and social-openness differences were not significant.Participants described Creact as a “persona mask” that could support consistent, professional self-presentation.Source: §7.3 · p. 19, §7.4 · p. 19

7.4.1 Persona Masks That Support Professional Self-Presentation.

  • Participants valued AI as an emotional buffer for difficult messages and expected it to help them respond to more people while maintaining their persona.They also described adjusting tone and detail to audience expectations, including using formal business language for partnership inquiries.Source: §7.4.1 · p. 19

7.4.3 Different Roles of AI Response Suggestions.

  • Response Suggestion offered a quick starting point or low-effort reply, while Enhance Message was seen as more authentic because creators drafted first.Some used suggestions to avoid sharing personal information; others said enhancement could elaborate their own ideas and better capture their intent.Source: §7.4.3 · p. 20

7.4.4 Acceptable Social Media Data Use for Making AI Skillful.

  • Participants generally accepted social-media data use for personalization, but some wanted clearer explanations and worried about including personal-account data.Participants believed data use could help AI reflect their personalities and reduce response-writing effort.Source: §7.4.3 · p. 20, §7.4.4 · pp. 20–21

7.5.1 The Need to Reduce Effort for On-boarding and Personalization.

  • Configuring style settings across DM types felt overwhelming, and the system may work poorly for creators with sparse data or an undeveloped persona.Participants said setup effort could make manual replies preferable, especially when they lacked an established way of responding to audiences.Source: §7.5.1 · p. 21

7.5.2 Multiple Dimensions of Personalization Beyond Writing Style.

  • Repeated stylistic patterns could flatten creators’ digital presence, and surface-level controls did not capture nuanced language or emotional expression.Participants cited habitual language, slang, humor, and empathy in sensitive contexts as difficult to represent through existing controls.Source: §7.5.2 · p. 21

7.5.3 New Labor added to Avoid Potential Relational Harms.

  • Participants manually reviewed and edited AI messages to preserve authenticity, while worrying that inconsistent or off-brand replies could disrupt fan relationships.They reported no significant difference in cognitive load or perceived controllability between conditions.Source: §7.5.3 · p. 21

8 DISCUSSION

Creact can buffer the strain of repetitive DM work and help creators maintain a professional persona, but it also shifts labor toward monitoring AI and raises risks around authenticity, cultural fit, and consent. The discussion recommends creator-centered, adaptable support while noting the study’s limited scope.

8.1.1 Protective Shield.

  • AI suggestions can buffer the strain of repetitive or demanding messages, but replies to threatening or abusive senders may give harassers an opening to escalate.The discussion favors helping creators selectively invest in audiences they want to build closeness with, rather than merely enduring draining interactions.Source: §8.1.1 · p. 22

8.1.2 Professional Persona Mask.

  • Creact can strengthen a consistent professional presentation, yet over-reliance may undermine perceived authenticity and misaligned outputs can add relational labor.The discussion identifies novice creators and creators facing language or professional-terminology challenges as possible beneficiaries; audiences’ perspectives were not evaluated.Source: §8.1.2 · pp. 22–23

8.2 Situating AI Writing Support Across Cultures and Platforms

  • Personalization must account for cultural norms and platform differences, rather than treating a creator’s persona as a uniform style transferable across contexts.The discussion cautions that culturally ungrounded suggestions may feel generic or tone-deaf, while platform-specific affordances can change which aspects of self creators foreground.Source: §8.2.1 · p. 23, §8.2.2 · p. 23, §8.2 · p. 23

8.3.1 Evolving AI for Professional Growth through Effortless Onboarding.

  • Onboarding should be lightweight and contextual, with style configuration evolving through creator feedback rather than requiring extensive manual setup.The discussion notes that some creators may struggle to specify subtle stylistic elements and that preferences shift with platform trends and audience reactions.Source: §8.3.1 · p. 24

8.3.2 Prioritizing Authenticity via Creator-Driven Workflows; 8.3.3 Preserving Individuality and Avoiding Persona Flattening.

  • DM workflows should preserve creator agency and individuality, since authenticity can depend on personal effort and LLM-based outputs risk flattening distinctive styles.The discussion favors flexible stylistic adjustments and comprehensive data and evaluation choices over relying only on dominant online personas.Source: §8.3.2 · p. 24, §8.3.3 · p. 24

8.3.4 Clear Data Usage Consent Considering Messaging Partners and Creators’ Usage Patterns.

  • Personalization using private DM data requires clear, communicative consent that considers conversation partners as well as creators.The discussion says partners may not agree to data use even when creators do, and suggests shared understanding of permissible use, potentially signaled through preference labels.Source: §8.3.4 · pp. 24–25

8.4 Limitations; 8.5 Conclusion

  • The evaluation involved 13 US-based creators and did not assess recipients’ perspectives, limiting generalizability and claims about audience authenticity or engagement.The study also evaluated a holistic intervention combining multiple personalization components; the conclusion argues that creator-facing support should extend beyond efficiency.Source: §8.4 · p. 25, §8.5 · p. 25

B.1 Conversation history summary; B.2 Response Suggestion; B.3 Enhance Message

  • The appendices specify prompts for summarizing conversation context and generating or revising creator-voiced replies.The response-generation prompt uses conversation history, sender intent, creator style, and message context; generated replies are instructed to stay under 100 words.Source: §B.1 · p. 31, §B.2 · pp. 32–33, §B.3 · p. 33

C PROMPT ENGINEERING FOR EXPERIMENTAL SETUP

The experimental prompt asks for realistic follower DMs tailored to a creator and message category. It specifies voice, length, and variation constraints to make the generated messages suitable as study stimuli.

C.1 Stimuli Generation

  • Generate 30–40 unique, realistic direct messages from imagined followers for experimental use in studying creator–audience interactions.The prompt asks for each message to target a specified creator and message category, with no explanatory headers in the output.Source: §C.1 · p. 34
  • Messages should use informal, emotionally expressive language and a parasocial tone, with high immediacy and varied emotional composure.The prompt gives first- and second-person language, emojis, and playful punctuation as examples of the desired style.Source: §C.1 · p. 34
  • Each message should include the creator’s name naturally and run one to three sentences.Source: §C.1 · p. 34
  • The prompt defines eight message categories, spanning personal and creator-related inquiries, content and knowledge questions, partnerships, compliments, critique, and general or meme-like messages.Source: §C.1 · p. 34
  • Messages should vary in fan voice and emoji use, favor open-ended questions, and align their tone with the assigned category.The prompt notes that partnership messages may be more formal than other categories.Source: §C.1 · p. 34
  • The output should contain only the generated messages, without category labels or explanations.Source: §C.1 · p. 34

D INTERVIEW SCRIPT

The interview script asks creators about their personas, their experiences with AI and non-AI writing tools, and how they compose responses for different messages. It also probes personalization, relationship maintenance, and data-use preferences.

D INTERVIEW SCRIPT

  • Creators are asked how they describe their online persona or brand and adapt it across social media activities and writing styles.Source: D INTERVIEW SCRIPT · p. 34
  • The script probes overall tool impressions, choices between AI and non-AI assistance, message revision, and differences in tool use across messages.Source: D INTERVIEW SCRIPT · p. 35
  • Questions examine how creators use response generation and message enhancement, which situations suit AI help, and when they distrust or reject suggestions.The script also asks about burdensome situations and how the features could better reflect specific contexts.Source: D INTERVIEW SCRIPT · p. 35
  • Creators are asked whether tone settings match the persona they want to project and how personalization might adapt to message types or audiences.Source: D INTERVIEW SCRIPT · p. 35
  • The script asks about relationship effects, real-workflow adoption, developer priorities, social-media or message-history data, and information creators would exclude from training.It closes with an open invitation to share anything else about the experience.Source: D INTERVIEW SCRIPT · pp. 35–36

E.1 Descriptive Analysis

The descriptive analysis reports frequent use of Creact’s AI features and breaks down how participants composed responses in the Creact condition.

E.1 Descriptive Analysis

  • 1.04 times per response was the average use of Creact AI features, compared with 0.182 times per response for AI assistance in the baseline.Creact feature use included response suggestions, message enhancement, or both.Source: §E.1 · p. 36
  • 3.00 times (SD = 2.83) was the average use of response suggestion-only, while enhance message-only was used 2.15 times (SD = 2.44).Source: §E.1 · p. 36
  • 1.31 times (SD = 1.44) was the average combined use of response suggestions and message enhancement; edited suggestion outputs averaged 0.31 times (SD = 0.63).Source: §E.1 · p. 36
  • Manual message writing averaged 1.15 times (SD = 1.14), while participants declined response composition 0.08 times (SD = 0.28).Source: §E.1 · p. 36

E.2 Response Examples

Table 10 presents examples of responses written by participants across conditions.

E.2 Response Examples

  • Table 10 presents participant-written response examples across conditions.Source: Table 10 · p. 37
Loading 2610.34770v1…