Evaluating the Impact of Personalization in Conversational Cybersecurity Assistants

• Across an automated evaluation and a 7-day user study, conversation-based personalization improved perceived helpfulness and likelihood of following advice, while static background personalization offered limited gains over a strong PMT baseline.Source: §1 · p. 2
• The study compares four personalization strategies, including profile-based approaches, expertise-aware few-shot prompting, and conversation-based personalization using prior questions, answers, and user feedback.Source: §1 · p. 2
• The automated evaluation used 1,045 real-world cybersecurity questions, followed by a 7-day deployment involving 57 participants and 1,066 questions.Source: §1 · p. 2
• Automated LLM-evaluation trends aligned with human evaluation, suggesting a scalable way to compare personalization strategies before costly user studies.Source: §1 · p. 2
• The findings suggest behavior-driven personalization is promising for LLM-powered cybersecurity assistants and support combining LLM and human evaluation.Source: §1 · p. 2

• Users may prioritize productivity over security, so accurate information alone can leave a gap between cybersecurity knowledge and action (Acquisti et al., 2015; Redmiles et al., 2016b).Source: §2 · p. 2
• PMT-based prompting can improve motivational quality and self-reported likelihood of following advice, while personalization may further improve LLM-generated advice.Source: §2 · p. 2
• Personalization can use explicit user information, few-shot examples, or interaction histories; memory-based approaches adapt over time from prior interactions and feedback (Brown et al., 2020; Xu et al., 2026; Chen et al., 2024; Zhang et al., 2025; Zhao et al., 2025).Source: §2 · p. 2
• This work compares personalization strategies across answer quality, motivating power, perceived usefulness, and willingness to follow advice, and examines whether LLM-evaluation trends correspond to human evaluations.Source: §2 · p. 2

3 Combining LLM and Human-Based Evaluations of Personalization Strategies

• The evaluation combines large-scale LLM-based assessments across a question corpus with an in situ user study to check whether automated trends translate to real users.Source: §3 · p. 3
• The automated evaluation used 1,045 real-world cybersecurity questions and corresponding answers, with background and behavioral data enabling several personalization forms.Source: §3 · p. 3

3.1 Personalization Strategies

• P1 and P2 personalize PMT prompts with selected or all survey responses, P3 uses expertise-tailored examples, and P4 adapts from prior questions, answers, alternatives, and feedback.Source: §3.1 · p. 3

3.2 LLM-Based Evaluation Procedure

• The automated procedure used GPT-4-turbo to compare personalized answers with a PMT-only baseline on understandability, actionability, and motivating power, without revealing answer origin to the evaluator.Selected strategies also received independent ratings, and the study compared the direction and relative magnitude of contrasts across LLM-based and human evaluations.Source: §3.2 · p. 3

• The assistant checked submitted questions for computer-security relevance, generated GPT-4.1 responses using PMT-based prompting and personalization, and logged interactions.Source: §4 · p. 3
• During interaction, each API call used only the current day’s message history; final-survey conversation personalization was separately constructed from the full 7-day history.Source: §4 · pp. 3–4
• The Chrome-embedded chat interface supported conversational questions and was designed for security questions arising during users’ everyday online activities.Source: §4 · p. 4

5 User Study and Evaluation

• The 7-day study enrolled participants to ask cybersecurity questions arising from daily online activities; 57 completed it and contributed 1,066 questions.Participants were assigned to PMT-only control or background-based personalization, and provided structured feedback on answers.Source: §5 · p. 4
• Participants completed an intake survey, interacted with the assistant for seven days, and provided feedback that contributed to histories used for conversation-based personalization.The interaction phase required at least two unique cybersecurity questions per day.Source: §5 · p. 4

5.1 Final Survey: Evaluation of Conversation-Based Personalization

• For the final survey, each participant compared original answers with newly generated answers conditioned on accumulated interaction history.Participants evaluated five previously asked questions; histories included other study questions, answers, and evaluations, with at least 13 prior interactions.Source: §5.1 · p. 4
• Control participants compared P4 with PMT-only answers, while experimental participants compared P4 with background-personalized answers, rating helpfulness and likelihood of following advice.The study also conducted optional post-study interviews with 10 participants for qualitative context.Source: §5.1 · p. 4, §5 · p. 4
• Analyses accounted for repeated participant measurements, used Benjamini–Hochberg correction across seven final-survey tests, and checked participant-level results when mixed models had singular fits.Participant-level analyses produced the same qualitative conclusions.Source: §5.1 · p. 4
• The evaluation framework describes GPT-4-turbo comparisons across three effectiveness metrics and independent 1–10 ratings for P1 and P4.Table 1 uses a −2 to 2 scale, with positive scores favoring personalized answers; Table 2 rates higher scores as better responses.Source: Table 1 · p. 5, Table 2 · p. 5

6.1 Automated LLM-Based Evaluation Results

• Across understandability, actionability, and motivating power, P1 and P2 scored negatively, P3 positively, and P4 highest; all observed differences were statistically significant.The reported ordering was P1 and P2 below baseline, P3 with moderate improvements, and P4 with the largest gains.Source: §6.1 · p. 5

6.2 In Situ Study Overview

• Of 168 intake-survey participants, 57 completed all seven days; participants asked 1,066 questions spanning cybersecurity topics.The 57 completers included 35 experimental and 22 control participants.Source: §6.2 · p. 6

6.3 Interaction-Phase Results

• During the interaction phase, evaluations showed no strong preference between background personalization and the already high-quality PMT-only baseline; the phase primarily gathered feedback and behavioral signals for P4.Daytime evaluations slightly favored baseline answers, while direct comparisons slightly favored background personalization.Source: §6.3 · p. 6

6.4 Effect of Conversation-Based Personalization: Results

• P4’s comparative advantage was not explained by response length, and LLM- and human-evaluation trends had identical directional ordering across shared contrasts.Background-based and conversation-based response lengths did not meaningfully differ (p = 0.21; d ≈ 0).Source: §6.4 · p. 7
• Conversation-based personalization significantly improved comparative helpfulness and likelihood-of-following ratings against both no-personalization and background-based baselines.The comparisons accounted for repeated evaluations within participants and applied FDR correction.Source: §6.4 · pp. 6–7, Table 3 · p. 7

6.5 Personalization Effects: A Few Examples

• Conversation-based personalization adapted advice to users’ tools, behaviors, and prior experiences, while interaction history helped infer intent and provide more targeted guidance.The observed improvements tended to depend on behavioral signals rather than self-reported attributes.Source: §6.5 · p. 8

6.6 Relevance of User Questions to Real-World Context

• Participants described questions tied to real security concerns, asked throughout the day, and spanning diverse cybersecurity domains.Most participants reported that their questions concerned real issues rather than primarily meeting study requirements.Source: §6.6 · p. 8

6.7 User Experience and Behavioral Insights

• Participants used the assistant for everyday security concerns and valued clear explanations, but effort shaped action: quick tasks were often done immediately while time-intensive advice was deferred.Participants also reported greater security awareness and considered more involved recommendations worthwhile even when postponed.Source: §6.7 · pp. 8–9

7 Discussion

• Conversation-based personalization yielded the most reliable improvements in user-perceived effectiveness, while static background-based personalization showed mixed or limited gains over a strong PMT baseline.The discussion concludes that how personalization is implemented matters substantially.Source: §7 · p. 9

7.1 From LLM-Based Evaluation to Behavioral Personalization

• LLM evaluations favored conversation-based answers across understandability, actionability, and motivating power, and the in situ study found significantly higher comparative ratings for helpfulness and likelihood of following advice.Helpfulness differences were modest, but the comparative judgments remained significant after accounting for repeated evaluations within participants.Source: §7.1 · p. 9
• Agreement between LLM-based and human evaluations suggests that automated evaluation can help compare personalization strategies before costly user studies.The paper notes that convergence across methods strengthens confidence that the improvements reflect user perceptions rather than artifacts of model-based evaluation.Source: §7.1 · p. 9
• Interaction history lets responses adapt to users’ evolving concerns and preferences, whereas static surveys may miss context captured by observed behavior.Qualitative examples show adaptation to users’ existing tools and behaviors, prior experiences, and targeted guidance.Source: §7.1 · p. 9
• Because deployed assistants naturally accumulate interaction logs, follow-up questions, and feedback, interaction-driven personalization can be supported without additional instrumentation.The discussion suggests moving beyond static profiles toward adaptive approaches.Source: §7.1 · p. 9

7.2 Privacy and Human-Centered Design Considerations

• Richer behavioral signals raise privacy, consent, and data-use questions; the study’s interaction histories came voluntarily from users’ assistant conversations.Future systems may have access to much richer behavioral context.Source: §7.2 · p. 9
• Trustworthy behavior-driven assistants require meaningful user transparency and control, balancing personalization with user agency and privacy.The discussion identifies this balance as especially relevant in domains such as cybersecurity, where the goal is to influence behavior.Source: §7.2 · p. 9

7.3 Limitations

• The evaluation measures perceptions and self-reported likelihood of following advice, not directly observed real-world behavior; it also relies partly on an LLM judge.A real-world user study partially mitigates the LLM-judge limitation, while additional human evaluation could strengthen the connection.Source: §7.3 · p. 10

8 Conclusion

• Conversation-based personalization improved comparative ratings of perceived helpfulness and self-reported likelihood of following advice across both studies, outperforming static background-based approaches and a PMT-based baseline.The evaluation used a corpus of real-world cybersecurity questions and a seven-day in situ study with 57 participants and 1,066 questions.Source: §8 · p. 10
• Agreement between LLM-based and human evaluations suggests that LLM evaluations may offer a scalable way to compare personalization strategies before costly user studies.The paper identifies behavior-driven personalization as a more effective foundation than static user profiling.Source: §8 · p. 10
• Conditioning on observed user behavior is presented as a promising mechanism for narrowing the gap between providing security advice and encouraging users to act, while preserving privacy and agency remains important.The conclusion emphasizes the need to understand how behavioral context can be leveraged while maintaining user privacy and agency.Source: §8 · p. 10

A.1 Prompts Used for Automated Personalization Evaluation

• Table 4 presents prompts used to generate personalized answers for the four personalization strategies.Its caption says the prompt type identifies the personalization type.Source: Table 4 · p. 12
• In Table 4, a “+” marks appended text, including the user question.Source: Table 4 · p. 12
• Table 5 presents evaluation prompts, distinguishing comparisons of two answers (“multi”) from ratings of an individual answer (“ind”).The caption says appended content can include the question and one or more answers.Source: Table 5 · p. 13

A.2 Full Context for Personalization Evaluation

• The low-comfort P3 example asks for more explanation and less jargon, and illustrates this style with advice on ransomware, compromised devices, and public Wi-Fi.Source: §A.2 · p. 14
• The high-comfort example requests detailed answers and illustrates them with technical ransomware guidance, including cryptography, backups, endpoint protection, and incident response.Source: §A.2 · p. 15
• Further examples show technical guidance for suspected device compromise and untrusted public Wi-Fi, including forensic indicators, isolation, VPNs, and network defenses.Source: §A.2 · pp. 15–16
• Table 6 supplies full personalization prompt contexts, including P3 examples for three technology-knowledge levels and P4 accumulated questions, answers, and user evaluations.Source: Table 6 · p. 16

A.3 Example Answers

• Examples contrast generic password guidance with Bitwarden-tailored advice and compare device-choice recommendations alongside banking safeguards.The conversation-based example recommends smartphone banking with the official app, biometric sign-in or a passkey, and a clean computer browser.Source: §A.3 · pp. 17, 19, Table 7 · p. 20

A.4 Evening and Final Survey Questions

• Final surveys omitted repeated explanations for advice-following and used a 1–4 forced-choice helpfulness scale rather than daily surveys’ 1–5 scale.The forced-choice scale removed a neutral midpoint to elicit clearer comparative judgments.Source: §A.4 · p. 21, Table 8 · p. 21

A.5 Relevance of User Questions to Real-World Context

• Table 9’s weighted average maps response categories—Almost never = 0, Rarely = 0.25, About half = 0.5, More often than not = 0.75, Almost all of the time = 1—to estimate the overall fraction of questions in each category.“Real Issue” asks how many questions concerned real security issues or situations encountered; “Study Req.” asks how often participants asked primarily to meet study requirements rather than because they were already wondering about the question.Source: Table 9 · p. 22

A.6 Distribution of Question Times

• Participant questions, aside from sleeping hours, were spread throughout the day; Figure 1 aligns question times to a 5 pm Eastern cutoff.Questions after 5 pm count toward the next study day.Source: Figure 1 · p. 22

A.7 Distribution of Question Topics

• Table 10 separates topic counts for experimental and control questions and describes a wide range of topics in both conditions.Some questions fall under multiple topics.Source: Table 10 · p. 23

A.8 Sustained Interest in the Tool.

• Table 11 reports responses to whether participants would be interested in continuing access to the tool.Source: Table 11 · p. 23

A.9 Evening Survey Results

• Table 12 reports evening-survey responses averaged across questions and days, marking significant control–experimental differences in green at α = 0.05.The results cover 97 participants and 1,066 questions.Source: Table 12 · p. 24

A.10 Directional Consistency Across LLM-Based and Human Evaluations

• Table 13 reports human and LLM comparative ratings on a −2 to 2 scale, where positive values favor P4.It covers P4 versus P1 and P4 versus the PMT-only baseline.Source: Table 13 · p. 25