Source-linked AI summary
Who Falls for SMiSh? Learning Through Survey Data Where to Best Target Awareness Training for Mobile Messaging Attacks
Cori Faklaris, Sarah Tabassum, Heather Richter Lipford
TL;DR
SMiShing is widespread, but evidence is limited on which mobile users are most vulnerable. Two large-scale surveys found that younger people and college students were particularly vulnerable, while participants struggled to identify legitimate messages.
Problem
Evidence is limited on who is most vulnerable to SMiShing, hindering more targeted interventions than a one-size-fits-all approach.
Method
Two large-scale online surveys assessed demographic vulnerability to SMiShing among representative samples of U.S. adult mobile phone users.
Results
Younger people and college students were particularly vulnerable, while participants struggled to identify legitimate messages; in Study 1, 23.5% identified real messages correctly.
Takeaways & Limitations
Researchers, regulators, and telecoms should consider targeted awareness interventions for young adults and college students, alongside improved mobile warnings and verifications.
Takeaways & Limitations
The cross-sectional design cannot establish cause-and-effect relationships, and the surveys may be affected by self-report and social desirability biases.
Abstract
from arXiv · showhide
As mobile phone adoption has surged, so have scams involving these devices. One such scam, known as SMiShing (or smishing) after Short Message Service (SMS), involves fraudsters sending phishing links via mobile texts. Despite the prevalence of SMiShing, there is a lack of data on who is most vulnerable to these attacks. Prior research on phishing (its email counterpart) suggests that susceptibility may vary by demographic and contextual factors. In two large-scale surveys, we use a previously published simulation method to collect data from representative samples of U.S. adult mobile phone users. Our findings indicate that younger individuals and college students are particularly vulnerable. Participants struggled to correctly identify legitimate messages, with the second study providing comparisons of financial message variants. Researchers, regulators, and telecoms can help users by creating mobile-specific interventions for under-24 and university customers and adding verifications and warnings.
1. Introduction
This introduction frames SMiShing as a growing mobile scam whose demographic and contextual vulnerabilities remain insufficiently understood. Two surveys of U.S. adult mobile phone users address identification accuracy, vulnerable groups, and associations with prior training or relevant experiences.
- Introduction: As mobile-phone use has expanded, device-targeting scams have increased, and SMiShing uses phishing links sent by text messages while commonly impersonating banks, delivery companies, retailers, and communication providers.By the end of 2022, phone calls and texts were the most common way scammers contacted people in the U.S.
- Introduction: Prior email-phishing research suggests demographic and contextual variation in vulnerability, but no study had systematically identified which factors make people more likely to fall for SMiShing.The gap matters for developing effective awareness training and interventions.
- Introduction: The paper reports two large-scale online surveys of U.S. adult mobile phone users using simulated real and SMiSh messages, with demographic, cognitive, and behavioral data collected to answer three research questions.The first survey included 1,007 users, and the follow-up examined variants of a Zelle bank payment notification.
- Introduction: Study 1 participants identified fake messages more accurately than real ones (81.4% versus 23.5%), while Study 2 showed 68.7% accuracy for fake messages and 58.8% for real ones.Younger people and college students scored worst on the online SMiShing assessment, while a previously observed gender difference did not appear.
- Introduction: The authors recommend targeted risk-awareness campaigns and educational interventions for young adults and college students, reporting no significant effect of self-reported security awareness or SMiSh-relevant training.They also recommend improved mobile-message verification and guidance, with regulators and telecom providers working with usability experts.
2. Background and Related Work
Prior research establishes phishing and SMiShing as socially engineered threats, but demographic susceptibility findings from email phishing remain unclear for mobile messaging. This work addresses that gap with representative U.S. surveys examining demographic and contextual factors associated with SMiShing vulnerability.
- Present study: The study bridges this gap by using large-scale, representative U.S. mobile-user surveys to examine demographic and contextual factors associated with SMiShing vulnerability.The samples measure gender, age, educational attainment, and college experience, enabling comparisons with prior phishing and SMiShing findings.
- Phishing research: Phishing research links susceptibility to message design, wording, URLs, spoofed brands, context, and user characteristics, with some studies finding greater vulnerability among women and younger people.Training and educational materials can reduce unsafe information entry, although they may also reduce users’ tendency to click legitimate links.
- SMiShing background: SMiShing uses fraudulent SMS messages to provoke clicks, calls, or disclosure of personal information, often exploiting emotion and trust in brands or authorities.Compared with email phishing, text messages provide fewer detection cues and are perceived as more personal and urgent, increasing vulnerability.
- SMiShing research: Existing SMiShing studies suggest that users attend more to message content than sender information and that personalized or spoofed messages are more effective [30], but often omit legitimate-message comparisons.These limitations motivate controlled comparisons of real and fraudulent SMS messages.
- Study design: Study 1 adapts prior SMiShing research and displays simulated scam and legitimate messages, while Study 2 experimentally tests real and fake variants of one message.Messages include URLs and cues such as source identifiers, typos, grammar, and spacing to support participant judgments.
3. Methods
The researchers conducted two online surveys of U.S. adult mobile-phone users to examine demographic and contextual vulnerability to SMiShing. Study 1 used 14 simulated messages, while Study 2 compared real and fake Zelle notifications.
- Study 1: Study 1 presented 14 simulated SMS messages, evenly divided between legitimate and fraudulent, with URLs and reward- or fear-based motivators.Messages adapted prior research and real-world examples involving entities such as the IRS, Walmart, and Facebook, avoiding the ethical risks of live attacks.
- Measures and piloting: Ground-truth message labels converted participants’ ratings into binary correctness outcomes without adding another survey item, while Figure 1 documented the message stimuli.The researchers piloted the questionnaires through cognitive interviews, expert review, and Prolific surveys; the target completion times were about 12 minutes for Study 1 and 6–8 minutes for Study 2.
- Covariates and data collection: Study 1 also tested whether answering as oneself versus “Pat Jones” affected judgments, finding no significant effect and excluding that covariate from reported analyses.The surveys collected demographic, mobile-use, and security-relevant measures; Study 1 additionally collected IP addresses and device metadata for modality and operating-system analyses.
- Study 2: Study 2 compared real and fake versions of a Bank of America Zelle transaction notification, a common SMiShing vector [9, 49–51].Participants classified each message as “Real” or “A scam” and reported whether they used Zelle or Bank of America.
- Participants and recruitment: Two online surveys recruited U.S. adult mobile-phone users: Study 1 used a quota-matched Qualtrics panel, and Study 2 used Prolific recruitment.Study 1 analyzed N=1,007 responses after quality checks; Study 2 analyzed N=1,073 responses after data cleaning.
4. Study 1 results
Study 1 participants correctly classified messages only 52.6% of the time, identifying simulated fraudulent texts far more accurately than legitimate ones. Accuracy varied significantly with demographic, usage, job, and security-experience factors.
- Overall message identification: Overall accuracy was 52.6%, with participants correctly identifying fraudulent messages 81.4% of the time but legitimate messages only 23.5%.The study found no significant overall effect of whether participants reported having an account with the message entity.
- Message-specific performance: Among real messages, the simulated Amber Alert was correctly identified by 61.3%, whereas among fake messages, participants performed best on the nude-image-themed message.The supplied passage gives the Amber Alert percentage but does not provide the corresponding percentage for the best-performing fake message.
- Intended responses: Only 38.7% would report a fake message, while 73.3% would delete or ignore it; for real messages, the corresponding rates were 25.4% and 61.3%.Participants often selected these responses for legitimate messages because they incorrectly identified them as fraudulent or likely fraudulent.
- Demographic and usage predictors: Accuracy varied significantly by age, four-year-degree enrollment, household size, mobile-phone use, and job category.Construction and Extraction participants performed worst among the reported job categories.
- Security experience and awareness: Security-breach experiences, proactive security awareness, and recently falling for a scam with a bad outcome significantly explained accuracy differences.The proactive-awareness measure was the Security Behavior Intentions Scale (SeBIS) subscale.
5. Study 2 Results
Study 2 participants correctly classified messages 63.8% of the time, performing better on simulated scams than legitimate texts. Message cues and participant characteristics shaped accuracy, with younger and college participants showing particular difficulty with legitimate messages.
- 5. Study 2 Results: Overall accuracy was 63.8%, with participants identifying simulated SMiSh messages more accurately than simulated legitimate messages (68.72% vs. 58.8%, p<.001).The comparison was significant overall, and no significant effect was found for having a Bank of America account; Zelle use was associated with correctly identifying scam messages (Adj. R2=0.003, β=.092, p<.05).
- 5. Study 2 Results: Among fake-message variants, participants most accurately identified fake URLs in the body text (84.1%) and least accurately identified unsubscribe-reply enticements (56.9%).Fake phone-number variants averaged 65.2% correct; the unsubscribe wording has appeared in real SMiShing examples.
- 5. Study 2 Results: Age 18–24 participants and four-year college students were negatively associated with correctly identifying legitimate messages after controlling for Bank of America and Zelle use.Both groups had mixed success with fake messages; only fake-email variants were consistently identified as scams by at least half, and differences across fake messages were nonsignificant.
- 5. Study 2 Results: Having fallen for a scam message within the previous three months was associated with better identification, particularly of fake messages, while security-awareness training showed no significant effect.The prior-scam association held for all seen messages (Adj. R2=0.008, β=.097, p=.001) and fake messages (Adj. R2=0.014, β=.114, p<.001).
6. Discussion
Across two representative U.S. samples, younger adults and college students were particularly vulnerable to SMiShing, while message headers and existing security training did not reliably improve judgments. The authors therefore recommend targeted interventions, mobile-specific training, and stronger verification aids for mobile messaging.
- Message judgment: Participants did not reliably use header information to identify fake mobile messages and also performed worse than expected when rating simulated real messages.These findings raise concerns for organizations that rely on mobile messaging for urgent or important communications.
- Vulnerability patterns: Younger adults aged 18–24 and people currently pursuing four-year degrees struggled to identify both legitimate and fraudulent messages, making them priority targets for awareness interventions.The authors suggest cognitive fatigue and the volume of unsolicited messages may contribute to students’ difficulty judging what is genuine.
- Vulnerability patterns: No significant gender differences appeared, while healthcare and education workers judged simulated messages better and prior exposure to security breaches was negatively associated with accurate judgment.The authors interpret domain expertise in evaluating information credibility as potentially protective against smishing.
- Training implications: Self-reported training in spotting fraudulent messages and overall security-awareness training showed no significant effects on smishing judgments.In Study 2, however, having fallen for a scam message within the previous three months was protective regardless of whether the incident was a test or caused harm.
- Mitigation and future research: The authors recommend targeted campaigns for young adults and college students, mobile adaptations of simulated phishing exercises, and verified-sender icons or external-sender warnings.They also call for further research on how age, education, job experience, and training shape vulnerability, especially for financial providers impersonated by scammers.
7. Limitations and Future Work
The studies provide useful survey evidence but cannot establish causality and may be affected by survey and simulation biases. Future work should use interviews, controlled message pairs, interface cues, and improved measures to clarify SMiSh susceptibility and legitimate-message identification.
- 7. Limitations and Future Work: The cross-sectional design cannot establish cause-and-effect relationships, and unavailable contact information prevented follow-up with participants.The authors recommend in-depth interviews to understand why participants judge simulated messages as fraudulent or legitimate.
- 7. Limitations and Future Work: Future studies should compare real and fake messages differing in one attribute and examine how account knowledge influences ratings.Relevant knowledge includes prior similar messages from the entity or reasons to believe the message is legitimate.
- 7. Limitations and Future Work: Future work will test mobile and wearable interface cues, including SMS short-code indicators or naming schemes, to make legitimate sources more prominent.Both studies suggest users need help identifying legitimate messages more readily.
- 7. Limitations and Future Work: Survey results may be skewed by self-report and social-desirability biases, while separating legitimacy ratings from confidence could validate binary correctness measures.The proposed replication would clarify participants’ cognitive assessments and the translation of interval confidence into binary correctness.
- 7. Limitations and Future Work: The simulation avoided unsolicited-message harms but gave participants no direct consequences for their responses, which may have biased results and contributed to differences from prior work.Examples of absent consequences include missed opportunities caused by excessive click risk aversion.
8. Conclusions
Across two large-scale panels of U.S. adult mobile phone users, younger people and college students were significantly more vulnerable to SMiSh, while participants generally struggled to identify legitimate texts.
- 8. Conclusions: Younger people and college students were significantly more vulnerable to SMiSh.This finding comes from two large-scale panels of U.S. adult mobile phone users.
- 8. Conclusions: Participants overall struggled to identify legitimate text messages.
- 8. Conclusions: Some participants were misled when fraudulent texts mentioned an entity they believed they had an account with.
Appendix A: Text of Displayed Messages · A.1 Study 1 messages – Diverse entities and scenarios R1 · A.2 Study 2 messages – Bank of America entity, Zelle scenario
Appendix A reproduces the displayed SMS messages used in both studies. Study 1 spans diverse entities and scenarios, while Study 2 holds Bank of America and Zelle constant while varying message details.
- A.1 Study 1 messages – Diverse entities and scenarios R1: Study 1 also included lower-context promotional and social messages, such as a scavenger-hunt prize, bankruptcy consultation, security scan, and TikTok birthday post.These messages varied sender formats, including short codes, phone numbers, named senders, and profile-icon presentation.
- A.1 Study 1 messages – Diverse entities and scenarios R1: Study 1 included messages impersonating universities, retailers, banks, employers, government services, social media, and personal contacts across varied scenarios.Examples included Baton Rouge U. career-fair access, Walmart surveys, Chase fraud alerts, Amazon hiring, IRS audits, Facebook security, and personal birthday content.
- A.1 Study 1 messages – Diverse entities and scenarios R1: The Study 1 set mixed apparent legitimate and suspicious messages, including links, reply requests, urgent warnings, account prompts, and informational alerts.Displayed examples included an AMBER Alert, an IRS audit warning, an Amazon shipping notice, and a Facebook policy violation message.
- A.2 Study 2 messages – Bank of America entity, Zelle scenario: Legitimate Study 2 variants directed recipients to bankofamerica.com/zelle, a phone number, reply INFO, or the number on the back of a BofA card.The messages varied sender presentation and included BofA formatting in some variants.
- A.2 Study 2 messages – Bank of America entity, Zelle scenario: Fake Study 2 variants used phone numbers or S3-hosted links while preserving the same payment details and Zelle scenario.Displayed fake messages came from numbers including (888) 262-5692 and 29450 and linked to boaonline.s3.amazonaws.com.
- A.2 Study 2 messages – Bank of America entity, Zelle scenario: Other fake variants combined BofA-style formatting with requests to reply INFO or 1, unsubscribe, or call 201-416-7037.These variants further changed sender presentation while keeping the same transaction narrative.
Appendix B: Testing a Role-Play Scenario
The role-play manipulation did not affect participants’ ability to identify messages correctly, supporting the use of either perspective in Study 1. This aligns with evidence that embedded role-play can elicit serious, character-consistent responses in simulated phishing research.
- Appendix B: Testing a Role-Play Scenario: Study 1 evenly randomized participants to judge SMS messages as themselves or as the fictional character “Pat Jones.”The comparison tested whether the established embedded role-play method produced results similar to direct self-judgments.
- Appendix B: Testing a Role-Play Scenario: Role-play assignment had no significant effect on CORRECT responses (Adj. R2=-.001, F(1,1004)=.104, p=.747).The researchers therefore did not control for condition assignment in subsequent inferential analyses.
- Appendix B: Testing a Role-Play Scenario: The null role-play effect supports Downs et al.’s argument that participants take simulation roles seriously and respond as instructed characters rather than themselves.Their embedded role-play method asked participants to respond to simulated emails and websites as “Pat Jones,” while many reported applying their own credibility judgments and experience.
Appendix C: Study 1 Open-Ended Responses
Study 1 respondents commonly proposed verifying suspicious texts through official channels, avoiding links, blocking or reporting senders, and checking accounts or websites directly. For emergency-style alerts, respondents often remained cautious while preserving information or watching for the reported vehicle.
- Appendix C: Study 1 Open-Ended Responses: Many respondents said they would ignore, delete, block, or report suspicious messages because links could be fraudulent and unsolicited texts appeared unsafe.Some explicitly cited manipulated links, inconsistent sender addresses, scam indicators, or the danger of clicking unexpected messages.
- Appendix C: Study 1 Open-Ended Responses: For financial or government-themed messages, respondents proposed checking transactions, contacting banks, using separate browsers, or recognizing that the IRS communicates by mail.These responses emphasized independent verification and concern that responding or clicking could expose personal information.
- Appendix C: Study 1 Open-Ended Responses: For an Amber Alert-style message, respondents would remain alert and preserve or review the information but often avoid clicking because such alerts should not contain links.Some said they would monitor news or the vehicle, retain the message, or call 911 if they obtained relevant information.
- Appendix C: Study 1 Open-Ended Responses: Respondents most often recommended contacting the purported organization or checking its official app, website, account, or phone number rather than trusting the text link.Examples included contacting a bank, Amazon, Chase, or the phone provider directly and accessing accounts without using embedded links.
Appendix D: Survey Protocols
The survey recruited U.S. mobile phone users aged 18 or older to interpret mobile text messages. Participants rated messages for legitimacy, confidence, likely responses, and reasons for those responses.
- Eligibility and format: Participants had to be U.S. mobile phone users aged 18 or older and completed an online survey consisting mostly of multiple-choice questions.
- Message evaluation: The protocol asked participants to judge whether each SMS message was legitimate or a scam and report their confidence.
- Response behavior: Participants selected all actions they would take after receiving each message, including clicking links, reporting, replying, or deleting it.
- Response motivations: They also identified reasons for their intended response, such as urgency, curiosity, trust, or seeking to avoid a bad outcome.
- Context checks: The survey additionally asked whether participants held an account with the entity named in the message, including Bank of America or Zelle.