Source-linked AI summary
Using Codebooks to Detect Cybercrime Topics in Text Narratives
Shufan Chai, Liangliang Sun, Jessica Staddon
TL;DR
Local governments need cybercrime detection support, but specialized models can exceed their resources. The paper prompts pretrained LLMs with qualitative-research codebooks and finds high precision and recall for impostor scams and identity theft across five Gemini and GPT models. The results suggest a resource-conscious, generalist-friendly approach for organizations detecting cybercrime in consumer narratives.
Problem
Resource-constrained local organizations need cybercrime detection, but existing approaches often rely on customized models they cannot train or maintain.
Method
The paper uses researcher-authored codebooks from qualitative cybercrime research to prompt pretrained LLMs to classify impostor scams and identity theft in consumer narratives.
Results
Across five Gemini and GPT models, codebook prompts improve average precision by at least .2, and all but optimized Gemini 2.5 Flash exceed .8 precision and recall.
Takeaways & Limitations
The method performs well with varied pretrained models, including potentially cheaper ones, without requiring domain expertise or substantial prompt-engineering expertise.
Takeaways & Limitations
The demonstrated generalizability is fairly narrow because the method depends on cybercrime attributes or events that lend themselves to qualitative analysis and codebook development.
Abstract
from arXiv · showhide
In the United States, management of cybercrime-related consumer complaints increasingly falls on state and city governments given de-staffing of federal agencies. AI, and in particular, large language models (LLMs), shows promise for detecting cybercrime in text complaints, but often via specialized models that local governments are not resourced to develop and maintain. We present an LLM prompting method that uses codebooks from qualitative cybercrime research to detect cybercrime topics in consumer narratives. For two cybercrime topics, impostor scams and identity theft, we demonstrate the method achieves high precision and recall across multiple runs of 5 models in the Gemini and GPT model families. This strategy suggests a path for resource-constrained organizations, like many local governments, to leverage frontier models to support community safety.
1 Introduction
The paper asks whether pretrained LLMs can use researcher-authored codebooks to identify cybercrime topics in consumer narratives. Across impostor scams and identity theft, the method improves precision and achieves high precision and recall without specialized model development.
- Motivation: The paper addresses local governments’ need for cybercrime detection tools that do not require the resources to develop and maintain customized models.Federal resource reductions are associated with communities turning to state and city governments for help, while many specialized approaches require unavailable training or data resources.
- Background: Codebooks are human-designed documents containing definitions, examples, and counter-examples from qualitative cybercrime research.The paper repurposes these research artifacts as guidance for LLM-based detection.
- Research Question: The research question is whether generalists can use pretrained LLMs and researcher-authored codebooks to accurately identify cybercrime in text narratives.The study focuses on two prominent security problems: impostor scams and identity theft.
- Contributions: The method improves precision over baseline and achieves high precision and recall for impostor scams and identity theft across pretrained, nonoptimized models.Across five Gemini and GPT models, all but optimized Gemini 2.5 Flash exceed .8 precision and recall when prompted with codebooks.
- Contributions: The contribution is a generalist prompting method for detecting cybercrime incidents in text narratives using pretrained LLMs and codebooks.The approach is motivated by prior use of human-authored policies to guide pretrained LLMs in content moderation.
2 Data and Methodology
The study evaluates LLM predictors for detecting impostor scams and identity theft, comparing prompts that include qualitative-research codebooks with baseline prompts. It constructs labeled CFPB complaint datasets and operationalizes codebook-based prompting through a simple reusable template.
- Study design: The study compares codebook prompts with baseline prompts across five models and two topics: impostor scams and identity theft.A predictor is a model–prompt pair that outputs whether a narrative discusses a topic.
- Data construction: The authors built labeled datasets from publicly available CFPB consumer complaint narratives, using issue labels and sampling strategies to identify relevant positive and negative examples.The data distinguish impostor scams from non-scam fraud and address the relatively small prevalence of target topics in the complaint database.
- Impostor scam data: For impostor scams, three authors refined an initial codebook through disagreement resolution, examples, and independent reliability checks before coding disjoint data subsets.Pairwise Cohen’s kappas on an additional shared set were 0.80, 1.00, and 0.80; the final dataset contains 1,357 narratives, including 187 impostor scams.
- Identity theft data: For identity theft, the authors used a statutory definition and produced a dataset of 981 narratives, including 427 identity-theft narratives, after independent coding and duplicate removal.The authors achieved perfect agreement on an additional shared set of 10 narratives.
- Prompt design: The prompting method inserts the topic-specific codebook into a common template that assigns a fraud-detection role and requires a yes-or-no classification.Without an inserted codebook, the same template is called the baseline prompt.
3 Case Studies
Across impostor scam and identity theft case studies, codebook prompts improved LLM precision over baselines and achieved high precision and recall across five Gemini and GPT models. The discussion identifies codebook quality, predictor intent, generalizability, misuse, and governance as important boundaries.
- Overall Results: Codebook prompts improved precision for every model and both topics, with all models except Gemini 2.5 Flash exceeding .8 precision under codebook prompting.The improvement was particularly large for identity theft because baseline performance was weak.
- Impostor Scams: .88 precision and .88 recall were achieved on average for impostor scam codebook runs, with precision improving .2 over baseline.The average covers 10 codebook experiments across five models.
- Identity Theft: .83 precision and .94 recall were achieved on average for identity theft codebook runs, with precision improving .23 over baseline.The average covers 10 experiments across all models; codebook precision ranged from .75 to .88.
- Discussion: Identity theft had poor baseline alignment because public guidance permits broader interpretations than the codebook’s federal-law-based scope.The codebook includes new accounts opened using stolen identity information, whereas some agency descriptions include unauthorized existing-credit-card purchases.
- Limitations and Risks: The method assumes rigorously developed codebooks, and conservative or mismatched codebook goals may make predictors inappropriate for some use cases.The authors also identify narrow generalizability, public-input gaming, and the need for governance as limitations or risks.
4 Conclusion and Open Problems
The paper presents a generalist-friendly codebook method that accurately predicts impostor scams and identity theft across pretrained models, while identifying limits requiring further study.
- The method accurately predicts impostor scams and identity theft even when models’ baseline performance without a codebook is poor.It performs well across a variety of pretrained models, including potentially cheaper models, without requiring domain expertise.
- Open Problems: Broader experimentation is needed to determine how widely the method applies across cybercrime topics and attributes.The authors tested only two topics and used a single prompt template; a justification-based template might improve performance.
- Open Problems: Experiments with codebooks of varying complexity could clarify the limits of what state-of-the-art models can leverage and help assess abuse risk.Evaluation rubrics used by frontier-model and annotation companies may also provide useful prompt foundations.
- Open Problems: Safe integration of an LLM-based detector into a governance framework remains unaddressed.The authors note that detectors should be combined with other techniques, such as community-based moderation, which may respond faster to changes in bad-actor strategy.
A Open Science
The paper supports open science by publishing the datasets and locating the human-authored codebooks and evaluation calculations in specified resources.
- The impostor scam and identity theft datasets are anonymously published, with access provided through a linked spreadsheet.
- The human-authored codebooks are provided in the paper’s sections on impostor scams and identity theft.
- Precision and recall were calculated using established formulas implemented in both spreadsheets and notebooks.
B Ethical Considerations
The study uses publicly released CFPB complaint narratives under an opt-in publication process, with scrubbing and review intended to remove direct personal identifiers.
- CFPB narratives are published only with consumer opt-in consent, and consumers may withdraw that consent.Before release, the CFPB applies automated checks and human review to scrub personal information that could directly identify individuals.
C Use of Generative AI
The paper was human-authored, while generative AI was used selectively to assist with code, document production, and draft critique.
- The authors used generative AI primarily as an assistant for coding, figure generation, LaTeX tables, BibTeX references, and critique of earlier drafts.
D Experiments and Codebooks
The experiments report precision and recall for baseline and codebook conditions across impostor scams, identity theft, and all LLMs. The paper provides complete codebooks for both topics, developed iteratively by the authors.
- Precision and recall are reported for baseline and codebook experiments covering impostor scams, identity theft, and all LLMs.
- The complete impostor scam codebook was iteratively developed by the authors using the process described in Section 2.1.1.
- The complete identity theft codebook was iteratively developed by the authors using the process described in Section 2.1.2.