Source-linked AI summary
Adversarial Attacks Reveal Divergent Deception Competence in Humans and Generative Language Models
Lucca Pfründer, Riccardo Loconte, Bruno Verschuere, Bennett Kleinberg
TL;DR
This study asks how humans and generative language models construct paraphrasing attacks on a deception classifier, using adversarial attacks to examine second-order beliefs about deception. Humans more effectively made truths read as lies, while the language model more effectively made lies read as truths.
Problem
Although machine-learning models typically outperform people at deception detection, the average person detects deception only barely above chance using intuition.
Source: Introduction · p. 2Method
The study used adversarial attacks as an instrument to compare human and machine approaches and measure their understanding of deception.
Source: General Discussion · p. 13Results
Humans shifted deception cues to make truths read as lies, whereas the language model made more substantial changes that almost exclusively made language signal truth.
Source: General Discussion · pp. 13–14Takeaways & Limitations
The contrasting attack strategies offer a working model of deception that people apply when judging deception but cannot articulate directly.
Source: Introduction · p. 3, General Discussion · p. 14The language model's strategy reliably misled the classifier to read truths as deceptive only under the zero-shot prompting condition.
Source: General Discussion · p. 12
Abstract
from arXiv · showhide
Machine learning models typically outperform humans in deception detection. A threat to machine learning models is their vulnerability to adversarial attacks: subtle yet targeted perturbations of input data that trick a model into misclassification. Yet, this also provides an opportunity: understanding how humans versus machines construct attacks offers new insights into second-order beliefs about deception. In a preregistered experiment (n = 501), we compare the effectiveness, efficiency, and nature of a paraphrasing attack by humans and generative language models on an automated deception classifier. We find that humans were more effective attackers when rewriting truthful statements to be deceptive, while the generative language model was more effective and efficient at making lies sound like truths. Computational text analysis suggests that this is because humans make targeted use of deception cues, while generative language models, instead, apply generic changes that signal truth. The failure to mimic deception is two-sided: while it exposes a blind spot in the machines increasingly used to detect lies, it also means that language models might facilitate successful lying.
SM1: Power analysis
The power analysis used Monte Carlo simulations based on pilot estimates to choose a sample size for testing the attack-modality-by-veracity interaction. The final sample was smaller than the target after exclusions but remained within the reported powered range under the fixed cluster structure.
SM1: Power analysis
- Monte Carlo simulations estimated power as the proportion of 1,000 simulated datasets with p < .05 for the modality-by-veracity interaction.The simulations used pilot-model estimates and sampled attack modality and original veracity from their pilot joint distribution.Source: SM1: Power analysis · p. 21
- The fixed-structure simulations reused the pilot topics, statements, and their nesting, varying only the number of attacks per original statement.The relaxed-structure scenario is also reported, but its structure is not described in these passages.Source: SM1: Power analysis · p. 21, Table S2 · p. 21
- The analysis checked Type I error with a zero interaction effect and reran both scenarios at reduced pilot-effect sizes to assess effect-size inflation.The zero-effect p-value distribution was approximately uniform, with Type I error near .05.Source: SM1: Power analysis · pp. 21–22
- At 75% and 50% of the pilot interaction effect, power remained at 0.8 under the fixed structure at n = 608 and n = 1216, respectively.Under the relaxed structure, power was 0.79 at n = 524 and 0.87 at n = 1310; neither structure reached acceptable power at 25%.Source: SM1: Power analysis · p. 22
- The target was 650 attack sequences, and 501 remained after exclusions; this sample fell between the fixed-structure estimates of 0.93 power at n = 456 and 0.97 at n = 608.The target was 325 attacks per modality, and 330 were collected per modality before exclusions.Source: SM1: Power analysis · p. 22
SM2: Hippocorpus dataset
The experiment used truthful and deceptive statements from Hippocorpus, capped statement length, and organized the statement pool using topic labels.
SM2: Hippocorpus dataset
- Participants attacked truthful and deceptive statements drawn from the Hippocorpus dataset.In the original dataset, participants recalled memorable events and wrote summaries; new participants used those summaries to fabricate connected stories.Source: SM2: Hippocorpus dataset · p. 22
- Statements were truncated at 500 characters, with mid-sentence cuts extended to the sentence end.Source: SM2: Hippocorpus dataset · p. 22
- BERTopic topic discovery and manual inspection supplied labels for the final statement pool through regular expressions.The topic model used default settings.Source: SM2: Hippocorpus dataset · p. 22
SM3: Candidate target classifiers
The authors fine-tuned four candidate classifiers and selected ModernBERT, which performed on par with the best Hippocorpus candidates and generalized no worse than other candidates beyond that dataset.
SM3: Candidate target classifiers
- The authors fine-tuned DistilBERT, BERT, SentenceBERT, and ModernBERT using five-fold cross-validation with an 80/20 split, then trained a final model on the full training set.The final model used the selected hyperparameters.Source: SM3: Candidate target classifiers · p. 22
- They compared candidate performance on an unseen Hippocorpus test set and deception datasets from different contexts.The comparison included deceptive opinions and real-life trial statements.Source: SM3: Candidate target classifiers · p. 22
- ModernBERT performed on par with the best Hippocorpus test-set candidates (AUC = 0.86, as SentenceBERT; accuracy = 0.78).Source: Table S3 · p. 22, SM3: Candidate target classifiers · p. 24
- Outside Hippocorpus, ModernBERT generalized no worse than the other candidates on four deception datasets.Source: Table S4 · p. 23, Table S5 · p. 23, Table S6 · p. 23
SM4: Demographics of human participants
Table S8 reports demographics for the analyzed human participants; demographic data were available for 221 of the 223 participants remaining after exclusions.
SM4: Demographics of human participants
- Table S8 presents demographics of the human participants included in the analyses.Source: SM4: Demographics of human participants · p. 24, Table S8 · p. 24
- Demographics were available for 221 of the 223 human participants remaining after exclusions.Source: SM4: Demographics of human participants · p. 24
SM5: Interface for human attack sequences
The interface guided human attackers through an ordered study sequence and showed feedback from earlier attempts. The task was to change the classifier’s initial prediction by paraphrasing a statement.
SM5: Interface for human attack sequences
- After earlier attempts, participants could see their rewrites, the model’s labels, and its confidence in those labels.Source: SM5: Interface for human attack… · p. 25
- Human attackers moved through welcome, consent, instructions, attention checks, a task reminder, the main task, feedback, and debriefing.The main task interface is shown in Figure S1.Source: SM5: Interface for human attack… · pp. 25–26, Figure S1 · p. 25
- Participants were told to modify statements so the AI prediction would change and had 10 attempts.The feedback form also asked about motivation, task difficulty, and strategies.Source: SM5: Interface for human attack… · p. 26
SM6: Candidate generative language model attackers
This section describes candidate generative language model attackers and their iterative paraphrasing task to reverse an AI classifier’s predictions while preserving statement meaning. It also covers the prompts, meaning-preservation checks, and automated quality-control procedures used in the task.
SM6: Candidate generative language model attackers
- Llama 3.3 was the most effective and efficient candidate attacker overall, so it was used in the final study.The comparison covered effectiveness and efficiency; Tables S9 and S10 report the relevant metrics.Source: SM6: Candidate generative language… · pp. 26, 28, Table S9 · p. 26, Table S10 · p. 27
- The candidate LLMs iteratively paraphrased truthful and deceptive statements to change the classifier’s veracity judgment while preserving meaning, grammar, and naturalness, for up to 10 attempts per sequence.Prior attempts and classifier feedback were included in adversarial-rewriting prompts.Source: SM6: Candidate generative language… · pp. 27–28
- LLM outputs were generated programmatically and entered the analysis pipeline without human review, editing, approval, or curation.Two automated exclusion filters were applied to attack sequences.Source: SM6: Candidate generative language… · p. 28
- Attackers had up to 10 attempts to lower the classifier’s confidence enough to reverse its original prediction, with live feedback from the model.A successful reversal ended the sequence immediately.Source: SM6: Candidate generative language… · p. 30
- Rewrites had to preserve recognizable core facts or events, remain grammatical and natural, and avoid unrealistic or absurd changes.The instructions emphasized changing the classifier’s prediction without changing the original statement’s meaning.Source: SM6: Candidate generative language… · pp. 30–31
- In each attack sequence, the previous modification and the model’s label and confidence feedback were appended to the next prompt.The prompt also asked attackers to describe their approach in 2–3 sentences.Source: SM6: Candidate generative language… · pp. 32–33
- The adversarial paraphrasing prompt instructed the model to preserve meaning, grammar, and naturalness while outputting only the requested modification.It framed the model as a study participant interacting with an AI trained to classify human statements as truthful or deceptive.Source: SM6: Candidate generative language… · pp. 34–35
- LLMs paraphrased statements from the public Hippocorpus dataset, and Prolific raters assessed meaning preservation without refining prompts or models.The ratings were used only to validate preservation of the original statement’s meaning.Source: SM6: Candidate generative language… · p. 36
- Post-processing was fully automated in R with no manual edits, and sequences were excluded for non-English output or rewrites more than 20 words shorter than the original.No new personal data were generated for the LLM analyses, and the text contained no identifying information.Source: SM6: Candidate generative language… · p. 36
- Eight adversarial-paraphrasing sequences were excluded for non-English output (all Qwen 3), and 22 plain-rewriting sequences were too short.Rewrites truncated at max_tokens were retained.Source: SM6: Candidate generative language… · pp. 36–37
SM8: Validity checks
Raters judged whether rewrites remained realistic given summaries of the original statements; the results supported removing a subset of human rewrites before subsequent analyses.
SM8: Validity checks
- Raters provided 3,490 realism judgments, analyzed with a mixed-effects model using condition as a fixed effect and random intercepts for rewrites and raters.The analysis covered 1,141 summary-statement items.Source: SM8: Validity checks · p. 37
- Human rewrites scored 3.15 (SD = 0.64), below originals (3.56, SD = 0.63) and model rewrites (3.66, SD = 0.63), which did not differ significantly from originals.Human rewrites also scored significantly lower than model rewrites.Source: SM8: Validity checks · p. 37
- After attention-check exclusions, researchers removed 70 human rewrites and retained 501 attack sequences: 223 human and 278 model.Human rewrites were iteratively removed from the lowest agreement score until the original-versus-human comparison had BF10 < 10.Source: SM8: Validity checks · p. 37
SM9: Linguistic features
This section operationalizes linguistic perturbations with deception-theory and part-of-speech features, and reports supplementary classifier analyses of attack outcomes.
SM9: Linguistic features
- DeCLaRatiVE combines 26 features grounded in four deception theories, with four additional part-of-speech features; perturbations were scored as modification-minus-original changes.Source: SM9: Linguistic features · pp. 37, 40
- The feature framework covers distancing, cognitive load, reality monitoring, and verifiability, including expected differences in self-reference, language complexity, perceptual details, and verifiable details.Source: SM9: Linguistic features · pp. 37–38
- Features were computed with textstat, LIWC-22, a concreteness dictionary, named-entity recognition, and spaCy part-of-speech tags.Source: SM9: Linguistic features · p. 38
SM10: Additional analyses of effectiveness (H1 & H2)
- Figure S2 presents bootstrap distributions of classifier AUC for unattacked statements and after overall, human, and model attacks; the dashed line marks chance at AUC = 0.50.Density width represents uncertainty in the AUC estimate.Source: SM9: Linguistic features · p. 40
- ASR was 52.2% for humans versus 4.2% for models rewriting truthful statements as deceptive, but 52.7% for humans versus 90.6% for models making deceptive statements sound truthful.The ASR interaction was significant in the robustness analysis.Source: SM9: Linguistic features · p. 41
SM11: Additional analyses of efficiency
Additional analyses found no difference between human and model attackers in attempts needed for the first classifier flip. Attempts to flip truthful statements took longer than attempts to flip deceptive statements, while successful human attacks showed declining classifier confidence across attempts.
SM11: Additional analyses of efficiency
- Humans (M = 3.34, SD = 2.86) and generative language models (M = 2.02, SD = 1.72) did not differ significantly in attempts needed for the first flip (c2(1) = 0.91, p = .340).Source: SM11: Additional analyses of efficiency · p. 41
- Making a truthful statement sound deceptive took more attempts to flip than making a deceptive statement sound truthful.The difference was significant (χ2(1) = 9.57, p = .002), with no significant interaction by attack modality.Source: SM11: Additional analyses of efficiency · p. 41
- Among successful human attacks, classifier confidence decreased with each attempt.Figure S3 reports mean confidence change per attempt by original statement veracity; change is relative to the original statement, with more negative values indicating a larger decrease.Source: SM11: Additional analyses of efficiency · p. 41, Figure S3 · p. 41
SM12: Additional analyses of attack strategies
Attack strategies were associated with success differently for humans and language models. Models reported only authenticity and emotion strategies, whereas humans reported all four identified strategies.
SM12: Additional analyses of attack strategies
- The four identified strategies were changing perceived authenticity, changing details, changing emotion, and general rephrasing; models reported only authenticity and emotion, while humans reported all four.Across responses, the strategy proportions were 40.12%, 9.78%, 33.73%, and 16.37%, respectively.Source: SM12: Additional analyses of attack… · p. 42
- The planned Poisson mixed-effects analysis was replaced by chi-square tests because models used only two strategies and random-effect variances were estimated at zero.The mixed-effects model’s strategy interactions were inestimable with empty cells.Source: SM12: Additional analyses of attack… · p. 42
- When successful, models more often reported modifying authenticity; emotion-based modifications were more common among unsuccessful model attacks.Authenticity modification was reported in 78.0% of successful versus 43.0% of unsuccessful attacks; emotion modification was reported in 22.0% versus 57.0%, respectively.Source: SM12: Additional analyses of attack… · p. 42
- Humans more often reported changing details in successful attacks, while rephrasing was more common in unsuccessful attacks.Changing details was reported for 30.8% of successful versus 12.3% of unsuccessful human attacks; rephrasing was reported for 29.1% versus 45.3%, respectively.Source: SM12: Additional analyses of attack… · p. 42