PaperlayerView exact revision on arXiv

Source-linked AI summary

A persistent accuracy ceiling in automated verbal deception detection

Riccardo Loconte, Jonas Festor, Zane Fatjanova, Mariam Bolkvadze, Bennett Kleinberg

arXiv:2610.12118v1cs.CL

TL;DR

Human verbal deception detection, including by professionals, often performs no better than chance, despite the importance of detecting deception in settings such as legal investigations and financial services. This review maps 25 years of research and meta-analyzes automated classifier performance, finding pooled accuracy of 74.4% and convergence with manual approaches around a 70–75% ceiling.

  • Problem

    Human deception detection, including by professionals, often performs no better than chance, while deception has consequential implications in legal, financial, and online settings.

    Source: Main Text · p. 2

  • Method

    The review mapped 25 years of research across 289 reports and 6,136 models and meta-analyzed classifier performance and moderators.

    Source: Main Text · p. 2

  • Results

    74.4% (95% CI: 71.2%-77.4%) was the pooled accuracy, converging with estimates from manual approaches around 70%-75%.

    Source: The performance ceiling around 70-75% · p. 5

  • Takeaways & Limitations

    Convergence across conceptually unrelated approaches suggests an accuracy ceiling in verbal deception detection.

    Source: The performance ceiling around 70-75% · p. 5

    Naturalistic designs better capture real-world deception but often use inferred rather than verified labels, while controlled experiments permit stronger verification but may oversimplify deception and limit generalizability.

    Source: Ground truth and the internal validity… · p. 7

Abstract

from arXiv · show

Automated methods have been proposed to overcome the limitations of human verbal deception detection, but evidence remains fragmented across disciplines. We systematically reviewed 25 years of research (289 reports, 6,136 classification models) and meta-analyzed 3,653 models nested within 97 datasets. Pooled accuracy was 74.4% (95% CI: 71.2%-77.4%) with substantial heterogeneity. Accuracy was driven by methodological quality (ground truth, data source, class balance, evaluation procedure) more than by model complexity: the adoption of embeddings and large language models has not translated into improved predictive performance. Only 12.46% of reports used data with verifiable ground-truth, and only 23.96% of models were evaluated on independent data. The pooled accuracy aligns with meta-analyses of manual approaches, suggesting a ceiling of 70-75%, unlikely to be lifted by current research conventions.

Main Text

Automated verbal deception detection emerged as advances in machine learning and NLP enabled computational analysis of text, amid concerns about limited human accuracy and the stakes of deception. This review maps research practices and meta-analyzes classifier performance and moderators across 25 years.

Main Text

  • Human deception detection, including by professionals, often performs no better than chance, while machine learning and NLP advances have opened new avenues for computational text analysis.The motivation spans consequential settings including legal and forensic investigations, financial fraud, and fake reviews.Source: Main Text · p. 2
  • 25 years of research are mapped across 289 reports and 6,136 classification models, alongside a meta-analysis of classifier performance and moderators.The review describes itself as the largest synthesis of automated verbal deception research to date.Source: Main Text · p. 2
  • The typical pipeline collects labeled written or transcribed statements, converts text into numerical vectors, trains models, and classifies statements as truthful or deceptive.Datasets may come from (quasi-)experiments or naturalistic research designs.Source: Figure 1 · p. 3
  • Text representations include feature counts, n-gram frequencies, embeddings, latent themes, and combinations of representation classes.Examples include dictionary-derived features, bag-of-words, word2vec, and latent Dirichlet allocation.Source: Main Text · p. 4
  • The reviewed predictors range from statistical-learning and ensemble models to neural networks, pretrained transformer language models, and large language models.The glossary describes large language models as transformer-based models using autoregressive next-token prediction, with few-shot or zero-shot classification possible through prompt engineering.Source: Main Text · p. 4

The performance ceiling around 70-75%

The meta-analysis estimates machine-learning accuracy near 74%, while comparisons with manual approaches point to a persistent 70–75% ceiling. Accuracy varies with class imbalance, and methodological quality is more strongly associated with performance than computational choices.

The performance ceiling around 70-75%

  • 77% accuracy is reported at a majority-class proportion of 0.60, while at 0.90 the average prediction is 83.6%.For highly skewed classes, accuracy fell below that of a trivial classifier always predicting the majority class.Source: The performance ceiling around 70-75% · p. 5
  • 70%-75% accuracy across conceptually unrelated approaches suggests a ceiling, with manual content-based analyses at 70% and cognitive-load elicitation at 71% against 56% for standard interviewing.The pooled estimate also exceeds earlier single-cue dictionary-based methods, which achieved 55.2% accuracy.Source: The performance ceiling around 70-75% · p. 5
  • Methodological quality factors, including ground truth and data source, were more strongly associated with performance than computational choices such as representation, architecture, and evaluation procedure.Both sets of factors were significantly associated with performance in the moderator analysis.Source: The performance ceiling around 70-75% · p. 5
  • 74.4% (95% CI: 71.2%-77.4%) is the pooled accuracy for balanced classes.The caterpillar plot shows model-level estimates and the pooled estimate’s 95% CI against balanced-class chance level.Source: The performance ceiling around 70-75% · p. 5, Figure 2 · p. 6

The methodological barriers of the ceiling effect

The review identifies weaknesses in ground-truth quality, data sources, and the scope of studied deception as barriers to interpreting performance. These factors involve trade-offs between realistic settings and reliable labels, and between data scale and disciplinary practice.

Ground truth and the internal validity problem

  • Only 12.46% of reports used clear, verifiable ground truth; relative to it, non-verifiable labels corresponded to +5.7 percentage points, while directly and indirectly inferred labels corresponded to −5.2 and −69.4 points.The indirectly inferred-ground-truth estimate warrants caution because that category was concentrated in online naturalistic data, confounding ground-truth and data-source effects.Source: Ground truth and the internal validity… · pp. 7–8, Table 2 · p. 8
  • Naturalistic designs better reflect real-world deception but often rely on inferred labels; controlled experiments enable stronger verification yet may oversimplify deception and limit generalizability.The tension remains unresolved: realistic study does not necessarily establish whether or where deception occurred.Source: Ground truth and the internal validity… · p. 7

Data source matters

  • Online experiments yielded 18.5 percentage points higher accuracy than offline experiments, potentially reflecting larger training sets; data scale and accessibility also vary with disciplinary conventions.The authors call for greater methodological exchange between psychology and computer science.Source: Data source matters · pp. 8–9

The narrow horizon of deception

  • Studies focused mainly on fabrication (63.67% of reports), English datasets (82.01%), fake reviews (41.52%), and past experiences (15.92%), limiting coverage of subtle, context-dependent, and culturally mediated deception.The authors call for research spanning other deceptive strategies, cultures, and contexts.Source: The narrow horizon of deception · p. 9

Computational choices in automated verbal deception detection research

The review finds that computational choices are associated with performance, but newer representations and architectures do not uniformly improve it. Evaluation on independent data remains limited, while shared tasks and common reporting practices could make estimates more credible and methods more transparent.

Text-embeddings are not necessarily better than simpler approaches

  • After a shift toward embeddings, predicted accuracy increased by only 1.5 percentage points; the pooled representation category spans different pipelines, so representation and architecture effects cannot be separated.Transformer-based models outperformed statistical learning models by 3.5 percentage points, whereas LLMs performed worse; the authors note that fine-tuning and prompting may differ.Source: Text-embeddings are not necessarily… · pp. 9–10
  • Only linguistic and statistical feature models could be directly connected to theories in this sample, offering interpretability that opaque alternatives may not provide in high-stakes settings.The authors argue that theory-grounded models can be advantageous even at the expense of reduced accuracy.Source: Text-embeddings are not necessarily… · p. 10

Large language models do not outperform simpler approaches

  • Transformer-based models outperformed statistical learning models by 3.5 percentage points, whereas LLMs performed significantly worse by 1.2 percentage points; greater model complexity did not uniformly improve detection.The authors suggest that current LLMs used without additional adaptation may not be optimized for deception detection.Source: Large language models do not… · p. 10

“Being accurate about accuracy”5

  • Only 23.96% of models were tested on independent samples; testing on new data reduced predicted accuracy by up to 4.7 percentage points, while hold-out testing raised it by 0.6 percentage points.The authors recommend robust procedures using inner cross-validation and independent samples for testing.Source: “Being accurate about accuracy”5 · p. 12

Shared data, shared tasks, and common reporting practices

  • More than half of reports reanalyzed existing datasets (56.06%), while 34.26% collected new data; formal reporting standards remain absent, and model cards and preregistration are proposed.Shared tasks pool independent analyses under common constraints and datasets, making predictability estimates more credible and less vulnerable to dismissal over overfitting or researcher degrees of freedom.Source: Shared data, shared tasks, and common… · pp. 12–13

Computational models remain the best available alternative

Computational deception detection offers a scalable alternative to manual approaches that can improve on unaided judgment but require intensive coding and often rely on small, in-sample evaluations. Its practical use is constrained by explainability and by unresolved questions about how human and algorithmic judgments should be combined.

Computational models remain the best available alternative

  • Computational models can integrate multiple observations and cues, update as data become available, and generate predictions at scale.Manual approaches can improve on unaided judgment but are difficult to scale and have often been validated on small samples using in-sample evaluation.Source: Computational models remain the best… · p. 13
  • In high-stakes settings, simpler theory-informed models should be preferred when opaque representations offer little or no performance advantage.Automated coding of theoretically meaningful verbal cues may offer a more immediate application by preserving some interpretability while easing manual coding’s resource and scalability limits.Source: Computational models remain the best… · p. 13
  • The practical question is how to allocate responsibility between humans and algorithms, since human involvement may reduce model performance while people may also over-rely on algorithmic advice.The reported effects occur under different conditions; the passage frames allocation of responsibility, rather than replacement, as the relevant question.Source: Computational models remain the best… · p. 13

Moving the needle of verbal deception research

The authors call for a shift from isolated datasets and domains toward shared, large-scale research infrastructure. They propose collaborative analysis and a broad foundation model as ways to support cumulative research across contexts and languages.

Moving the needle of verbal deception research

  • A unified, large-scale multilingual corpus could support mega-analyses and collaborative shared tasks across contexts, domains, and languages.Pooling independent analyses across teams could strengthen findings and the theoretical framework on verbal deception.Source: Moving the needle of verbal deception… · p. 14
  • A foundation model trained across available deception corpora could provide a shared starting point for downstream tasks across languages, deception forms, and topics.The proposed model could learn cues that generalize across these settings rather than rely on the idiosyncrasies of a single dataset.Source: Moving the needle of verbal deception… · p. 14

Conclusion

The review synthesized 25 years of automated verbal deception research and found pooled accuracy near a 70–75% ceiling, with methodological choices more strongly associated with performance than computational choices. Its methods combined a systematic search and screening process, model-level coding, and multilevel meta-analysis.

Conclusion

  • 74.4% pooled accuracy (95% CI: 71.2%-77.4%) across 3,653 models supports a 70%-75% ceiling; methodological choices were more strongly associated with performance than computational choices.The review mapped 289 reports and 6,136 models; only 12.46% of reports clearly specified ground truth, and 23.96% of models were tested on new data.Source: Conclusion · p. 14

Materials and Methods

  • The search covered seven databases and included English-language empirical studies of automated verbal-cue extraction and prediction, while excluding non-verbal detection and several adjacent research domains.The initial search identified 23,773 records; 289 reports were ultimately included after screening.Source: Materials and Methods · p. 15
  • Screening used active learning with human relevance labels, followed by an updated search, full-text eligibility assessment, and a quality check of excluded reports.The final review included 289 reports; the updated search was screened by a single reviewer, while initial title-and-abstract screening used two screeners.Source: Materials and Methods · pp. 16–17
  • Reviewers coded each model separately, piloted the coding scheme on 42 reports, and reached percentage agreement of 0.78-0.79.Variables were coded inductively from reports and mapped into broader categories; data source and ground truth used predefined codes.Source: Materials and Methods · pp. 17–18
  • The multilevel meta-analysis pooled reported accuracy and modeled dataset- and report-level dependencies, with class imbalance centered at 0.50 for balanced-class estimates.Models without accuracy, with non-standard contrasts, or lacking key information were excluded from the quantitative synthesis.Source: Materials and Methods · p. 18
  • Publication bias was not statistically assessed because model performance usually lacked p-values, though the review included preprints and coded all reported models to mitigate bias.Source: Materials and Methods · p. 18

Supplementary Information

The supplementary analyses map the evidence base and show that reported accuracy varies with dataset composition and methodological choices, with substantial residual heterogeneity.

Supplementary Results

  • The search yielded 289 reports and 6,136 models; experimental designs covered the widest range of deception forms, while quasi-experimental work focused almost exclusively on fabrication.The included reports spanned 1982–2025, with publication volume increasing over the last 15 years.Source: Supplementary Results · p. 38
  • Online naturalistic datasets were large (median 8,897.50; range 103–2,542,553), while offline datasets were smaller; Table S5 tabulates model-level methodological choices.Table S5 covers textual representation, model category, evaluation procedure, and performance metric.Source: Supplementary Results · p. 38
  • 74.43% pooled accuracy (95% CI [71.17%-77.43%]) was predicted after class-imbalance adjustment; at majority-class proportion 0.90, accuracy was 83.6%, below the 90% majority-class baseline.Accuracy rose from 77.0% at a majority-class proportion of 0.60, but remained below the trivial majority-class classifier at both reported proportions.Source: Supplementary Results · pp. 38–39
  • Dataset differences accounted for 87.2% of residual variability; methodological fixed effects explained 30.1% of total variance, and the full model accounted for 91.2%.Report-level residual variability accounted for a further 12.6%.Source: Supplementary Results · p. 39
  • Accuracy tracked data source and ground-truth quality; hybrid representations and transformers had positive effects, LLMs trailed, and independent-data testing reported lower accuracy than within-CV.Mixed and online sources were associated with higher estimates; inferred ground truth was associated with lower estimates than clear, verifiable ground truth.Source: Supplementary Results · p. 39
  • Scope excludes fake reviews and misinformation; sparse moderator combinations and inferred coding in short reports may limit attribution and warrant caution.The authors note that substandard reporting may have introduced coding errors, particularly for rarely specified variables such as deception type and ground truth.Source: Supplementary Results · pp. 39–40

Data S1. (separate file)

Data S2. (separate file)

Loading 2610.12118v1…