PaperlayerView exact revision on arXiv

Source-linked AI summary

Responses to Critiques on Machine Learning of Criminality Perceptions (Addendum of arXiv:1611.04135)

Xiaolin Wu, Xi Zhang

arXiv:1611.04135v3cs.CV

TL;DR

Research on face-based social perception has focused more on its prevalence and consequences than on the accuracy of character inference from still images. The authors use supervised machine learning to assess criminality from face images and report reliable inference alongside lower facial variation among non-criminals after demographic controls.

  • Problem

    Relatively little research has assessed the accuracy of character inference based solely on still face images, despite extensive study of appearance-induced inference and its consequences.

    Source: §1 · pp. 4–5

  • Method

    The authors train four supervised classifiers on facial images of 1,856 real people, controlled for race, gender, age, and facial expressions, to distinguish convicted criminals from non-criminals.

    Source: Abstract · p. 4, §1 · p. 5

  • Results

    The authors report reliable criminality inference and significantly less facial variation among law-abiding people than criminals after controlling for race, gender, and age.

    Source: §6 · p. 13

  • Takeaways & Limitations

    The authors say uncontrolled socioeconomic status may affect performance and that this possibility is why they saw potential significance for social sciences.

    Source: §5 · p. 3

    Socioeconomic status was uncontrolled because the authors could not access the relevant metadata, and they speculate classifier performance would drop if it were controlled.

    Source: §5 · p. 3

Abstract

from arXiv · show

In November 2016 we submitted to arXiv our paper "Automated Inference on Criminality Using Face Images". It generated a great deal of discussions in the Internet and some media outlets. Our work is only intended for pure academic discussions; how it has become a media consumption is a total surprise to us. Although in agreement with our critics on the need and importance of policing AI research for the general good of the society, we are deeply baffled by the ways some of them mispresented our work, in particular the motive and objective of our research.

1. Name calling

The authors frame their work as testing whether machine learning can capture humanlike social perceptions from faces, extending face analysis beyond biometric attributes. They say criminality was chosen as an expedient test case, while acknowledging terminology and data-labeling concerns.

1. Name calling

  • The inquiry tests whether machines can reproduce human first impressions of strangers, extending face recognition from biometric to sociopsychological attributes.The authors say criminality was chosen as an easy test case.Source: §1 · p. 1
  • Criminality was chosen as a test because the authors expected criminals and non-criminals to be comparatively easy to distinguish if facial and social attributes correlate.They reasoned that low classification performance would negate face-induced social inference.Source: §1 · p. 1
  • They call treating court conviction as machine-learning ground truth a serious oversight and acknowledge possible input-data bias.Source: §1 · p. 1
  • The authors expected machine learning and computer vision to refute physiognomy, but say the outcomes turned out otherwise.Source: §1 · p. 1
  • The authors say they neither advocated law-enforcement use nor advanced from correlation to causality.Source: §1 · p. 1
  • The authors regret using “physiognomy,” which they describe as carrying a dirty connotation in English-language academia.Source: §1 · p. 2

2. Base Rate Fallacy

The authors argue that media discussion can exaggerate classifier performance by overlooking the low background crime rate. Their Bayesian example yields a much lower probability of criminality after a positive result, and they reiterate opposition to practical use.

2. Base Rate Fallacy

  • The authors say media coverage focuses on the CNN’s 89% true positive rate while neglecting China’s 0.36% crime rate.Source: §2 · p. 2
  • 4.39% is the estimated chance of committing a crime after a positive classifier result, using the authors’ stated rates.They report P(+|C) = 0.89, P(C) = 0.0036, and P(+|N) = 0.07.Source: §2 · p. 2
  • They describe base-rate neglect as a way media can sensationalize or exaggerate unfamiliar technological advances.Source: §2 · p. 2
  • The authors condition their argument about objective automated inference on training labels being free of human bias.They say well-trained experts can strive for objective legal decisions independent of facial appearance.Source: §3 · p. 2

4. Risk of Overfitting

The authors acknowledge a small sample and overfitting risk, then describe random-label tests and controlled image characteristics as checks on that risk. They say access constraints prevented them from obtaining more convicted-male ID images.

4. Risk of Overfitting

  • The authors acknowledge a relatively small sample and overfitting risk, and say they could not obtain more convicted Chinese male ID images.Source: §4 · p. 2
  • All classifiers failed to distinguish randomly labeled faces, which the authors say suggests reported performance was not due to overfitting.They also refer to other, more challenging tests.Source: §4 · pp. 2–3
  • The 2,000-image sample was restricted to one race, gender, and head pose, with aligned landmarks, which the authors say reduced variability and overfitting risk.Source: §4 · p. 3

5. White Collar

The authors clarify that only face regions, not full ID portraits, were used for training and testing. They could not control for socioeconomic status because the relevant metadata were inaccessible, and speculate that controlling for it could lower classifier performance.

5. White Collar

  • The authors segmented face regions from ID images and used those face-only images for training and testing.Complete ID portraits appeared in the paper only for illustration.Source: §5 · p. 3
  • Socioeconomic status was not controlled because confidentiality issues prevented access to the relevant metadata.Source: §5 · p. 3
  • The authors speculate that controlling for socioeconomic status would lower classifier performance and could raise a social-injustice concern.They say this is why they thought the results might matter to social sciences.Source: §5 · p. 3

6. Smiley

The authors revisit whether the classifier’s results could reflect smiling rather than criminality, while noting limits in expression control and describing steps against image-source artifacts.

6. Smiley

  • The experiments controlled overt expressions but not faint micro-expressions, which the authors plan to control more tightly when reliable algorithms become available.Examples of uncontrolled micro-expressions include relaxed versus strained expressions.Source: §6 · p. 3
  • Chinese students and colleagues did not detect the smile cue that some Western viewers perceived, describing the bottom-row faces as more relaxed instead.The authors suggest cultural differences may explain the differing perceptions.Source: §6 · p. 3
  • The authors also report taking steps to prevent classifiers, particularly CNNs, from exploiting superficial differences such as compression noise and camera variation.They describe these measures in Section 3.3.Source: §6 · p. 3

Abstract

The paper applies supervised machine learning to infer criminality from still face images, reporting successful classification and distinct criminal and non-criminal face manifolds.

Abstract

  • Four supervised classifiers trained on 1,856 real persons’ face images performed consistently well at distinguishing convicted criminals from non-criminals.The data were controlled for race, gender, age, and facial expressions, and nearly half the subjects were convicted criminals.Source: Abstract · p. 4
  • Criminal and non-criminal face images populated two quite distinctive manifolds, with greater variation among criminal faces.The authors describe the manifolds as concentric, with the non-criminal manifold lying in the kernel with a smaller span.Source: Abstract · p. 4
  • The authors report that non-criminal faces have greater resemblance to one another than criminal faces.They characterize this pattern as a law of “normality” for non-criminal faces.Source: Abstract · p. 4

1. Introduction

The introduction motivates automated study of social perception from faces and asks whether criminality can be inferred from still images. It presents supervised learning as a way to assess that question without human evaluators and situates the work against earlier studies.

1. Introduction

  • Research on facial recognition is extensive, but relatively little work analyzes and quantifies social perception and attributes from faces.The introduction notes the topic’s relevance across academic disciplines.Source: §1 · p. 4
  • The authors use supervised learning with logistic regression, KNN, SVM, and CNN on images of 1,856 real people, half of whom were convicted criminals.They describe the study as the first of its kind to their knowledge.Source: §1 · p. 5
  • The paper asks whether criminality can be inferred accurately from still face images, a question with relatively little prior study.The authors identify the difficulty of removing human prejudice and preconditioning as one reason for the limited research.Source: §1 · p. 5
  • Automating assessment removes human evaluators from the process, while machine learning may detect facial nuances below the threshold of untrained observers.The authors frame the latter as a theoretical possibility.Source: §1 · p. 5
  • The authors distinguish their work from earlier studies using synthetic faces and from Valla et al.’s real-face study, which used human classifiers.They describe their own data as real-person ID photos controlled for race, gender, age, and facial expression.Source: §1 · p. 5

2. Data preparation

The study uses 1,856 Chinese male ID photos, split into criminal and non-criminal sets, and applies image normalization and processing to prepare them for analysis.

2. Data preparation

  • The dataset contains 1,856 Chinese men aged 18–55, selected without facial hair, scars, or other markings, and divided into criminal and non-criminal subsets.The non-criminal subset contains 1,126 people from varied professions and social statuses.Source: §2 · p. 6
  • The criminal subset contains 730 people, including 330 wanted suspects and 235 convicted of violent crimes; their photos are normal ID portraits, not mugshots.The remaining criminal cases involved non-violent crimes.Source: §2 · p. 6
  • Images were cropped to the face and upper neck, background-removed, aligned to 80 × 80 pixels, converted to luminance, and intensity-normalized.The intensity normalization matched each image histogram to the dataset’s average histogram.Source: §2 · p. 6
  • JPEG soft decoding was applied to remove small compression artifacts and destroy device-dependent signal signatures.The photos had JPEG quality factors of 90 or higher.Source: §2 · p. 6

3. Validity of Face Classifiers on Criminality

The section tests whether still face images can distinguish criminals from non-criminals using four classifiers and multiple feature representations. CNN accuracy reached 89.51%, while validation tests examined random labels, image noise, and changes in lighting and orientation.

3.1. Methods

  • The experiments used logistic regression, KNN, SVM, and CNN; the first three were tested with four feature-vector types, alongside CNN without explicit features.The feature types included facial landmarks, modular PCA, LBP histograms, and their concatenation; the CNN retrained AlexNet parameters while retaining its architecture.Source: §3.1 · p. 6

3.2. Results

  • 89.51% accuracy was achieved by CNN, while the other three classifiers were only a few percentage points behind across the thirteen cases.Figure 2 plots the accuracies for all four classifiers and cases.Source: Figure 2 · p. 7, §3.2 · p. 7
  • The authors report that ROC curves and AUC results establish prediction power for the four tested binary face classifiers.The supplied passages do not give the AUC values.Source: §3.2 · p. 7, Figure 5 · p. 7, Table 1 · p. 7

3.3. Validation

  • Random-label tests yielded 48% average accuracy, about 51% false negatives, and about 50% false positives; the authors say these tests rule out data overfitting.On three additional randomly labeled sample sets, no classifier achieved a true positive rate above 53%.Source: §3.3 · p. 8
  • With visible noise at σ = 0.03, accuracy remained above 73% for SVM and 76% for CNN, although both classifiers' accuracy decreased as noise increased.For landmark-based KNN, LR, and SVM, the authors report no material performance changes across noise levels.Source: §3.3 · pp. 8–9, Table 2 · p. 8, Table 3 · p. 8
  • Classification results on photographs taken under varied lighting and face orientations were consistent with probability higher than 83%.The test used four photographs each from 10 male Chinese students.Source: §3.3 · p. 9

4. Discriminating Features

The authors use a machine-learning feature generator to identify facial measurements that distinguish the two groups, then compare their distributions and summary statistics. The reported differences are largest for lip curvature and mouth angle, while eye-corner distance differs less.

4. Discriminating Features

  • The Feature Generating Machine identifies eye-corner, mouth, and philtrum regions; the resulting features are upper-lip curvature ρ, inner-eye-corner distance d, and mouth angle θ.ρ is measured on standard ID photos with neutral facial expression.Source: §4 · p. 9, Figure 8 · p. 9
  • Hellinger distances are 0.3208 for ρ, 0.2971 for d, and 0.3855 for θ.The distance ranges from 0 for identical distributions to 1 for completely different distributions.Source: §4 · p. 9
  • Criminals’ θ is on average 19.6% smaller, ρ is 23.4% larger, and d is 5.6% narrower than in non-criminals.The authors also report larger variance for θ among criminals.Source: §4 · p. 9, Table 4 · p. 9, Figure 9 · p. 10

5. Face Clustering on Manifolds

The authors argue that a single average face poorly represents these groups and use manifold analysis and clustering to examine their facial variation. In this sample, criminal faces show more variation and form four separable clusters, compared with three for non-criminal faces.

5. Face Clustering on Manifolds

  • A conventional average face fails to distinguish the groups clearly, and the authors argue that a single representative face is unsuitable for a social label such as criminality.The displayed averages use either landmark-point warping or eigenface representations.Source: §5 · p. 10, Figure 10 · p. 10
  • The authors propose that mingled but separable class distributions and similar sample means could reflect concentric manifolds, with the non-criminal manifold inside the more spread criminal manifold.The distance ordering is Dc > D× > Dn.Source: §5 · pp. 10–11
  • Isomap indicates that the data can be represented reasonably well in four to six dimensions, with the first four dimensions used to visualize both groups.The visualization shows the data clouds from three perspectives.Source: §5 · p. 11, Figure 11 · p. 11, Figure 12 · p. 12
  • The authors conclude that a meaningful typical criminal face does not exist and use geodesic K-means to identify representative subtypes instead.The clustering assigns each face to its nearest centroid in geodesic distance.Source: §5 · p. 11
  • The representative faces are formed by aligning and averaging faces within each converged cluster; the same process is applied to non-criminal faces.The synthesized criminal and non-criminal subtype faces were assessed by 50 Chinese student participants using binary judgments.Source: §5 · p. 11, Figure 13 · p. 12
  • Within-cluster variation drops quickly through K = 4 for criminal faces, while non-criminal faces do not form as many separable clusters by geodesic distance.The analysis supports four criminal subtypes and three non-criminal subtypes.Source: §5 · p. 11, Figure 14 · p. 12
  • Criminal faces show appreciably greater facial variation than non-criminal faces, within the examined sample of young or middle-aged Chinese men.The authors qualify this conclusion to the class of subjects studied.Source: §5 · p. 12

6. Conclusions

The authors report that supervised classifiers can infer criminality from facial images without relying on human observers’ subjective judgments. They also conclude that, after controlling for race, gender, and age, non-criminal faces vary significantly less than criminal faces.

6. Conclusions

  • Supervised machine-learning classifiers make reliable inferences on criminality from facial images, without relying on subjective judgments by human observers.The authors describe their study as the first of this kind.Source: §6 · p. 12
  • After controlling for race, gender, and age, the general law-abiding public has significantly less facial variation than criminals.The authors describe this pattern as a law of normality for non-criminal faces.Source: §6 · p. 13
Loading 1611.04135v3…