Automated Analysis and Prediction of Job Interview Performance

1 INTRODUCTION

• Automated data-driven quantification of verbal and nonverbal behavior together had not been explored in job interviews, despite established interview advice based largely on intuition and limited manual coding.The paper focuses on social and behavioral skills rather than job-specific qualifications.Source: §1 · pp. 1–2
• Language, speaking style, prosody, and facial expression can reflect personality and mental states, but their combined influence on interpersonal communication is difficult to quantify.The paper illustrates the issue by contrasting answers that differ in fluency, filler words, vocal delivery, and visible engagement.Source: §1 · p. 2
• The framework extracts lexical, facial, and prosodic features to predict interview ratings and investigate their relative importance.The research questions ask whether the recordings can support automatic assessment, modality comparisons, and recommendations for social traits.Source: §1 · p. 2
• r > 0.65 and AUC = 0.81 (baseline 0.50) for overall interview ratings; engagement, excitement, and friendliness reached r ≥0.75 and AUC > 0.85.The framework also predicts the likelihood of getting hired and 14 other social traits.Source: §1 · p. 2
• The article extends previously presented research with additional facial features and more comprehensive results and analysis.It studies 138 interview sessions with 69 individuals.Source: §1 · p. 2

2.1 Nonverbal Behavior Recognition

• Earlier social-interaction prediction work often relied on manual behavioral coding, while automated systems commonly focused on one modality despite interest in multimodal analysis.Prior work included prosody, facial expression, gesture, and word-use approaches, as well as combined prosodic and linguistic analyses that omitted facial cues.Source: §2.1 · p. 3
• Nguyen et al. predicted hiring decisions from nonverbal cues in 62 interview videos; this work adds language alongside prosody and facial expressions and examines their relative influences across social traits.The paper also reports 15 social traits, including friendliness, excitement, and engagement.Source: §2.1 · p. 3

2.2 Social Coaching for Job Interviews

• Interview coaching systems such as MACH and TARDIS provide behavioral feedback or recognize lower-level cues, while this framework aims to predict overall interview ratings and higher-level traits for coaching.TARDIS's NovA module uses manually annotated cues to infer traits; predicting higher-level traits is described as future work.Source: §2.2 · p. 3

3 DATASET DESCRIPTION

• The MIT Interview Dataset contains 138 audio-visual mock interviews with internship-seeking MIT students, totaling nearly 10.5 hours.The interviews averaged 4.7 minutes each and were conducted by professional career counselors.Source: §3 · p. 3

3.1.1 Study Setup

• Camera 1 recorded the interviewee's video and audio, while Camera 2 recorded the interviewer.The room had a desk, two chairs, and wall-mounted cameras with microphones.Source: Figure 2 · p. 4, §3.1.1 · p. 4

3.1.2 Participants

• Ninety MIT juniors initially participated, and 69 permitted their recordings to be used for research; all participants were native English speakers.Two professional MIT career counselors conducted the interviews.Source: §3.1.2 · p. 4

3.1.3 Procedure

• Each session used five career-service-recommended questions, and no job description was given; interviewers rated 16 categories, including overall performance, hiring recommendation, and behavioral traits.The first two questions assessed overall performance, while the remaining questions assessed dimensions such as warmth, presence, competence, and content.Source: §3.1.3 · p. 4

3.2 Data Labeling

• Mechanical Turk workers rated interview performance using the same questionnaire, to reduce bias that might arise from counselors' interactions with interviewees.Workers could pause and replay videos to rate them more thoroughly.Source: §3.2 · pp. 4–5

4 PREDICTION FRAMEWORK

• SVM and LASSO regression models are trained on automatically extracted features from interview videos.The framework uses the models both to predict ratings and to examine the relative importance of modalities and their interplay.Source: §4 · p. 5
• The models predict Turker ratings for overall performance and individual behavioral traits.Prediction covers both the overall score and each rated trait.Source: §4 · p. 5
• The learned models are also used to quantify the relative importance of modalities and their interplay.Source: §4 · p. 5

4.1.1 Prosodic Features

• Prosodic features summarize each interviewee’s five answers and include pitch, intensity, formants, spectral energy, pauses, voicing, jitter, and shimmer.Source: §4.1.1 · p. 5, Table 2 · p. 5

4.1.2 Lexical features

• To reduce sparse, high-dimensional unigram features, the system uses 23 LIWC psycholinguistic word categories and additional speaking-rate and fluency features.Source: §4.1.2 · p. 6, Table 4 · p. 6

4.1.3 Facial features

• Facial features include averaged smile intensity, nods and shakes, landmark-based brow, lip, and eye measures, and separately incorporated head-pose features.Source: §4.1.3 · pp. 6–7, Figure 3 · p. 7

4.1 Feature Extraction

• The system combines prosodic, lexical, and facial features into one vector and normalizes each feature to zero mean and unit variance.Source: §4.1 · p. 5, §4.1.4 · p. 7

4.1.4 Feature Normalization

• The feature vectors are standardized so features with different value ranges can be treated uniformly.Source: §4.1.4 · p. 7

• The model treats each interview’s true rating and each Turker’s reliability as latent quantities estimated from the collected ratings.Source: §4.2 · p. 7
• An EM-style iterative procedure alternates updates for the true ratings and Turker reliabilities until convergence, then uses estimated ratings to train prediction models.Source: §4.2 · p. 7
• Ratings are modeled as real-valued observations of the true rating with additive Gaussian noise, with each Turker’s inverse variance representing reliability.Source: §4.2 · p. 7

4.3 Score Prediction from Extracted Features

• SVR and Lasso were the best-performing models among the regression methods tested on this dataset.The other tested approaches included L1-regularized logistic regression and Gaussian process regression.Source: §4.3 · p. 8

4.3.1 Support Vector Regression (SVR)

• SVR predicts ratings from interview features with a learned weight vector and bias, while feature importance is interpreted through weight magnitude.The objective uses an epsilon tolerance and a tunable C parameter that trades off goodness of fit against generalization.Source: §4.3.1 · p. 8

4.3.2 Lasso

• Lasso minimizes residual prediction error with L1 regularization, which can push irrelevant feature coefficients to zero and reduce predictor variance.The regularization constant λ controls the sparsity of the learned weight vector.Source: §4.3.2 · p. 8

5 RESULTS

• The Mechanical Turk analyses assess rating quality by measuring agreement among workers.They also examine how individual traits relate to overall interview ratings and how segment ratings relate to ratings for entire videos.Source: §5 · p. 8
• The rating analyses identify traits associated with overall ratings and examine how ratings for video segments relate to whole-interview ratings.Source: §5 · p. 8
• The automated-feature analyses report SVR and Lasso prediction accuracy and examine the influence of modalities and individual features.Source: §5 · p. 8

5.1.1 Inter-Rater Agreement

• Engagement, excitement, and friendliness had relatively good inter-rater agreement, while stress, authenticity, speaking rate, and pauses had low agreement.The authors suggest workers may not be well positioned to judge some categories from video alone.Source: §5.1.1 · p. 9

5.1.2 Correlation among the Behavioral Traits

• Staying focused showed a 73% correlation with overall ratings; engaging tone, low awkwardness, excitement, and an appropriate smile were also among the top traits.The hiring-recommendation rating also had high correlation and mutual information with the overall rating.Source: §5.1.2 · p. 9, Figure 5 · p. 9

5.1.3 First (and Last) Impression Matters

• Ratings for the first interview question correlated most with whole-interview ratings, and that correlation gradually declined for later questions.This pattern also appeared for excitement, smiling, eye contact, engaging tone, and friendliness; a spike for the final question suggests perceptions could change during that response.Source: §5.1.3 · p. 9
• Pause, calmness, and stress did not follow the temporal pattern, but their ratings had very low correlations to begin with.The authors caution that the first-question association cannot distinguish an effect of question order from the verbal content; mock interviews opened with a background question.Source: §5.1.3 · p. 9

5.2.1 Prediction Accuracy using Trained Models

• 0.75 or higher correlation coefficients were achieved for engagement, excitement, and friendliness; overall performance and hiring recommendation reached r 0.70 for SVM.Source: §5.2.1 · p. 10, Figure 7 · p. 10
• AUC > 0.80 for SVM was reported for engagement, excitement, friendliness, hiring recommendation, and overall score, against a 0.50 baseline.Interviews were divided into positive and negative groups at each trait's median score.Source: §5.2.1 · p. 10, Table 5 · p. 11
• Lower prediction accuracy was associated with unreliable ground-truth ratings for some traits or missing informative features, such as eye tracking for eye contact.Source: §5.2.1 · p. 10

5.2.2 Feature Analysis

• Prosodic features received higher weights for predicting engagement and excitement, while facial features—particularly smiles—were significant for friendliness.Across five traits, prosody mattered for all, and using all three modalities produced the best correlation in most cases.Source: §5.2.2 · pp. 10–11, Figure 8 · p. 11

5.2.3 Recommendation from our Framework

• Higher speaking rate, fluency, and word use, along with fewer fillers and breaks, were associated with being perceived as a better interview candidate.Higher scores also correlated with more use of “They” and “We” and less use of “I”; smiling more was associated with better performance, and using more “We” than “I” with greater friendliness.Source: §5.2.3 · pp. 11–12

6 DISCUSSION AND CONCLUSION

• For job interviews, the proposed model predicted human interview ratings with correlation r > 0.65 and AUC ∼0.80 (baseline AUC = 0.50); engagement, excitement, and friendliness had r ∼0.75 and AUC > 0.85.Source: §6 · p. 12
• To the authors’ knowledge, the dataset is the largest collection of job-interview videos, with professional counselors and questions relevant to real-world interviews.They describe the recording settings as reasonably realistic.Source: §6 · p. 12
• All participants were MIT junior undergraduates; job-description-free ratings may diverge from hiring decisions and favor nonverbal cues, while mock interviews lacked real-interview stress and strong job-offer incentives.The authors propose more diverse populations and experiments with specific job descriptions and stronger incentives.Source: §6 · p. 12
• Nine Mechanical Turk workers rated each video, but their judgments may not match professional experts; the authors plan expert ratings and revalidation.Ratings were aggregated with an EM algorithm that accounted for worker reliability.Source: §6 · p. 12
• Video-wide averaged features cannot model fine-grained temporal interactions, such as smile and intonation while a specific word is spoken.The authors leave modeling such interactions across modalities for future work.Source: §6 · p. 12
• Rare momentary mistakes, such as swearing, can spoil an interview; the authors suggest anomaly detection may handle such events more effectively.Their rarity makes these phenomena difficult to model in the current framework.Source: §6 · p. 12
• Feature-weight analysis offers quantitative insights into successful interviews, with possible applications in interview feedback, social-difficulty support, and professional training.The authors name underprivileged youth, customer-service professionals, and medical professionals as possible beneficiaries.Source: §6 · p. 12