Source-linked AI summary
Investigating Audio, Visual, and Text Fusion Methods for End-to-End Automatic Personality Prediction
Onno Kampman, Elham J. Barezi, Dario Bertero, Pascale Fung
TL;DR
The paper investigates how audio, text, and video contribute to Big Five personality prediction, motivated by the importance of personality-aware virtual agents. It combines three modality channels using decision-level and neural-network fusion, and reports that full-backpropagation fusion improves on the best individual modality.
Problem
Personality prediction for emotional and empathetic virtual agents should account for behavioral and verbal cues expressed through language, speech, and visual input.
Source: §1 · p. 1Method
The paper uses separate audio, text, and video channels and compares decision-level fusion with concatenated representations trained with frozen channels or full backpropagation.
Source: §2 · p. 2, §2.4 · pp. 3–4Results
0.0938 average MSE across traits with full-backpropagation fusion, 9.4% better than the best individual modality, visual.
Source: §4 · p. 5Takeaways & Limitations
Full backpropagation performs better than linear-combination fusion, and the authors report that modalities interact in a non-trivial manner.
Source: §3.4 · p. 5
Abstract
from arXiv · showhide
We propose a tri-modal architecture to predict Big Five personality trait scores from video clips with different channels for audio, text, and video data. For each channel, stacked Convolutional Neural Networks are employed. The channels are fused both on decision-level and by concatenating their respective fully connected layers. It is shown that a multimodal fusion approach outperforms each single modality channel, with an improvement of 9.4\% over the best individual modality (video). Full backpropagation is also shown to be better than a linear combination of modalities, meaning complex interactions between modalities can be leveraged to build better models. Furthermore, we can see the prediction relevance of each modality for each trait. The described model can be used to increase the emotional intelligence of virtual agents.
1 Introduction
Personality prediction is relevant to emotional and empathetic virtual agents, and should account for behavioral and verbal cues across language, speech, and visual input. This paper investigates those modalities and fusion methods to understand their contributions and improve prediction over individual modalities.
1 Introduction
- Multimodal prediction is motivated by the use of behavioral and verbal cues in human personality judgments and the goal of developing emotional, empathetic virtual agents.The paper identifies language, speech, and visual cues as inputs a prediction model should consider.Source: §1 · p. 1
- The paper models personality with the Big Five descriptors, defining an individual’s personality as five scores in [0, 1] for Extraversion, Agreeableness, Conscientiousness, Neuroticism, and Openness to Experience.Source: §1 · p. 1
- Prior ChaLearn 2016 teams used neural networks with traditional audio features, while another approach fed raw audio and video to a network without adapting or merging modalities.The paper notes that the challenge largely targeted computer vision, with many teams using facial features.Source: §1 · p. 1
- The study examines speech, language, and video frames, first separately and then through fusion methods, to assess modality contributions and improve performance over individual channels.The authors also aim to understand how personality is expressed across modalities.Source: §1 · p. 1
2 Methodology
The method uses separate audio, text, and video channels, then compares decision-level and neural-network fusion. End-to-end training updates channel parameters to learn multimodal representations.
2 Methodology
- The three modality channels are trained end-to-end from raw inputs to outputs, with all parameters learned from data, and the architecture combines their representations through decision-level and neural-network fusion.The channels predict the five traits together rather than using separate models for each trait.Source: §2 · pp. 1–2
2.1 Audio channel
- The audio channel processes raw waveforms and squared-amplitude signals through four convolutional layers, then pools layer outputs and maps them to five trait scores.Global average pooling captures characteristics across the audio frame; training also randomizes amplitude to reduce recording-volume bias.Source: §2.1 · p. 2
2.2 Text channel
- The text channel feeds 300-dimensional pretrained word2vec embeddings from transcriptions into a CNN that encodes sentences using three-, four-, and five-word windows.The window encodings are max-pooled separately before being mapped to the Big Five scores.Source: §2.2 · p. 2
2.3 Video channel
- The video channel uses a random frame for appearance-only prediction, extracting image representations with pretrained VGG-face and training only its final two fully connected layers.The approach omits LSTMs because it does not use temporal or movement information, and avoids manual face cropping to limit preprocessing.Source: §2.3 · pp. 2–3
2.4 Multimodal fusion
- Fusion compares a trait-specific weighted ensemble of channel predictions with concatenated channel representations trained either with frozen channels or full backpropagation.The ensemble weights are learned by minimizing development-set MAE and expose modality relevance by trait; full backpropagation also updates audio and text parameters to learn more complex interactions.Source: §2.4 · pp. 3–4
3 Experiments
The experiments evaluate personality prediction on short ChaLearn vlog clips and compare individual modalities with several fusion approaches. Full backpropagation performs best among the reported modalities and fusion methods, while the discussion highlights trait-specific modality relevance and interactions between channels.
3.1 Corpus
- The evaluation uses ChaLearn First Impressions YouTube vlog clips of about 15 seconds, with 6,000 training clips and 2,000 validation clips as the test set.Twenty percent of the training set was used for validation during training.Source: §3.1 · p. 4
3.2 Setup
- Models were trained and evaluated by minimizing and measuring average MSE across the five traits, using Adam.The implementation used Keras.Source: §3.2 · p. 4
3.4 Discussion
- Trait-specific fusion weights expose modality contributions; text adds little beyond Agreeableness and Conscientiousness, while Extraversion is recognizable from tone and appearance.Extraversion may be evident from a short slice, whereas Openness may require longer temporal information.Source: §3.3 · p. 4, Table 1 · p. 4, §3.4 · p. 5
3.3 Results
- 0.0938 average MSE is the best reported fusion result, improving 9.4% over video and about 2.9% over limited backpropagation.The results are lower than estimating the average from training-set labels.Source: §3.3 · p. 4, Table 2 · p. 5
- The tri-modal method with full backpropagation is comparable to the two ChaLearn 2016 competition winners.Table 3 reports mean accuracy for individual modalities, fusion approaches, the winners, and the training-label average.Source: §3.3 · p. 4, Table 3 · p. 5
3.4 Discussion
- Full backpropagation performs significantly better for every trait except Agreeableness, consistent with non-trivial interactions between modalities.Decision-level fusion and limited backpropagation have very similar performance.Source: §3.4 · p. 5
4 Conclusion
The paper proposes a deep-network fusion method for predicting Big Five traits from audio, language, and appearance. Combining and fine-tuning the modalities achieves the best reported result, while the modalities carry different amounts of predictive information.
4 Conclusion
- The proposed fusion method uses deep neural networks to predict personality traits from audio, language, and appearance.The approach combines the network layers and fine-tunes parameters.Source: §4 · p. 5
- 0.0938 average MSE is the best reported result, 9.4% better than the best individual modality, visual.The conclusion attributes this result to combining the last network layers and fine-tuning parameters.Source: §4 · p. 5
- Language or speech patterns seem least relevant overall; video appearance is slightly more relevant than audio information.Each of the three modalities nevertheless contains a signal relevant to personality prediction.Source: §4 · p. 5