Source-linked AI summary
Visual Affect Around the World: A Large-scale Multilingual Visual Sentiment Ontology
Brendan Jou, Tao Chen, Nikolaos Pappas, Miriam Redi, Mercan Topkara, Shih-Fu Chang
TL;DR
Visual-affect methods often assume that emotion is universal, while their generalization across languages and cultures remains largely unexplored. The paper builds a multilingual visual sentiment ontology using language-specific discovery and hierarchical clustering. It reports a pipeline applied across 12 languages, with coverage and diversity as goals, and releases the ontology and image dataset publicly.
Problem
Prior visual-affect methods often assume emotion is universal, while their applicability and generalization to other languages and cultures remain largely unexplored.
Source: §1 · p. 2Method
The paper discovers multilingual adjective-noun concepts using language-specific part-of-speech labeling and progressive filtering, then unifies them through two-stage hierarchical clustering.
Source: §7 · p. 10Results
The authors report applying their discovery method on social multimedia for 12 languages and using language-specific labeling and progressive filtering to achieve coverage and diversity.
Source: §7 · p. 10Takeaways & Limitations
The authors make MVSO and its image dataset available to the public as resources for multilingual visual-affect research.
Source: §7 · p. 10The study omits an in-depth analysis of Flickr geolocation because of the noise it can add, and some languages have insufficient data.
Source: §7 · p. 10
Abstract
from arXiv · showhide
Every culture and language is unique. Our work expressly focuses on the uniqueness of culture and language in relation to human affect, specifically sentiment and emotion semantics, and how they manifest in social multimedia. We develop sets of sentiment- and emotion-polarized visual concepts by adapting semantic structures called adjective-noun pairs, originally introduced by Borth et al. (2013), but in a multilingual context. We propose a new language-dependent method for automatic discovery of these adjective-noun constructs. We show how this pipeline can be applied on a social multimedia platform for the creation of a large-scale multilingual visual sentiment concept ontology (MVSO). Unlike the flat structure in Borth et al. (2013), our unified ontology is organized hierarchically by multilingual clusters of visually detectable nouns and subclusters of emotionally biased versions of these nouns. In addition, we present an image-based prediction task to show how generalizable language-specific models are in a multilingual context. A new, publicly available dataset of >15.6K sentiment-biased visual concepts across 12 languages with language-specific detector banks, >7.36M images and their metadata is also released.
1. INTRODUCTION
The paper asks how images in different languages express affective concepts and convey sentiment and emotion across cultures. It introduces a multilingual ontology built from adjective-noun pairs to address the limited exploration of cross-language generalization in visual affect.
1. INTRODUCTION
- The paper investigates how images in different languages express affective concepts and how those concepts convey emotion and sentiment across languages.Examples include “beautiful place” and “delicious food.”Source: §1 · p. 1
- The construction process starts from images and metadata retrieved using emotion keywords, forms candidate adjective-noun pairs from tagged words, then filters candidates.The figure describes the final ontology as aiming for diversity and coverage.Source: Figure 2 · p. 2
- Prior visual sentiment work uses adjective-noun pairs to connect detectable nouns with affective adjectives, but cross-language generalization remains largely unexplored.For example, “beautiful sky” and “scary dog” pair detectable nouns with sentiment or emotion.Source: §1 · p. 2
- MVSO is a multilingual visual sentiment ontology and dataset covering adjective-noun pairs in 12 languages.The listed languages include Arabic, Chinese, Dutch, English, French, German, Italian, Persian, Polish, Russian, Spanish, and Turkish.Source: §1 · p. 2
- The ontology organizes visually detectable noun clusters with sentiment-biased adjective-noun subclusters and includes a multilingual visual concept detector bank.The contributions also include a large-scale image collection and benchmark cross-lingual sentiment prediction.Source: §1 · p. 2
2. RELATED WORK
Prior work spans visual-affect datasets, psychology, and multilingual text and music analysis, but the authors identify limited attention to cultural context in visual affect. They position MVSO as a multimedia-driven, multilingual approach and contrast its discovery pipeline with earlier VSO methods.
2. RELATED WORK
- IAPS, GAPED, and VSO provide affective image resources, while the authors note that existing datasets and methods assume visual affect is universal rather than accounting for culture or language.IAPS focuses on induced emotions, GAPED narrows image themes, and VSO was created from Flickr data.Source: §2 · p. 2
- Unlike work that often uses handpicked data, the authors gather images and metadata “in the wild” from a popular multilingual social multimedia platform.Source: §2 · p. 3
- Cross-cultural psychology retains competing views of emotion as pan-cultural, culture-specific, or hybrid, and the authors describe their work as the first computational multimedia investigation of visual affect and culture, as far as they know.The cited survey is.Source: §2 · p. 3
- Multilingual text sentiment and music retrieval research provide precedents, but this paper instead studies cross-cultural visual sentiment in online image collections.The related work discusses text sentiment methods and, and multilingual music approaches and.Source: §2 · p. 3
- Compared with VSO and SentiBank, MVSO uses language-specific part-of-speech taggers, automatically discovers emotion-correlated adjective-noun pairs, and applies stronger selection criteria.The authors characterize MVSO as extendable to any language and as achieving greater coverage and diversity than VSO.Source: §2 · p. 3
3. ONTOLOGY CONSTRUCTION
MVSO construction discovers language-specific adjective–noun pairs from emotion-seeded social images, filters them for linguistic, sentiment, frequency, and diversity properties, and validates candidates with native speakers.
3. ONTOLOGY CONSTRUCTION
- Language and culture are treated interchangeably, and 12 languages spanning regions were selected based on available NLP tools and emotion ontologies; Flickr supplied multilingual social-media data.The authors describe language as a lens through which culture can be observed.Source: §3 · p. 3
3.1 Adjective-Noun Pair Discovery
- Plutchik’s 24 graded emotion keywords were translated or supplemented by 12 speakers per language, then used to retrieve Flickr images and related tags.The seed ontology groups eight basic emotions into three valences each.Source: §3.1 · pp. 3–4
- Exact-pair tag co-occurrence and language-specific POS taggers identified ANPs; about ∼86% of VSO pairs overlapped the English discoveries.A pair had to occur as that exact phrase in image tags for its emotion.Source: §3.1 · p. 4
3.2 Filtering Candidate Adjective-Noun Pairs
- Candidates were filtered for language, entities, technical terms, and non-neutral sentiment; conflicting adjective/noun polarity uses the adjective score alone, otherwise scores are summed.Zero-sentiment candidates were removed.Source: §3.2 · pp. 4–5
- Usage filters retained non-English ANPs seen once but required 40 English tags and three uploaders; per adjective, noun diversity was capped at 100 frequent ANPs.The uploader threshold was used to reduce bias from highly prolific accounts.Source: §3.2 · p. 5
3.3 Crowdsourcing Validation
- Crowdsourcing addressed residual named-entity, language-mixing, grammar, and semantic errors; the release also includes a pre-crowdsourced ontology version.Native-speaking workers evaluated candidate ANP correctness after automatic filtering.Source: §3.3 · p. 5
3.3.1 Crowdsourcing Setup
- At least three workers judged each pair; native-language expertise, reputation, grammar/language/generality/semantics criteria, and seven-of-ten entry tests controlled quality.For Persian, Polish, and Dutch, workers were filtered by country because the platform did not assess language expertise.Source: §3.3.1 · p. 5
3.3.2 Crowdsourcing Results
- Three in five candidate ANPs were approved in English, Spanish, and Russian; agreement averaged 87%, exceeding 90% in Arabic, German, and Polish.German, Dutch, Persian, and French had more rejections than approvals, often due to mixed-language pairs.Source: §3.3.2 · p. 6
4. DATASET ANALYSIS & STATISTICS
The dataset analysis compares MVSO with VSO and examines sentiment and emotion patterns across languages. It also reports the scale of the image collection associated with the 12-language ontology.
4. DATASET ANALYSIS & STATISTICS
- The collection contains 7,368,364 images across 15,630 ANPs in 12 languages.English, Spanish, and Italian contributed the most images.Source: §4 · p. 6
4.1 Comparison with VSO [5]
- English MVSO overlaps VSO by about 86%, converging to 100% at higher frequency thresholds, and exceeds it in ANP count and adjective/noun diversity.The comparison also finds that VSO has more ANPs than most other languages, with Spanish, Italian, and French exceptions at low thresholds.Source: §4.1 · p. 6, Figure 3 · p. 6
- VSO has more ANPs than most other languages, while Spanish, Italian, and French exceed its adjective and noun diversity at t ≤ 10^3.German and Dutch also exceed VSO's diversity at smaller thresholds; the remaining languages have smaller diversity over most thresholds.Source: §4.1 · pp. 6–7, Figure 3 · p. 6
4.2 Sentiment Distributions
- Across languages, sentiment trends positive overall; Spanish has the highest median, while Persian and Arabic have the lowest, followed by English.Spanish also has the greatest sentiment variation, whereas German's distribution is most concentrated; the lowest-median languages cluster near zero, from 0 to -0.1.Source: §4.2 · p. 7, Figure 4 · p. 7
4.3 Emotion Distributions
- The five most frequent emotions across languages are joy, serenity, interest, grief, and fear, with joy highest in Russian, Chinese, and Arabic.The emotion rankings are more concrete for languages with over 1,000 ANPs and less conclusive for Arabic and Persian, which have fewer ANPs.Source: §4.3 · p. 7, Figure 5 · p. 7
5. CROSS-LINGUAL MATCHING
The section aligns translated ANPs across languages using exact phrase matches and approximate clustering. Approximate alignment organizes concepts into noun clusters and sentiment-biased ANP subclusters, forming a multilingual hierarchy.
5. CROSS-LINGUAL MATCHING
- For exact translations, German ANPs matched Dutch ANPs in 39 shared phrases versus 23 for Chinese, despite fewer Dutch translation phrases overall.The analysis aligned ANP translations to English and applied the pipeline's validation filters.Source: §5 · pp. 7–8, Figure 6 · p. 8
- Approximate matching produced a hierarchical ontology with 3,329 ANP subclusters; German and Dutch share 118 clusters, and German and Chinese share 101.The approach first clusters translated nouns, then clusters full phrases within each noun cluster.Source: §5 · p. 8
- The examples show noun groupings such as dog-cat-pony, with sentiment-biased subclusters including funny dog and adopted dog.The figure visualizes word2vec vectors projected with t-SNE.Source: §5 · p. 8, Figure 7 · p. 8
6. VISUAL SENTIMENT PREDICTION
The section evaluates language-specific visual sentiment models and cross-lingual transfer using MVSO images. Models perform best on their own language, while transfer patterns vary by language, with Chinese especially difficult and Latin-language transfer comparatively strong.
6. VISUAL SENTIMENT PREDICTION
- The prediction task tests whether one language’s visual sentiment can be predicted by models trained on other languages, using binary positive-versus-negative labels.The authors chose sentiment rather than emotion prediction to reduce variables while examining cross-lingual differences.Source: §6 · p. 8
6.1 Visual Sentiment Concept Detectors
- The six selected languages—English, Spanish, Italian, French, German, and Chinese—accounted for 94.7% of MVSO ANPs and 98.4% of dataset images.The section reports higher top-1 and top-5 accuracy than DeepSentiBank, but the supplied table passage gives no cell values.Source: §6 · p. 8, §6.1 · p. 9, Table 4 · p. 9
- The visual-concept detector bank used an AlexNet-styled architecture and fine-tuned six CNN models, one for each language, training only on ANPs with at least 125 images.For each ANP, images were randomly split 80/20% into train/test sets; this selection covered 63.5% of ANPs and 92.0% of images.Source: §6.1 · p. 8, §6 · p. 8
6.2 Sentiment Prediction on Flickr
- Sentiment prediction used linear SVMs trained on CNN image features, with ANP sentiment scores binarized into positive and negative labels.Training used images associated with ANPs whose sentiment scores had absolute values above 0.05, and positive/negative splits were stratified across languages.Source: §6.2 · p. 9
- Penultimate-layer softmax features performed best; models did best on their own language, while Chinese was hardest to predict cross-lingually and Latin-language transfer was comparatively strong.Italian best predicted Spanish and French sentiment, while Spanish best predicted Italian; the English model had the least accuracy variance across languages despite lower average performance.Source: §6.2 · p. 9, Figure 9 · p. 9
- Cross-lingual examples included an Italian traditional-clothing image labeled positive but predicted negative by the German model, and a Chinese model misclassifying an English foggy-morning image as positive.A French village image was correctly classified as positive by the Spanish predictor; the authors describe these examples as highlighting similarities and differences in cultural expression and perception.Source: §6.2 · pp. 9–10, Figure 10 · p. 10
7. CONCLUSION & FUTURE WORK
The paper concludes that its multilingual discovery pipeline and MVSO dataset support analysis of visual sentiment across 12 languages. Its preliminary analysis suggests both commonalities and distinctions in how visual affect is expressed and perceived, while future work includes additional user metadata and geo-location.
7. CONCLUSION & FUTURE WORK
- The authors propose a multilingual discovery method and hierarchical ontology, and report preliminary evidence of both commonalities and distinctions in visual affect across cultures.Their pipeline uses language-specific part-of-speech labeling and progressive filtering, then two-stage hierarchical clustering to unify the ontology across languages.Source: §7 · p. 10
- The authors make MVSO, its pre- and post-crowdsourcing versions, and an image dataset publicly available.Source: §7 · p. 10
- Future work includes studying user metadata and geo-location, which was omitted because location data can be noisy.The authors note that geo-location could help distinguish subcultures sharing a language, while user metadata could include age, gender, and profession.Source: §7 · p. 10