Source-linked AI summary
Automatic Understanding of Image and Video Advertisements
Zaeem Hussain, Mingda Zhang, Xiaozhong Zhang, Keren Ye, Christopher Thomas, Zuha Agha, Nathan Ong, Adriana Kovashka
TL;DR
The paper addresses the lack of computer-vision methods for understanding advertisements’ persuasive messages beyond their physical content. It creates richly annotated image and video ad datasets, analyzes persuasive strategies, and establishes baselines, including challenging message-focused question answering.
Problem
Automatic advertisement understanding is a novel computer-vision problem because ads communicate persuasive messages, actions, reasons, sentiments, and symbols beyond their objective physical content.
Method
The paper builds annotated datasets of 64,832 image ads and 3,477 video ads, covering topics, sentiments, actions, reasons, strategies, and symbolic references, then defines baseline prediction tasks.
Results
Baseline performance is higher on topic and sentiment recognition than on message-focused question answering, which achieves 11.48% accuracy.
Takeaways & Limitations
The datasets, strategy analysis, symbolism experiments, and baselines provide resources for progress on decoding advertisement rhetoric.
Takeaways & Limitations
Less than 50% of strategy instances are straightforward enough for perfect object recognition, and data for each portrayal domain may be limited to single examples.
Abstract
from arXiv · showhide
There is more to images than their objective physical content: for example, advertisements are created to persuade a viewer to take a certain action. We propose the novel problem of automatic advertisement understanding. To enable research on this problem, we create two datasets: an image dataset of 64,832 image ads, and a video dataset of 3,477 ads. Our data contains rich annotations encompassing the topic and sentiment of the ads, questions and answers describing what actions the viewer is prompted to take and the reasoning that the ad presents to persuade the viewer ("What should I do according to this ad, and why should I do it?"), and symbolic references ads make (e.g. a dove symbolizes peace). We also analyze the most common persuasive strategies ads use, and the capabilities that computer vision systems should have to understand these strategies. We present baseline classification results for several prediction tasks, including automatically answering questions about the messages of the ads.
1. Introduction
The paper introduces automatic advertisement understanding as a computer-vision problem requiring interpretation of persuasive visual rhetoric, and develops annotated image and video datasets plus baseline tasks to advance it.
- Motivation: Ads persuade viewers through messages conveyed by body language, visual associations, common-sense reasoning, symbolism, and non-photorealistic object recognition.Understanding these messages requires more than identifying objects, their locations, or producing image captions.
- Datasets: 64,000+ annotated image ads cover topics, intended sentiments, persuasive strategies, viewer actions, reasons to act, and symbolic references.The image collection includes both product advertisements and public service announcements.
- Datasets: 3,000+ annotated video ads provide similar labels, with additional annotations for whether an ad is funny or exciting.The video annotations omit symbolism but add properties associated with video ads.
- Baselines: The paper formulates message decoding as visual question answering and provides baselines for question answering, topic recognition, sentiment recognition, and symbolism.The proposed questions ask what the viewer should do and why, while topic and sentiment recognition are more approachable tasks.
- Applications: Automatic ad understanding could support ad-effectiveness prediction, viewer awareness, interest-based targeting, and richer access for visually impaired people.These applications depend on predicting persuasive strategies or decoding the messages conveyed by advertisements.
2. Related work
The paper positions advertisement understanding as a previously unaddressed computer-vision problem related to visual attributes, persuasion, sentiment, advertising analysis, and visual question answering.
- Research gap: The paper identifies analyzing visual rhetoric as an aspect of visual data not previously tackled in computer vision.Visual rhetoric has instead been studied in Media Studies and related advertising questions in marketing research.
- Beyond objects: Prior work on semantic visual attributes extends image descriptions beyond object labels to properties such as “furry”, “smiling”, and “metallic”.Other cited work addresses subjective or abstract image content, including athletic performance, artistic style, and architectural style.
- Visual persuasion: Visual-persuasion research detects positive or negative portrayals of politicians using facial expressions, gestures, and backgrounds, whereas advertisements use additional techniques and often omit people.The paper therefore treats advertisement rhetoric as broader than person portrayal.
- Sentiment: Advertisement sentiment prediction differs from generic image-emotion recognition because ads are purposefully created to convey emotion and require domain-specific decoding.The paper reports that a prior generic-image method does not transfer successfully to predicting emotion in ads.
- Prior work on ads: Existing advertising studies predict click-through rates, viewer liking, or ad placement, while this paper decodes what advertisements are about and what messages they carry.These goals are distinguished from decoding advertisements’ messages.
- Visual question-answering: The paper frames advertisement-rhetoric decoding as visual question answering, a vision-and-language problem related to image captioning.This formulation supplies the context for asking what an advertisement is trying to communicate.
3. Image dataset
The image dataset combines large-scale advertisement collection with annotations for topics, sentiments, persuasive messages, symbolic references, and related quality-control procedures.
- Image collection: The dataset contains product advertisements and public service announcements, with the advertised product or subject recorded as the topic.Images were gathered using hierarchical keyword queries spanning coarse categories, fine-grained brands, and PSA subjects.
- Image collection: About 190,000 noisy images remained after SIFT bag-of-words de-duplication, followed by human and ResNet-based filtering for advertisements.The classifier operated on 168,000 unannotated images with 80% recall and 85% held-out precision; annotators rejected 10% of its ad predictions.
- Topics and sentiments: Annotators selected among 38 topics and 30 sentiments, with topic labels collected for all ads and sentiment labels for 30,340 ads.Each image received annotations from 3 to 5 workers, and topic-label inter-annotator agreement was 85%.
- Questions and answers: The dataset includes three question-answer pairs per image asking what action the viewer should take and why.Responses were elicited in the form “I should [Action] because [Reason]” and later split into separate action and reason questions.
- Symbols: Symbolism annotations identify non-literal ads, bounding-box signifiers, and their signified concepts; 13,938 images contained symbolism across 221 retained symbols.The two-stage process first distinguished literal from non-literal interpretations, then labeled the signifier and signified.
- Annotation quality and cost: Annotation collection addressed speed, cost, and quality through adaptive pricing, majority voting, heuristic screening, and manual review.Collecting annotations for both image and video ads cost $13,030.
4. How can we decode ads?
The paper frames ad decoding as understanding persuasive strategies beyond literal object and text recognition. It develops a nine-strategy taxonomy, analyzes strategy prevalence and topic correlations, and identifies major unresolved visual-understanding challenges.
- Strategy taxonomy: The authors define a nine-strategy taxonomy spanning literal, physical-process, property-transfer, symbolic, cultural-reference, experiential, atypical-object, humor, and contrast-based ads.Five authors labeled 100 ads each, after which similar strategies were merged for consistency.
- Strategy prevalence: Straightforward and symbolic ads are the most common strategies in both author and crowdsourced annotations.The crowdsourced chart used 4,000 ads and showed slightly higher frequencies for rarer strategies.
- Strategy prevalence: Less than 50% of strategy instances are straightforward enough to decode with perfect object recognition alone.The authors therefore argue that ad decoding requires capabilities beyond object recognition, while noting that each strategy has relatively few examples.
- Required capabilities: Ad decoding requires symbolism, physical-process understanding, recognition of non-photorealistic objects, and detection of surprise or humor.The paper describes symbolism and physical-process understanding as needing further techniques, atypical-object recognition as challenging, and surprise or humor detection as largely unsolved.
- Topic-strategy correlations: Strategy usage correlates with topic: symbols are most common in smoking ads, atypical portrayals in financial ads, and surprise in healthcare and safety ads.These correlations are analyzed in Figure 3.
5. Video dataset
The paper builds a video-ad dataset by combining provider data with popular, positively rated YouTube videos and collecting annotations about content, sentiment, actions, and reasons. Video ads show more humor and excitement but less symbolism than image ads.
- Dataset construction: The video dataset is smaller because commercials are expensive to produce and air, leaving fewer examples available on the web.The authors state that video advertisements may cost several million USD to air.
- Dataset construction: The authors combined 949 videos from an Internet service provider with YouTube videos meeting popularity and like-to-dislike criteria, then automatically deduplicated them.For each video, they sampled 30 frames from both the beginning and end before reducing resolution and averaging.
- Annotations: Video annotations include topic, sentiment, and answers to what viewers should do and why they should do it.The topic and sentiment options overlap with those used for the image dataset.
- Annotation statistics: Cheerfulness is most common for beauty and soda ads, eagerness for soda ads, creativeness for electronics ads, and alertness for political ads.These response statistics are reported in Figure 5 and Table 6.
- Video-specific characteristics: Video ads contain more annotations for being funny or exciting and are less likely to use symbolism than image ads.The authors attribute this contrast to videos having more space and time to convey their messages.
6. Experiments
The experiments benchmark automatic understanding of image and video advertisements across question answering, symbolism, topic, sentiment, and affective-label prediction. Results show that message and symbolism decoding remain difficult, while topic and some video affective labels are more feasible.
- 6.1. Question-answering for image ads: 11.48% accuracy was achieved on single-word image-ad question answering, indicating that existing methods struggle with implicit visual rhetoric and persuasion.Answers were counted correct when matching any of three human answers, but annotator disagreement makes the task challenging.
- 6.1. Question-answering for image ads: 48.45% accuracy was achieved when image-ad explanations were clustered into 30 prototype answers, simplifying the question-answering task.The prototypes capture recurring patterns in the reasons annotators gave for “why” questions.
- 6.2. Symbolism prediction: 15.79% F-score was achieved for distinguishing 221 symbolic concepts, reflecting the difficulty of detecting symbols with variable visual realizations.An attention model performed slightly better than direct classification, and symbols may correspond to different image regions.
- 6.2. Symbolism prediction: 11.96% accuracy on 1000-way QA and 50% on 30-way QA followed from adding symbolism or topic, sentiment, and symbolism features, versus 11.48% and 48.45% baselines.The authors describe this symbolism-assisted question-answering experiment as preliminary and leave better prediction and use of symbolism for future work.
- 6.3. Question-answering for video ads: 8.83% QA accuracy was obtained for video advertisements using video features from the video representation described in the experiments.The video QA process followed the same process as the preceding image-ad experiment.
- 6.5. Topic and sentiment on video ads: 35.1% topic accuracy and 32.8% sentiment accuracy were obtained for video ads, while humor and excitement reached 78.6% and 78.2% accuracy against majority-class baselines of 58% and 60.8%.Video topic and sentiment used multi-class SVMs on averaged C3D features; humor and excitement used binary SVMs on fc7 features.
7. Conclusion
The paper contributes annotated image and video advertisement datasets, analyses capabilities needed to decode ad rhetoric, an initial symbolism solution, and baselines including message-focused question answering. Future work targets richer symbolism, knowledge-based decoding, improved video and audio modeling, and additional visual capabilities.
- 7. Conclusion: The paper provides a large annotated image-ad dataset and a companion annotated video-ad dataset for studying advertisement understanding.It also reports analyses of capabilities needed for vision systems and baselines on several prediction tasks, including question answering about subtle ad messages.
- 7. Conclusion: Future work includes improved symbolism detection, knowledge bases, stronger video and audio modeling, and using topic, sentiment, humor, and excitement predictions for question answering.The authors also identify recognizing atypical objects and modeling physical processes as future directions.