Source-linked AI summary
Integrating Text and Image: Determining Multimodal Document Intent in Instagram Posts
Julia Kruk, Jonah Lubin, Karan Sikka, Xiao Lin, Dan Jurafsky, Ajay Divakaran
TL;DR
Prior image-text work often treated one modality as secondary, despite the complex ways text and images can combine to shape author intent. This paper introduces a labeled Instagram dataset and taxonomies for these relationships. Multimodal models improve classification across the taxonomies, with the greatest semiotic gain for divergent pairs.
Problem
Prior image-text work generally treated either the image or text as primary, although their meanings can combine beyond simple addition or intersection.
Source: §1 · p. 1Method
The paper introduces MDID, a dataset of 1,299 Instagram posts labeled for authorial intent and contextual and semiotic image-caption relations.
Source: §1 · p. 2Results
Multimodal models improve across the taxonomies, with the largest semiotic gain on divergent pairs: 4.4% compared with the image-only model.
Source: §6.2 · p. 7, §6.3 · p. 7Takeaways & Limitations
The findings point to the importance of complex image-text relations, especially semiotic divergence, for studying the meanings of Instagram posts.
Source: §7 · p. 9The dataset and baseline classifiers are preliminary; future work should examine larger datasets, additional data, richer classification schemes, and more sophisticated classifiers.
Source: §7 · p. 9
Abstract
from arXiv · showhide
Computing author intent from multimodal data like Instagram posts requires modeling a complex relationship between text and image. For example, a caption might evoke an ironic contrast with the image, so neither caption nor image is a mere transcript of the other. Instead they combine -- via what has been called meaning multiplication -- to create a new meaning that has a more complex relation to the literal meanings of text and image. Here we introduce a multimodal dataset of 1299 Instagram posts labeled for three orthogonal taxonomies: the authorial intent behind the image-caption pair, the contextual relationship between the literal meanings of the image and caption, and the semiotic relationship between the signified meanings of the image and caption. We build a baseline deep multimodal classifier to validate the taxonomy, showing that employing both text and image improves intent detection by 9.6% compared to using only the image modality, demonstrating the commonality of non-intersective meaning multiplication. The gain with multimodality is greatest when the image and caption diverge semiotically. Our dataset offers a new resource for the study of the rich meanings that result from pairing text and image.
1 Introduction
Instagram posts combine image and text in ways that make author intent difficult to infer from either modality alone. The paper introduces taxonomies and a dataset for these relationships, then tests multimodal classification.
1 Introduction
- Prior image-text work often treated one modality as primary and the other as a complement, although their meanings can combine in ways beyond addition or intersection.Meaning multiplication can include irony or indirection that requires inference beyond either component’s literal meaning.Source: §1 · p. 1
- Models of text-image relationships and author intent could support computational studies of advertising, propaganda, and other persuasive text.Source: §1 · p. 2
- The paper defines three taxonomies for authorial intent, literal contextual relations, and semiotic relations between image and caption, and introduces MDID with 1,299 annotated Instagram posts.The taxonomies are adapted from semiotics, rhetoric, and media studies.Source: §1 · p. 2
- The authors build a deep neural network model to classify posts and report that combining text and image improves classification, especially when the caption and image diverge.They position this work as a computational framework for investigating multimodal meaning multiplication.Source: §1 · p. 2
2 Prior Work
Prior research examined text-image meaning across semiotics, media studies, and computational tasks, but often treated one modality as subordinate. This paper builds on work addressing intent and non-literal interaction between images and text.
2 Prior Work
- Earlier accounts often assigned text or image a subordinate role; Marsh and Domas White (2003) proposed a taxonomy for image-text relations in textbooks and manuals, which this work adapts.The authors note that Instagram’s connotational aspects require additions to that taxonomy.Source: §2 · p. 2
- Computational multimodal research addressed captioning, visual question answering, literal or connotative meaning, image context in interaction, and images’ perlocutionary force.The cited work on perlocutionary force includes memorability, saliency, popularity, and virality.Source: §2 · p. 2
- Studies of intent covered political portraits, advertisements, persuasive video, recipe coherence, and visual rhetoric, while other work examined non-literal image-caption meaning.Zhang et al. (2018) categorized slogan-image relations and identified complex, non-literal interaction; Weiland et al. (2018) used a knowledgebase to generate pair gist.Source: §2 · pp. 2–3
3 Taxonomies
The paper proposes three complementary taxonomies: one for the intent of an image-caption pair and two for distinct relationships between image and text. The contextual taxonomy addresses literal meanings, while the semiotic taxonomy captures signified meanings and their interaction.
3 Taxonomies
- Three taxonomies address speaker intent and two aspects of image-caption relations, offering an answer to how words change an image’s meaning.Source: §3 · p. 3
3.1 Intent Taxonomy
- The intent taxonomy generalizes illocutionary categories for Instagram and uses eight intents; annotators label the image-caption pair as a whole despite category overlap.The categories include advocative, promotive, exhibitionist, expressive, informative, entertainment, and two provocative types: controversial and discrimination.Source: §3.1 · p. 3
3.2 The Contextual Taxonomy
- The contextual taxonomy classifies literal image-text relations as minimal, close, or transcendent, adapting categories from Marsh and Domas White (2003) to be symmetric for Instagram.Minimal and close can be viewed as degrees of semantic overlap, whereas transcendent indicates meaning expansion beyond that scale.Source: §3.2 · pp. 3–4
3.3 The Semiotic Taxonomy
- The semiotic taxonomy classifies signified meanings as divergent, parallel, or additive: opposing directions, shared meaning, or mutual amplification or modification, respectively.Contextual and semiotic labels need not align: a pair can have minimal literal overlap but additive semiotics.Source: §3.3 · p. 4
- Examples show divergent pairs ranging from irony and unrelated captions to exhibitionist images paired with inside jokes, reflections, or an individual’s opinions and plans.The figure’s first three pairs exemplify the three semiotic categories, while six more illustrate divergent relationships observed on Instagram.Source: §3.3 · p. 5, Figure 3 · p. 5
4 The MDID Dataset
MDID contains 1,299 public Instagram posts collected to represent the eight intent categories. The authors annotated each image-caption pair by consensus across intent, contextual relation, and semiotic relation.
4 The MDID Dataset
- MDID comprises 1,299 public Instagram posts collected to provide a diverse set for each of the eight illocutionary intent types.For each intent, the authors selected at least 16 hashtags or users likely to yield relevant posts.Source: §4 · p. 5
- Collection sources were matched to intent categories, including political or social hashtags for advocative posts, #ad for promotive posts, and #selfie or #ootd for exhibitionist posts.Other sources included expressive hashtags, informative accounts, and entertainment or provocative tags.Source: §4 · pp. 5–6
- Annotators checked whether each post was relevant and in English, then labeled intent, contextual relation, and semiotic relation; two authors resolved labels by consensus and removed unresolved disagreements.Albums were preprocessed into single image-caption pairs before annotation.Source: §4 · p. 6
5 Computational Model
The model offers a baseline for exploring meaning multiplication by predicting labels from Instagram posts using image, text, or both. It combines modality-specific encoders through a shared embedding and fusion layer before classification.
5 Computational Model
- The model predicts taxonomy labels from image, text, or both to provide a baseline and explore meaning multiplication.Its inputs are encoded by modality-specific components, then passed through fusion and class prediction.Source: §5 · p. 6
- For multimodal input, encoded image and text vectors are projected into a shared space, added, and used to predict class scores.The model uses ResNet-18 for images and an RNN over either word2vec-style or ELMo caption embeddings.Source: §5 · p. 6
6 Experiments
Across three Instagram-post taxonomies, models using both image and text generally outperform single-modality models, with the largest semiotic gain for divergent pairs. The experiments also show that text-only cues can support predictions and that intent classifications vary with captions and label distinctions.
6. Experiments
- Experiments predict intent, contextual relationships, and semiotic relationships using image, text, or both modalities.The study uses 1,299 posts, five-fold cross-validation, and macro-averaged ACC and AUC; captions receive basic preprocessing, while images do not.Source: §6 · p. 6, §6.1 · p. 6
6.2 Quantitative Results
- 82.6% was the intent accuracy for Txt-ELMo, compared with 76.0% for Img; on semiotic labeling, Txt-emb achieved 67.8% versus 66.5% for Txt-ELMo.ELMo also improved contextual-taxonomy performance, but not semiotic-taxonomy performance.Source: §6.2 · pp. 6–7, Table 2 · p. 7
- 4.4% was the multimodal gain over the image-only model for divergent semiotics, the largest gain in that taxonomy.For intent, Img + Txt-ELMo achieved 85.6% compared with 82.6% for Txt-ELMo; joint models also improved relationship labeling.Source: §6.2 · p. 7, Table 2 · p. 7
6.3 Discussion
- Multimodal gains were greatest for divergent semiotics and minimal contextual relationships, and smaller for additive or parallel semiotic pairs.Text alone reached 72.7% AUC for divergent semiotics; examined cases often had irony or sarcasm cues, while combining modalities consistently improved both relationship taxonomies.Source: §6.3 · pp. 7–8
- Informative and exhibitionist intents were relatively easy to detect, while expressive/exhibitionist and provocative/advocative pairs were often confused.The authors associate these confusions with category distinctions: topic versus poster, and similar point-making aims, respectively.Source: §6.3 · pp. 7–8, Figure 4 · p. 7
6.4 Sample Outputs
- With the same royal-family image, changing the caption from “the royal family” to “my happy family” changed the predicted intent from entertainment to expressive.Figure 5 also shows ranked predictions for the three taxonomies; another example pairs youthful language with older people, classified as divergent and minimal.Source: §6.4 · p. 8, Figure 5 · p. 9
7 Conclusion
The paper presents a model and three taxonomies for capturing meaning multiplication in multimodal Instagram posts. It finds multimodal classification most helpful when image and text diverge semiotically, while describing the dataset and baseline classifiers as preliminary.
7 Conclusion
- The proposed model captures complex meaning multiplication between image and text in Instagram posts.The three taxonomies code literal, semiotic, and illocutionary relationships between text and image.Source: §7 · p. 9
- The dataset and baseline classifiers are preliminary; future work should examine larger datasets, hashtags, richer classification schemes, and more sophisticated classifiers.The authors also identify domain-specific coherence relations as a possible way to enrich the taxonomies.Source: §7 · p. 9
- Multimodal classification was most helpful when image and text diverged semiotically.The authors argue this finding highlights the importance of complex image-text relations.Source: §7 · p. 9