PaperlayerView exact revision on arXiv

Source-linked AI summary

Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response

Ferda Ofli, Firoj Alam, Muhammad Imran

arXiv:2004.11838v1cs.CVcs.CYcs.LGcs.MM

TL;DR

Disaster-response social media research has largely focused on text, despite the complementary information available in images. This paper learns a joint text-image representation with deep-learning models for two classification tasks; multimodal models outperform image-only models across both tasks’ reported measures.

  • Problem

    Disaster-response social media research has largely focused on text, although text and images can provide complementary information for time-critical analysis.

    Source: INTRODUCTION · p. 1

  • Method

    Parallel text and image deep-learning architectures produce features that are combined into a shared representation for classification.

    Source: INTRODUCTION · p. 2

  • Results

    Multimodal models perform about 1% better than image-only models on all informativeness measures and about 2% better on all humanitarian-task measures.

    Source: Results · p. 6

  • Takeaways & Limitations

    The experiments report that combining text and image modalities improves performance over single-modality models on the two studied disaster-response tasks.

    Source: INTRODUCTION · p. 2

    Co-occurring social-media text and images need not be strongly aligned and may convey different or contradictory information.

    Source: Challenges and Future Work · p. 9

Abstract

from arXiv · show

Multimedia content in social media platforms provides significant information during disaster events. The types of information shared include reports of injured or deceased people, infrastructure damage, and missing or found people, among others. Although many studies have shown the usefulness of both text and image content for disaster response purposes, the research has been mostly focused on analyzing only the text modality in the past. In this paper, we propose to use both text and image modalities of social media data to learn a joint representation using state-of-the-art deep learning techniques. Specifically, we utilize convolutional neural networks to define a multimodal deep learning architecture with a modality-agnostic shared representation. Extensive experiments on real-world disaster datasets show that the proposed multimodal architecture yields better performance than models trained using a single modality (e.g., either text or image).

INTRODUCTION

Disaster-response social media analysis has largely emphasized text, although text and images can provide complementary signals and time-critical information. This study combines both modalities for two classification tasks and reports better performance than single-modality baselines.

INTRODUCTION

  • Text and images can supply complementary signals, and single-modality analysis may miss information such as the scale of earthquake damage.The cited example contrasts a tweet reporting a 6.1 magnitude earthquake with its attached image showing destruction.Source: INTRODUCTION · p. 1
  • Earlier disaster-response social media research mainly analyzed text, while relatively little attention had gone to shared images.The introduction cites prior image-based work on infrastructure damage and recent multimodal analysis of informative tweet-image pairs.Source: INTRODUCTION · pp. 1–2
  • The study classifies whether tweets are informative for humanitarian aid and whether they contain useful humanitarian information such as injuries or infrastructure damage.The two problems are handled as separate classification tasks using text and image modalities.Source: INTRODUCTION · p. 2
  • The proposed architecture learns a joint representation from parallel text and image networks, then uses it for prediction.The image network uses VGG16 features; the text network is a CNN, and the joint representation is followed by a dense layer and softmax.Source: INTRODUCTION · p. 2
  • Multimodal models outperform single-modality baselines on informativeness and humanitarian classification, with F1-scores of 84.2 and 78.3, respectively.The experiments use CrisisMMD, and the reported F1-scores are for the best model on each task.Source: INTRODUCTION · p. 2
  • The paper presents baseline results on CrisisMMD for both tasks using state-of-the-art unimodal and multimodal deep-learning approaches.The authors describe this as the first study to report these baselines together and identify further improvement as future work.Source: INTRODUCTION · p. 3

RELATED WORK

Prior disaster-response work has studied textual, visual, and multimodal social media signals, while this study evaluates feature fusion on CrisisMMD across two tasks. Its setup differs from earlier studies in dataset scope, task coverage, and fusion approach.

RELATED WORK

  • Prior image studies examined disaster relevance, affected-area location, and damage severity using imagery and associated metadata (Peters and Joao 2015; Daly and Thom 2016; Alam, Ofli, et al. 2018b).Other cited work also applied deep learning to social-media image analysis (Nguyen, Alam, et al. 2017; Nguyen, Ofli, et al. 2017; Alam, Imran, et al. 2017).Source: RELATED WORK · p. 3
  • Earlier multimodal studies combined text, images, and contextual features or compared modalities for informative-tweet classification.Chen et al. reported an F1-score of 70.5%, 5.7% higher than text-only classification; Mouzannar et al. 2018 studied damage categories.Source: RELATED WORK · p. 3
  • Traditional multimodal fusion combines features early or classifier decisions late, while deep-learning approaches can combine hidden layers by concatenation (Nagrani et al. 2018).The study follows CNN-based architectures for unimodal and multimodal experiments.Source: RELATED WORK · p. 3
  • For this study, modality-specific text and image networks provide high-level features that are concatenated into a shared representation for classification.The image features come from VGG16; VGG16 is a 16-layer network with small 3×3 convolution filters.Source: RELATED WORK · pp. 3–4
  • Unlike Mouzannar et al. 2018, this study uses CrisisMMD rather than a home-grown dataset, and unlike Gautam et al. 2019, it covers two tasks with feature fusion rather than one with decision fusion.The authors caution that differences in data splits mean their results are not exactly comparable with Gautam et al.'s.Source: RELATED WORK · p. 4

DATASET

The study uses CrisisMMD tweet-image pairs from seven natural disasters in 2017 and focuses on informativeness and humanitarian categorization. It filters and restructures the data to address mismatched pair labels, sparse categories, and repeated tweet text across splits.

DATASET

  • CrisisMMD contains tweets and associated images from seven natural disasters in 2017, annotated for informativeness, eight humanitarian categories, and three damage-severity classes.The study excludes damage severity because that task was applied only to images, and focuses on the first two tasks.Source: DATASET · p. 4
  • Informativeness classification identifies whether disaster-event text or an image is useful for humanitarian aid.The humanitarian-categories task classifies the type of information in disaster-event text or an image.Source: DATASET · p. 4
  • The study keeps only text-image pairs with the same task label, then merges sparse, semantically related categories into five humanitarian classes.Injured or dead and missing or found people become “affected individuals”; vehicle damage is merged into “infrastructure and utility damage.”Source: DATASET · p. 4
  • To prevent duplicate tweet text across splits, tweets with multiple images are assigned only to training, while maintaining a 70%:15%:15% train-development-test split.The resulting training-set counts of tweet texts and images differ because of this assignment strategy.Source: DATASET · p. 4

EXPERIMENTS

The experiments compare text-only, image-only, and combined text-image models on informativeness and humanitarian classification. Performance is assessed using standard classification metrics, with the dataset categories and splits summarized in Table 1.

EXPERIMENTS

  • For each of the informativeness and humanitarian tasks, the experiments train text-only, image-only, and combined text-image models.The study uses the same three modality settings for both classification tasks.Source: EXPERIMENTS · p. 4
  • Model performance is measured with accuracy, precision, recall, and F1-score.The experiments section describes preprocessing, model architectures, and training details before reporting results.Source: EXPERIMENTS · p. 4
  • Table 1 lists the categories and data split for the different tasks.The table summarizes dataset organization rather than model-performance results.Source: Table 1 · p. 5

Data Preprocessing

Tweet text is cleaned of common noise, while image pixels are scaled and normalized using ImageNet statistics.

Data Preprocessing

  • Tweets are cleaned by removing stop words, non-ASCII characters, numbers, URLs, and hashtag signs.Punctuation marks are replaced with white spaces.Source: Data Preprocessing · p. 5
  • Punctuation marks in tweets are replaced with white spaces.Source: Data Preprocessing · p. 5
  • Image pixels are scaled to 0–1, then each channel is normalized with respect to ImageNet.Source: Data Preprocessing · p. 5

CNN: Text Modality

The text branch uses a five-hidden-layer CNN on word-level tweet representations, followed by convolutional and pooling operations to produce classification features.

CNN: Text Modality

  • The text classifier is a CNN with five hidden layers, chosen based on reported performance in crisis-related tweet classification.Source: CNN: Text Modality · p. 5
  • Tweets are zero-padded and converted to word-level matrices using 300-dimensional word2vec vectors trained on 364 million disaster-related tweets.The embeddings use CBOW with a context window of 5 and k = 5 negative samples.Source: CNN: Text Modality · p. 5
  • Convolution and max pooling produce fixed-size tweet features, which pass through fully connected layers to a softmax output.ReLU is used in convolutional and fully connected layers; the output layer uses softmax.Source: CNN: Text Modality · p. 5
  • Training uses Adam, a 0.01 learning rate, up to 50 epochs, 0.02 dropout, and development-set early stopping with patience 10.Filter counts are 100, 150, and 200 for window sizes 2, 3, and 4, respectively; batch normalization is also applied.Source: CNN: Text Modality · p. 5

VGG16: Image Modality

The image branch transfers ImageNet-pretrained VGG16 features to informativeness and humanitarian classification tasks.

VGG16: Image Modality

  • Image features come from VGG16 initialized with weights pretrained on ImageNet.The paper describes this as transfer learning from broad image classification to the disaster classification tasks.Source: VGG16: Image Modality · p. 5
  • The original 1,000-way softmax layer is adapted to the informativeness and humanitarian classification tasks.Source: VGG16: Image Modality · p. 5
  • Image models use Adam with an initial learning rate of 10^-6, reduced by a factor of 0.1 when development accuracy stops improving for 100 epochs.Training is capped at 1,000 epochs with early stopping.Source: VGG16: Image Modality · p. 6

Multimodal: Text and Image

The multimodal model combines text and image representations for disaster classification, and outperforms image-only models across both tasks.

Multimodal: Text and Image

  • Each modality has a 1,000-unit hidden layer before concatenation, followed by another hidden layer and a softmax output.The equal-sized modality layers are intended to give both modalities equal contribution; their size was not specifically optimized.Source: Multimodal: Text and Image · p. 6
  • The multimodal experiment uses Adam, minibatches of 32, early stopping, and ReLU, without tuning hyperparameters.The paper describes the basic network architectures as a baseline study, leaving room for improvement.Source: Multimodal: Text and Image · p. 6, Results · p. 6

Results

  • About 1% better than image-only in all informativeness measures and about 2% better in all humanitarian measures, multimodal models improve both tasks.Tables 2 and 3 present results for informativeness and humanitarian classification, respectively.Source: Results · p. 6, Table 2 · p. 7, Table 3 · p. 7
  • More than 2% average improvement in informativeness and more than 6% in humanitarian classification distinguish image-only from text-only models.Image-only models perform better than text-only models in both tasks.Source: Results · p. 6

DISCUSSION

Across both tasks, multimodal models reduce some false negatives, though the informativeness task shows no corresponding reduction in false positives. The discussion also notes a limited paired-label subset and category-specific errors.

DISCUSSION

  • Requiring matching text and image labels leaves a limited subset of CrisisMMD for multimodal training.The authors report comparable performance over unimodal models for both modalities under this condition.Source: DISCUSSION · p. 7
  • 155, 114, and 101 informative tweets were missed by text-only, image-only, and multimodal models, respectively.These are false negatives: the model labels a tweet not informative when human annotators label it informative.Source: DISCUSSION · p. 7, Table 4 · p. 7
  • False positives in the informativeness task numbered 139 for text-only, 145 for image-only, and 139 for multimodal models, showing no multimodal improvement.These are messages predicted informative despite a not-informative ground-truth label.Source: DISCUSSION · p. 7, Table 4 · p. 7
  • 41, 13, and 10 infrastructure-and-utility-damage instances were missed by text-only, image-only, and multimodal models, respectively.For rescue, volunteering, or donation effort, the corresponding false-negative counts were 37, 43, and 33; other relevant information remained confusing for all models.Source: DISCUSSION · pp. 7–8, Table 5 · p. 8
  • The text-and-image examples show joint predictions correcting some unimodal errors, including an infrastructure-damage label in Figure 3(e).In that example, the text-only model predicts other relevant information and the image-only model predicts not-humanitarian.Source: DISCUSSION · p. 8, Figure 3 · p. 8

Challenges and Future Work

The authors identify weak or absent alignment between co-occurring social-media text and images as a challenge for multimodal learning.

Challenges and Future Work

  • Social-media text and images are not guaranteed to be strongly aligned and may convey different or even contradictory information.The authors contrast this with tasks such as image captioning and visual question answering, where modalities are strongly coupled.Source: Challenges and Future Work · p. 9
  • Existing multimodal classification approaches assume a common label across modalities, while some tweet-image pairs have disagreeing labels.The authors propose training on heterogeneous inputs with disagreeing labels as a future direction.Source: Challenges and Future Work · p. 9
  • CrisisMMD in its full form could be used to train a multimodal algorithm on heterogeneous tweet-text and image pairs.This future direction addresses the current shared-label assumption.Source: Challenges and Future Work · p. 9

CONCLUSION

The conclusion reports feature-fusion multimodal models outperforming unimodal models on both CrisisMMD tasks, while noting that substantial room for improvement remains.

CONCLUSION

  • Feature-fusion multimodal models outperform unimodal models on both informativeness and humanitarian tasks in CrisisMMD experiments.The paper also provides baseline results for unimodal and multimodal approaches on both tasks.Source: CONCLUSION · p. 9
  • The study's two contributions are baseline results for both tasks and evidence that feature-fusion multimodal networks can outperform unimodal models.The authors describe this as underlining the importance of multimodal analysis of crisis-related social media data.Source: CONCLUSION · p. 9
  • The authors state that substantial room for improvement remains despite the multimodal classifiers' better performance.Source: CONCLUSION · p. 9
Loading 2004.11838v1…