PaperlayerView exact revision on arXiv

Source-linked AI summary

VIS-DICT: A Visual Dictionary for Missing Modality Imputation in Social Network Depression Detection

Hamed Marvi, Mohammad Mehdi Keikha, Abolfazl Nadi

arXiv:2609.05537v1cs.CV

TL;DR

Depression detection from social-media posts can benefit from multimodal information, but missing images complicate its use. Vis-Dict links words to visual embeddings for user-timeline analysis; in sequential experiments it ties a generative network while using no trainable parameters for image generation.

  • Problem

    Social-media posts may lack images or other modalities, while multimodal models often assume complete inputs, making missing-modality handling a challenge.

    Source: §2.2 · p. 3

  • Method

    Vis-Dict builds a TF-IDF visual dictionary by associating words with image vectors from complete posts, then combines estimated visual features with text in user-timeline analysis.

    Source: §1 · p. 2, §3.2.3 · p. 8, §5 · p. 22

  • Results

    0.9454 ± 0.0128 F1-score and 0.9890 ± 0.0009 ROC-AUC; Vis-Dict statistically tied T2V’s F1 = 0.9468 ± 0.0048 in user-level sequential experiments.

    Source: §4.4 · p. 21

  • Takeaways & Limitations

    In sequential experiments, combining text and image information across a user’s full timeline was essential for accurate classification, and Vis-Dict achieved competitive performance with zero trainable parameters for image generation.

    Source: §5 · p. 22

    Vis-Dict depends on word-image pairs and can produce uninformative visual features for short posts with sparse or neutral vocabulary.

    Source: §4.2.2 · p. 14, §5 · p. 22

Abstract

from arXiv · show

Tracking social media posts can help spot early signs of depression. Recent studies show that combining text and images works better for detecting depression than using text alone. However, many social media posts do not have images, which makes it hard to use multimodal models. Most existing methods fill in missing images using retrieval or generative models that need extra training. In this paper, we introduce Vis-Dict, a dictionary-based method that builds missing visual features by linking words to average image vectors from complete training posts. These estimated visual features are then combined with text to track changes in user behavior over time. We tested Vis-Dict on a social media dataset using user timelines of up to 512 posts and compared it with other missing-data methods. The results show that Vis-Dict performs on par with generative networks, reaching an F1-score of 0.9454 and an ROC-AUC of 0.9890. Most importantly, Vis-Dict achieves this strong performance with zero trainable parameters for image generation. These findings show that directly connecting words to visual features is an effective and practical way to handle missing images in depression detection systems.

1. Introduction

The introduction motivates tracking social-media behavior for earlier depression detection and identifies missing images as a challenge for multimodal models. It proposes Vis-Dict to connect words directly with visual embeddings and evaluates it for missing-visual-modality depression detection.

1. Introduction

  • Traditional assessment can delay early detection when people do not recognize symptoms, mistake them for temporary stress or fatigue, or avoid care because of stigma.Clinical assessments also depend on individuals being willing to describe their experiences honestly.Source: §1 · pp. 1–2
  • Behavioral changes can emerge gradually, while short clinical meetings may not reveal the long-term patterns useful for spotting early signs.The cited passage describes changes in speech, emotional expression, habits, and activity levels.Source: §1 · p. 2
  • Social-media posts offer naturally shared, regular records of daily experiences that researchers can track over time, although they do not capture a person’s entire mental-health situation.The passage contrasts these posts with the formal constraints of clinical interviews.Source: §1 · p. 2
  • Combining text and images can capture complementary behavioral cues and improve depression detection over text alone.Social-media platforms increasingly include images, audio, and video alongside text.Source: §1 · p. 2
  • Missing modalities vary across posts, making multimodal fusion challenging when models expect every required modality to be available.Posts may contain text without images, images without captions, or videos without accompanying text.Source: §1 · p. 2
  • Vis-Dict directly associates words with visual embeddings, addressing a gap in general recovery methods that do not maintain word–visual correspondence.The introduction reports that the method is evaluated on a social-media depression dataset against existing missing-modality methods.Source: §1 · p. 2

2. Related Work

Prior work covers multimodal depression detection and general strategies for handling incomplete modalities. Those strategies include simple imputation, retrieval or composition, generation, and representation recovery, while the review identifies room for depression-specific approaches.

2.1. Multimodal Depression Detection.

  • Multimodal depression models combine text and visual information, with newer transformer- and attention-based approaches modeling their relationships and outperforming single-modality models.Images can contribute information such as color, brightness, and photo composition that is unavailable from text alone.Source: §2.1 · p. 3

2.2. The Challenge of Missing Data.

  • Multimodal models often assume all modalities are available at training and inference, but incomplete social-media posts can prevent models designed for complete inputs from processing them directly.Missing images, text, or other modalities make handling incomplete data an important research problem.Source: §2.2 · p. 3
  • Simple imputation replaces missing modalities with values such as zeros, random vectors, or learned embeddings, enabling processing but not recovering the missing modality’s semantic information.The review also describes statistical zero, mean, and median imputation.Source: §2.2 · p. 3
  • Retrieval and composition methods reuse information from other samples, but retrieval can produce inaccurate recovery when suitable, relevant samples are unavailable.The examples include Modal-Mixup, which combines modalities from incomplete samples of the same class, and MISSRAG, which retrieves similar samples.Source: §2.2 · pp. 3–4
  • Generation-based methods reconstruct missing data and can offer flexible reconstructions, but are generally more computationally expensive and need more training data than retrieval-based methods.Representation-recovery methods instead estimate compact missing-modality embeddings; examples include SMIL and TRML.Source: §2.2 · p. 4
  • Existing retrieval, generation, and representation-recovery methods are general-purpose rather than designed specifically for depression detection.The review identifies this as room for task-specific approaches.Source: §2.2 · p. 4

3. Proposed Method

The framework targets missing-image depression detection by extracting tweet text and image features, testing imputation strategies, and modeling user timelines over time.

3.1. Feature Extraction and Modality Encoders.

  • XLM-RoBERTa extracts text features and CLIP ViT extracts image features; the visual encoder can be trained independently or jointly with text alignment.Joint training uses a multi-task loss balancing classification and alignment objectives.Source: §3.1 · pp. 5–6

3.2. Addressing the Missing Modality Challenge.

  • Because social-media posts may lack images, the method tests five imputation strategies that preserve a consistent input size across posts.The strategies range from simple replacements to context-aware estimates based on text.Source: §3.2 · p. 7

3.2.1. Baseline Imputation.

  • The baseline methods replace missing image features with either zeros or a globally learned placeholder vector.Zero-filling treats the missing modality as conveying no information, while the learned vector is optimized with the network.Source: §3.2.1 · p. 7

3.2.2. Context Aware Baselines.

  • Context-aware baselines retrieve image vectors for sufficiently similar posts or train a text-to-visual network to predict image embeddings.MissRAG averages the top five matches when cosine similarity is at least 0.5; otherwise it uses the learned placeholder.Source: §3.2.2 · pp. 7–8

3. Proposed Method

  • Vis-Dict estimates missing visual features from word-level visual cues, then combines them with text features for tweet classification and user-level timeline modeling.The framework also compares five strategies for filling missing image representations.Source: §3 · p. 4, Figure 1 · p. 5, §3.2.2 · p. 8

3.2.3. Proposed TF-IDF Visual Dictionary - Vis-Dict.

  • Vis-Dict averages image vectors from complete training posts for each dictionary word, then combines matching word vectors using normalized TF-IDF weights.Words appearing fewer than five times are filtered, and posts without dictionary words use a constant learned vector; imputation adds no trainable parameters.Source: §3.2.2 · p. 8, §3.2.3 · pp. 8–10

3.3. Multimodal Data Fusion and Classification.

  • The system projects text and image representations into 512 dimensions, concatenates them into a 1024-dimensional vector, and classifies each tweet.The same fusion path is used whether the image representation is real or imputed.Source: §3.3 · p. 10

3.4. User-Level Detection.

  • User-level detection compares majority voting and probability averaging with a Sequential Transformer that processes timelines of up to 512 tweets in chronological order.The Transformer uses self-attention and masked mean pooling over non-padding tweets.Source: §3.4 · pp. 10–11

4. Experiments and Results

The experiments evaluate Vis-Dict on a depression-detection dataset, first at the tweet level and then with full user timelines.

4. Experiments and Results

  • The study describes its dataset and experimental setup before evaluating the models.The experiments are framed around addressing missing images.Source: §4 · p. 11
  • Tweet-level tests examine text and image features separately, then assess their combination and estimated images.Source: §4 · p. 11
  • The full timeline model is tested at the user level to compare missing-image strategies.Source: §4 · p. 11

4.1. Dataset and Experimental Setup.

The setup balances user cohorts and timeline volumes, reflects substantial image sparsity, and specifies training and modeling procedures for the visual dictionary and timeline classifier.

4.1. Dataset and Experimental Setup.

  • Depressed users were labeled using diagnosis self-reports, and the final anchor tweet was removed before feature extraction and training to prevent leakage.Source: §4.1 · p. 11
  • Posts inherit their user's depression label, creating unavoidable noise in single-tweet tests because users may post neutral or happy messages.The passage contrasts this with timeline modeling, which can learn behavioral patterns over time.Source: §4.1 · p. 11

4.1.1. Dataset Statistics and Modality Sparsity.

  • The dataset contains 3,504 users, balanced between 1,752 Normal and 1,752 Depressed users, with user-level train, validation, and test splits.The split is 80%, 10%, and 10%; timeline matching balances tweet volume across classes.Source: §4.1 · pp. 11–12, §4.1.1 · p. 12
  • About 90.4% of posts contain text only, motivating imputation for timelines with sparse visual data.Users contribute an average of 237 posts.Source: §4.1.1 · p. 12

4.1.2. Experimental Setup and Implementation Details.

  • Vis-Dict averages word-level visual vectors with TF-IDF weights, using complete training posts and words occurring at least five times.The compared alternatives include a learned vector, MissRAG retrieval, and a T2V network.Source: §4.1.2 · p. 13
  • The user-level Temporal Transformer processes up to 512 chronologically ordered tweets and uses masked mean pooling to ignore padding.Its specified setup includes two encoder layers and eight attention heads.Source: §4.1.2 · p. 13, Table 2 · p. 13

4.2. Tweet-Level Evaluation.

Tweet-level evaluation finds that genuine images offer useful cues, while text remains the primary signal and imputation is close to simple baselines on isolated posts. Feature-diversity and error analyses characterize the limits and behavior of the imputed representations.

4.2.1. Single-Modality Baselines.

  • The text baseline achieves Accuracy 0.7633, ROC-AUC 0.8578, and Depressed F1-score 0.7598; visual F1 falls from 0.7911 Unjoint to 0.7206 Joint.The Joint visual model's Accuracy is 0.7937±0.0280 and ROC-AUC is 0.8707 ± 0.0041; Unjoint values are 0.8676 ± 0.0165 and 0.9361 ± 0.0048, respectively.Source: §4.2.1 · p. 14, Table 3 · p. 14

4.2.2. Multimodal Classification with Missing Images.

  • On posts with genuine images, models reach F1 ≈0.81–0.85 and ROC-AUC ≈0.90–0.93; imputation scores are close to zero-filling and learned-vector baselines.For single tweets, text remains the primary signal; the table reports Depressed F1 and ROC-AUC across subsets.Source: §4.2.2 · p. 14, Table 4 · p. 15

4.2.3. Evaluation of Imputed Visual Representations and Mode Collapse.

  • Vis-Dict variance is 0.0374 Unjoint and 0.1387 Joint, below T2V at 0.4288, MissRAG Joint at 0.4835, and real images at 0.6842.T2V embeddings also have average cosine similarity 0.8773 with corresponding text embeddings; Vis-Dict's weighted word vectors can pull sparse or neutral posts toward a neutral centroid.Source: §4.2.3 · p. 15, Table 5 · p. 15
  • Low single-post variance is described as a regularized prior, with timeline aggregation across T = 512 posts canceling post-level noise and capturing consistent behavioral signals.Source: §4.2.3 · pp. 15–16

4.2.4. Modality Agreement and Error Analysis.

  • 22,037 correct text predictions preserved accurate fused classifications despite wrong visual predictions, while text-visual disagreements remained the primary contributor to single-tweet errors.Visual cues corrected 1,467 text errors, and fusion corrected 144 cases where both unimodal predictions missed; errors included 10,623 text-led and 1,078 vision-led misclassifications.Source: §4.2.4 · p. 16, Table 6 · p. 16

4.3. User-Level Depression Detection.

User-level depression detection evaluates whether aggregation across timelines of up to 512 tweets captures behavioral patterns better than isolated-post classification. The section compares voting, probability averaging, and a sequential Transformer.

4.3. User-Level Depression Detection.

  • Majority voting labels a user by the proportion of timeline posts classified as depressed; the evaluated threshold range was 0.3–0.7.The section evaluates user histories of up to 512 tweets and uses 0.5 as the standard threshold for threshold-based methods.Source: §4.3 · p. 16, §4.3.1 · p. 17

4.3.1. Majority Voting.

  • At θ = 0.6, majority voting raised imputation-method F1-scores to around 0.89–0.90 and ROC-AUC scores to around 0.89–0.91.Vis-Dict reached F1-score 0.8929 (ROC-AUC: 0.8977) unjoint and 0.8882 (ROC-AUC: 0.8920) joint.Source: §4.3.1 · p. 17, Figure 6 · p. 17, Table 7 · p. 18

4.3.2. Softmax Probability Averaging.

  • Softmax probability averaging uses the mean depressed-class probability across a user’s posts, with its best F1-score at θ = 0.5.This strategy retains continuous confidence scores rather than making a hard post-level decision.Source: §4.3.2 · p. 18, Figure 7 · p. 18
  • Vis-Dict achieved F1-scores of 0.9086 unjoint and 0.9061 joint under softmax averaging, performing competitively with T2V at 0.9056 / 0.9034.Across configurations, softmax averaging yielded F1-scores of 0.90–0.92 and ROC-AUC scores of 0.90–0.92.Source: §4.3.2 · pp. 18–19, Table 8 · p. 19

4.3.3. Sequential Transformer.

  • Vis-Dict (Unjoint) reached an F1-score of 0.9454 ± 0.0128 and ROC-AUC of 0.9890 ± 0.0009, statistically tying T2V in F1 with zero trainable imputation parameters.The text-only baseline reached F1-score 0.9359±0.0127 and ROC-AUC 0.9869±0.0005.Source: §4.3.3 · p. 19, Table 9 · p. 20
  • Without positional encoding, Vis-Dict remained stable at F1 = 0.9416 and ROC-AUC = 0.9897, while T2V and MissRAG F1 scores fell from 0.9468 to 0.9024 and 0.9408 to 0.9249.The ablation removes tweet order; the authors attribute Vis-Dict’s stability to accumulating vocabulary across the timeline.Source: §4.3.3 · pp. 20–21, Table 10 · p. 20
  • On missing-image users, Vis-Dict (Unjoint) recorded an F1-score of 0.9434 and ROC-AUC of 0.9839, while unjoint training scored 0.9062 versus 0.8684 for joint training on full-modality users.The full-modality subset contains genuine text and images; the missing-modality subset requires image imputation.Source: §4.3.3 · p. 21, Table 11 · p. 21

4.4. Discussion.

The discussion reports that text is the main single-post signal, while estimated visual features can correct some text-model errors. Across whole user timelines, Vis-Dict performs competitively without trainable image-generation parameters.

4.4. Discussion.

  • For single tweets, text is the main depression signal, while images provide helpful extra context.Source: §4.4 · p. 21
  • In 1,467 cases, estimated image features corrected mistakes made by the text model, showing that visual information can help when text is unclear.Source: §4.4 · p. 21
  • Vis-Dict reached F1-score 0.9454 ± 0.0128 and ROC-AUC 0.9890 ± 0.0009 in sequential user-level experiments, statistically tying T2V in F1 without trainable image-generation parameters.All missing-image methods achieved strong results with the Sequential Transformer.Source: §4.4 · p. 21

5. Conclusion

The paper proposes TF-IDF Visual Dictionary (Vis-Dict) for missing images in social-media depression detection, evaluating it at single-tweet and user levels. Vis-Dict ties a generative network in sequential user-level results and has a stated limitation for short posts without clear keywords.

5. Conclusion

  • The paper proposes a TF-IDF Visual Dictionary and evaluates missing-image methods on single tweets and user timelines using a Sequential Transformer.Source: §5 · p. 22
  • Vis-Dict (Unjoint) reached ROC-AUC 0.9890 and F1-score 0.9454 in sequential experiments, statistically tying the generative T2V network with zero trainable image-generation parameters.Source: §5 · p. 22
  • Vis-Dict can struggle to create useful visual features for short posts that lack clear keywords because it depends on word-image pairs.Source: §5 · p. 22

Related papers

Loading 2609.05537v1…