Source-linked AI summary
Attend to You: Personalized Image Captioning with Context Sequence Memory Networks
Cesc Chunseong Park, Byeongchang Kim, Gunhee Kim
TL;DR
The paper addresses the lack of personalization in image captioning by generating image descriptions that use user-specific prior knowledge. It proposes CSMN for hashtag prediction and post generation, and reports improved performance over state-of-the-art captioning models in quantitative evaluations and AMT user studies. The study is conducted on Instagram data, with scope limited by the dataset’s single ground-truth text and hashtag list per test example.
Problem
Prior image-captioning research does not address generating descriptions that account for users’ active vocabularies, writing styles, and other prior knowledge.
Method
CSMN uses memory to store multiple context types and previously generated words, while CNNs jointly represent nearby ordered memory slots for personalized captioning.
Results
CSMN improves personalized captioning and outperforms other state-of-the-art captioning models in quantitative evaluations and AMT user studies.
Takeaways & Limitations
Personalized image captioning is demonstrated for hashtag prediction and post generation on Instagram posts.
Takeaways & Limitations
Each test example has only one ground-truth post text and hashtag list, so absolute metric values may be lower than benchmark-dataset values.
Abstract
from arXiv · showhide
We address personalization issues of image captioning, which have not been discussed yet in previous research. For a query image, we aim to generate a descriptive sentence, accounting for prior knowledge such as the user's active vocabularies in previous documents. As applications of personalized image captioning, we tackle two post automation tasks: hashtag prediction and post generation, on our newly collected Instagram dataset, consisting of 1.1M posts from 6.3K users. We propose a novel captioning model named Context Sequence Memory Network (CSMN). Its unique updates over previous memory network models include (i) exploiting memory as a repository for multiple types of context information, (ii) appending previously generated words into memory to capture long-term information without suffering from the vanishing gradient problem, and (iii) adopting CNN memory structure to jointly represent nearby ordered memory slots for better context understanding. With quantitative evaluation and user studies via Amazon Mechanical Turk, we show the effectiveness of the three novel features of CSMN and its performance enhancement for personalized image captioning over state-of-the-art captioning models.
1. Introduction
This work introduces personalized image captioning, which generates image descriptions using user-specific prior knowledge, and applies it to hashtag prediction and post generation. It proposes CSMN, a memory-based model whose context storage and sequence modeling improve personalization, evaluated on a large Instagram dataset.
- Motivation: Personalized image captioning generates image descriptions using prior knowledge such as users’ active vocabularies and writing styles.The paper addresses a personalization gap in prior image-captioning research.
- Applications: Hashtag prediction and post generation are formulated as two post-automation tasks for personalized captioning.The former predicts hashtags, while the latter generates text containing words, emojis, and hashtags.
- Model: CSMN uses memory as a repository for multiple context types, including users’ active vocabularies and image descriptors.This design is intended to support more focused prediction amid diverse topics and writing styles.
- Model: CSMN sequentially stores generated words so the model can attend to informative history without suffering from vanishing gradients.The memory retains prior words explicitly rather than relying only on recurrent hidden-state summaries.
- Model: CSMN applies CNNs to nearby ordered memory slots to represent correlations that time embeddings alone cannot capture.This is the model’s third stated update over previous memory-network designs.
- Evaluation: The evaluation uses 1.1M Instagram posts from 6.3K users and reports improvements over state-of-the-art captioning models using quantitative metrics and AMT studies.The dataset uses post body text or hashtag lists as ground-truth captions.
2. Related work
Prior image-captioning work largely uses RNN-based decoders, while personalization and practical post-completion applications remain less addressed. Related memory-network research provides mechanisms for storing information over long timescales.
- RNN-based decoders commonly generate word sequences from encoded image representations.
- The paper replaces an RNN-based decoder with a sequence memory and applies image captioning to post completion and hashtag prediction.
- Personalization research has considered user information for hashtag prediction, but not hashtag prediction as image captioning or post completion.
- Memory-network research enables neural networks to store variables and data over long timescales.
3. Dataset
The authors construct an Instagram dataset for post completion and hashtag prediction by collecting, filtering, and normalizing English image posts from diverse users. The resulting data are constrained by user and caption-length rules and use task-specific vocabularies.
- The dataset separates examples for post completion and hashtag prediction.
- 3,455,021 raw posts from 17,813 users are collected using 270 hashtag-search keywords spanning 27 Pinterest categories.
- Language filtering excludes posts with more than 20% non-English words, hyperlinks, or users exceeding the non-English and advertisement threshold.
- User and caption filtering limits users to 50–1,000 posts and removes captions or hashtag lists outside selected length ranges.
- Task-specific vocabularies contain the most frequent 40K words for post completion and 60K hashtags for hashtag prediction after text normalization.
4. The Context Sequence Memory Network
CSMN represents image, user, and generated-word context in memory, attends to that memory during sequence generation, and uses CNNs to fuse nearby heterogeneous memory cells. It then predicts words greedily and writes each generated word back into memory.
- CSMN uses one sequence-prediction model for both tasks, changing only the dictionary; hashtags are generated as ordered sequences rather than unordered bags.
- The memory stores image representations, TF-IDF-selected user vocabulary, and previously generated words as separate context types.
- ResNet-101 image features occupy either one pool5 memory cell or 49 res5c spatial cells for region-focused attention.
- At each step, attention matches the current input with memory cells, rescales output-memory representations, and forms attended image, user, and word memories.
- The model selects the highest-probability vocabulary word, repeats until EOS, and feeds each generated word into the word-output memory at the next step.
- Training uses teacher forcing and softmax cross-entropy over predictions at every time step.
5. Experiments
Experiments evaluate CSMN on post generation and hashtag prediction using quantitative metrics, user studies, ablations, and user-disjoint or post-disjoint splits. CSMN generally outperforms baselines, with personalization and its memory components contributing to performance.
- Experimental setup: The dataset is split by users to prevent retrieval of closely related posts from the same users in training and testing.A separate hashtag experiment also evaluates a split by posts, where each user appears in both sets.
- Experimental setup: The evaluation compares nearest-neighbor, language-only, nonpersonalized captioning, and CSMN variants on post generation and hashtag prediction.Metrics include BLEU, CIDEr, METEOR, ROUGE-L, and hashtag F1; user studies are also conducted through AMT.
- Quantitative results: CSMN-* significantly outperforms baselines on most metrics in both tasks, while CSMN-P5 achieves the best overall performance.CSMN-NoUC-P5 ranks first among nonpersonalized methods; pool5 features outperform 7×7 res5c features in this evaluation.
- Ablation results: Removing the memory CNN, user context, or generated-word feedback substantially worsens performance, supporting the contribution of each component.The best tested user-context memory size is 60; larger memories make attention learning harder.
- Split comparison: CSMN-P5 remains better than 1NN-UsrIm under the post-split setting, but its margin decreases from 0.121 to 0.01 in F1 score.The post split favors nearest-neighbor retrieval because training and test posts may have nearly identical hashtags.
- User studies: AMT annotators significantly prefer CSMN outputs over baseline outputs, consistent with the quantitative comparisons.Among baselines, 1NN-UsrIm is preferred most because its performance gap with CSMN is smallest.
6. Conclusions
The paper presents CSMN as a personalized image-captioning approach for hashtag prediction and post generation. Quantitative evaluation and AMT studies on Instagram data show that it outperforms other state-of-the-art captioning models, while future work broadens tasks and social-network coverage.
- Contributions: CSMN is proposed as a first personalized image-captioning approach addressing hashtag prediction and post generation.The model is evaluated on a newly collected Instagram dataset using quantitative measures and AMT user studies.
- Findings: The authors report that CSMN outperforms other state-of-the-art captioning models in quantitative evaluation and AMT studies.The conclusion identifies post commenting and additional social networks as future directions.
A. Why using memory CNN can be helpful?
The memory CNN represents relationships among nearby ordered memory slots rather than relying only on time embeddings. This supports sequence-sensitive representations for generated words and ordered user-context words.
- Memory ordering: Memory CNNs model structural ordering with stronger representation power than conventional memory networks using time embeddings.They represent correlations between multiple nearby memory slots.
- Memory ordering: For word-output memory, the CNN represents the sequential order of previously generated words.For user-context memory, it can capture the importance order of context words and their combinations.
B. Details of Dataset Collection
The dataset statistics guide vocabulary and user-context memory sizes for Instagram captions and hashtags. Captions are concentrated among frequent words, whereas hashtags are more diverse, and overly large context memories make learning harder.
- Collection procedure: The collection procedure uses 270 search keywords drawn from the ten most common hashtags across 27 Pinterest categories.The listed categories span topics including design, food, travel, technology, sports, pets, and fashion.
- Vocabulary statistics: The Instagram dataset uses vocabulary sizes of 40K for captions and 60K for hashtags based on word-frequency statistics.The top 40K caption words cover 97.3% of occurrences, while hashtag vocabulary is described as highly diverse.
- Context-memory sizing: User-context memory sizes are selected from active-vocabulary statistics because memories that are too small miss writing style, while oversized memories hinder learning.The selected sizes roughly correspond to the sum of mean and standard deviation values.
C. Model Details
The model details describe formulation changes for pool5 image features, including altered memory outputs, concatenation, and context dimensionality. Adding ReLU to Eq.(14) slightly improves performance.
- The formulation changes when using pool5 features modify the memory CNN output representation and related equations.The text contrasts these changes with the res5c-feature formulation.
- Adding ReLU to Eq.(14) slightly improves performance.
- The context vector dimension changes from 2,700 for res5c features to 1,800 for pool5 features.The dimensions are given as 3 × 3 × 300 and 2 × 3 × 300, respectively.
D. Training Details
Training samples are grouped into minibatches containing sentences of similar lengths, then batch order is shuffled to mix short and long minibatches.
- Training samples are randomly grouped into minibatches whose sentences have the same length as possible.Although the model accepts variable-length sequences, this grouping speeds up training.
- The minibatch order is randomly shuffled so short and long minibatches are mixed.The authors report empirically better training with this curriculum-learning-style arrangement.