Source-linked AI summary
No Metrics Are Perfect: Adversarial Reward Learning for Visual Storytelling
Xin Wang, Wenhu Chen, Yuan-Fang Wang, William Yang Wang
TL;DR
Visual storytelling requires expressive, subjective narratives that connect photo streams with emotions and concepts not explicit in the images, while standard metrics and hand-crafted rewards are limited. AREL learns an implicit reward from human demonstrations and optimizes policy search with it; automatic gains are slight, but human evaluation reports significant improvement in producing more human-like stories.
Problem
Visual storytelling remains challenging because stories are expressive, subjective, and include imaginary concepts, while hand-crafted automatic rewards can be biased or sparse.
Method
AREL learns an implicit reward function from human demonstrations and uses the learned reward to optimize the policy for story generation.
Results
Human evaluation reports significant improvement in generating more human-like stories, while automatic evaluation shows only slight performance gains over baseline systems.
Takeaways & Limitations
AREL improves human-rated relevance, expressiveness, and concreteness and motivates analyzing the limitations of automatic story metrics.
Takeaways & Limitations
Automatic metrics correlate poorly with human evaluation and may either be gamed or assign nearly zero scores to relevant, coherent stories.
Abstract
from arXiv · showhide
Though impressive results have been achieved in visual captioning, the task of generating abstract stories from photo streams is still a little-tapped problem. Different from captions, stories have more expressive language styles and contain many imaginary concepts that do not appear in the images. Thus it poses challenges to behavioral cloning algorithms. Furthermore, due to the limitations of automatic metrics on evaluating story quality, reinforcement learning methods with hand-crafted rewards also face difficulties in gaining an overall performance boost. Therefore, we propose an Adversarial REward Learning (AREL) framework to learn an implicit reward function from human demonstrations, and then optimize policy search with the learned reward function. Though automatic eval- uation indicates slight performance boost over state-of-the-art (SOTA) methods in cloning expert behaviors, human evaluation shows that our approach achieves significant improvement in generating more human-like stories than SOTA systems.
1 Introduction
Visual storytelling extends captioning by requiring expressive, subjective narratives that connect photo streams with emotions and concepts not explicit in the images. AREL addresses biased reward metrics by learning an adversarial reward from human demonstrations; automatic gains are slight, while human evaluation reports stronger improvements in human-like storytelling.
- Motivation: Visual storytelling summarizes photo streams into subjective narratives containing emotions, imagination, and concepts absent from the images.Unlike captioning, it has few standard templates, and one photo stream can support diverse stories.
- Motivation: Behavioral-cloning methods tend to generate simple descriptions, while hand-crafted string-match rewards can be biased or sparse for story generation.The paper reports metric-gaming behavior under METEOR and very low scores for some relevant, coherent stories under BLEU and CIDEr.
- Method: AREL learns an implicit reward function from human demonstrations and uses it to optimize the policy for visual story generation.The framework is proposed to reduce bias from hand-coded evaluation metrics and produce more human-like stories.
- Results: Automatic metrics show slight gains over baseline systems, whereas human evaluation indicates significant improvement in human-like story generation.The paper therefore examines the mismatch between automatic and human evaluation.
- Results: On the VIST dataset, the approach achieves state-of-the-art results on automatic metrics and improves relevance, expressiveness, and concreteness in human evaluation.These claims are presented as contributions of the AREL framework.
2 Related Work
Prior visual-storytelling systems build on captioning, behavioral cloning, reinforcement learning, and adversarial methods, while automatic metrics remain widely used despite poor correlation with human judgments. AREL is situated within inverse reinforcement learning as a way to acquire rewards automatically.
- Visual Storytelling: Visual storytelling generates narrative stories from photo streams and requires understanding the event flow across ordered images.Earlier methods include storyline generation and multi-task learning, but remain based on behavioral cloning.
- Reinforcement Learning in Sequence Generation: Reinforcement learning treats word generation as sequential actions and optimizes a policy by maximizing expected return.The related work contrasts this with maximum-likelihood training, which is associated with exposure and label bias.
- Rethinking Automatic Metrics: BLEU, CIDEr, METEOR, and ROUGE enable rapid evaluation but have been criticized for bias and poor correlation with human judgments.The criticism is reported across several generative tasks.
- Generative Adversarial Network: GANs estimate difficult probabilities through alternating optimization of a generator and discriminator in a min-max game.For discrete problems, generator updates can use Monte Carlo policy-gradient estimation.
- Inverse Reinforcement Learning: Inverse reinforcement learning infers an expert reward function, reducing reliance on extensive feature and reward engineering.Adversarial inverse reinforcement learning is presented as an efficient and scalable route to automatic reward acquisition.
3 Our Approach
AREL models visual storytelling with a policy that generates stories from image streams and a reward model learned adversarially from human and generated stories. The framework alternates reward and policy optimization so the learned reward guides story generation.
- Problem Statement: AREL formulates visual storytelling as generating a word sequence from five ordered images using a policy model and a reward model.The policy selects words sequentially, while the reward model evaluates generated stories.
- Policy Model: The policy encodes image features with a bidirectional GRU and uses RNN decoders to generate five sentences in parallel before concatenating them into one story.The five decoders share weights.
- Reward Model: The reward model uses CNN-extracted n-gram features, combines sentence and image representations, and computes fine-grained partial rewards for individual sub-stories.Partial rewards are intended to provide better guidance than one overall story score.
- Adversarial Reward Learning: AREL defines a reward-induced Boltzmann distribution and trains the reward to distinguish human-annotated stories from policy-generated stories.The empirical training distribution supplies positive examples for reward learning.
- Adversarial Reward Learning: The policy and reward models are optimized alternately: the reward favors empirical stories over generated ones, while the policy maximizes similarity to the reward-induced distribution.Negative sampling bypasses the intractable partition function, and policy training uses a variance-reducing baseline.
- Training and Testing: During testing, the policy generates stories with beam search.The training procedure alternates stochastic-gradient updates for the two models.
4 Experiments and Analysis
Experiments on VIST compare AREL with state-of-the-art and reinforcement-learning baselines using automatic metrics and human studies. AREL achieves the best automatic results, while human evaluation reveals substantially stronger human-like story quality and exposes weaknesses in hand-crafted metrics.
- Automatic Evaluation: AREL achieves new state-of-the-art results across all automatic metrics on the VIST dataset.The comparison includes prior methods, XE-ss, GAN, and AREL variants.
- Limitations of Automatic Metrics: Automatic metrics are imperfect rewards because optimizing individual metrics can produce nonsense or damage other metrics.METEOR-RL and ROUGE-RL maximize their target metrics while losing coherence and semantic correctness; a nonsense example scores 33.8 on ROUGE-L.
- Limitations of Automatic Metrics: BLEU-RL and CIDEr-RL do not consistently improve over XE-ss, and many nearly zero-scoring stories remain relevant and coherent.This illustrates that low string-match scores do not necessarily indicate poor story quality.
- Limitations of Automatic Metrics: BLEU-3 and CIDEr correlate poorly with human evaluation because exact n-gram matching and reference similarity do not fully capture story quality.AREL instead learns a reward function from human-annotated stories to provide more robust policy guidance.
- Human Evaluation: In the Turing test, AREL fools 45.8% of workers, compared with 28.3% for XE-ss, 32.1% for BLEU-RL, 19.7% for CIDEr-RL, and 39.5% for GAN.The authors interpret this as evidence that AREL generates more human-like stories than the baselines.
- Human Evaluation: Pairwise human comparisons favor AREL over competing systems in relevance, expressiveness, and concreteness, despite these gains being weakly reflected by automatic metrics.A qualitative comparison also reports greater grammatical and semantic correctness, coherence, and photo-stream accuracy than XE-ss.
5 Conclusion
The paper introduces adversarial reward learning for generating more human-like stories from image sequences and analyzes limitations of automatic story-evaluation metrics.
- AREL generates more human-like stories from image sequences using adversarial reward learning.The paper also empirically analyzes limitations of automatic metrics for story evaluation.
A Error Analysis
Error analysis identifies failures in human-like storytelling and dataset bias. AREL can lack emotion and imagination, while VIST contains gender and event imbalances that models may magnify.
- Failure Case in Turing Test: 4 out of 5 workers correctly identified a human-created story in a failed Turing-test example.The AREL story lacked emotion and imagination compared with the human-generated story.
- Data Bias: VIST exhibits severe gender and event biases, and models magnify the influence of majority patterns.The reported gender ratio changes from 2.06:1 in training and 2.16:1 in testing to 3.44:1 in model outputs.
B Training Details
The implementation uses separate policy and reward models for sequential story generation and alternately optimizes them after schedule-sampling pretraining.
- The policy model uses tied multiple-RNN decoders, with each RNN generating a sub-story for one photo.Image features come from a pretrained ResNet-152 visual encoder.
- The visual encoder is a bidirectional GRU with 256 hidden units per direction, producing a 512-dimensional story-generator input.Five album images are supplied separately to five RNN decoders.
- The decoder is a single-layer GRU with hidden dimension 512 and uses scheduled sampling during training.At each step, it receives either the last predicted token or the ground-truth token.
- The reward model uses CNN-extracted n-gram features and projects them into a single predicted reward value.It uses kernel windows of 2, 3, and 4 with 128-dimensional input embeddings and 128 filter dimensions.
- Training alternates policy and reward-model optimization with Adam at learning rate 2e-4 after 5–10 hours of schedule-sampling warm-up.Beam search with beam size 3 approximates the search space during testing.
C Amazon Mechanical Turk
Human evaluation uses Amazon Mechanical Turk surveys to compare stories for human-likeness, matching, coherence, concreteness, relevance, and expressiveness.
- Workers select which story is more likely to have been generated by a human.The survey includes a Turing-style comparison form.
- Head-to-head comparisons evaluate stories for matching, coherence, and concreteness.The comparison survey is documented with a dedicated form.
- Relevance measures whether a story describes the image stream and covers its main objects.The criterion emphasizes accurate coverage of events and objects.
- Expressiveness includes coherence, grammatical and semantic correctness, non-repetition, and expressive language style.The survey materials also distinguish concrete stories from repetitive or overly abstract examples.
- The survey presents image streams followed by alternative stories and asks workers which story better describes the images, is more coherent, or is more concrete.These questions operationalize the principal human-evaluation dimensions.