Source-linked AI summary
Hierarchy Parsing for Image Captioning
Ting Yao, Yingwei Pan, Yehao Li, Tao Mei
TL;DR
Neural captioning models often produce plausible captions without adequately modeling compositional visual structure. The paper introduces HIP, which builds a three-level hierarchy of instances, regions, and the whole image and refines its features with Tree-LSTM. On COCO, HIP improves captioning performance, reaching 130.6% CIDEr-D when combined with GCN-based semantic relations.
Problem
Neural captioning models tend to exploit training-data correlations and produce generic plausible captions while lacking visual understanding of compositional image patterns.
Method
HIP constructs a three-level hierarchy from the whole image to regions and instances, then uses Tree-LSTM to enhance features across all levels for captioning.
Results
130.6% CIDEr-D is achieved on COCO when HIP features are further enriched with semantic relations through GCN.
Takeaways & Limitations
HIP provides a pluggable image-encoder feature refiner that supports hierarchical representations in neural captioning models.
Abstract
from arXiv · showhide
It is always well believed that parsing an image into constituent visual patterns would be helpful for understanding and representing an image. Nevertheless, there has not been evidence in support of the idea on describing an image with a natural-language utterance. In this paper, we introduce a new design to model a hierarchy from instance level (segmentation), region level (detection) to the whole image to delve into a thorough image understanding for captioning. Specifically, we present a HIerarchy Parsing (HIP) architecture that novelly integrates hierarchical structure into image encoder. Technically, an image decomposes into a set of regions and some of the regions are resolved into finer ones. Each region then regresses to an instance, i.e., foreground of the region. Such process naturally builds a hierarchal tree. A tree-structured Long Short-Term Memory (Tree-LSTM) network is then employed to interpret the hierarchal structure and enhance all the instance-level, region-level and image-level features. Our HIP is appealing in view that it is pluggable to any neural captioning models. Extensive experiments on COCO image captioning dataset demonstrate the superiority of HIP. More remarkably, HIP plus a top-down attention-based LSTM decoder increases CIDEr-D performance from 120.1% to 127.2% on COCO Karpathy test split. When further endowing instance-level and region-level features from HIP with semantic relation learnt through Graph Convolutional Networks (GCN), CIDEr-D is boosted up to 130.6%.
1. Introduction
Existing neural captioning models can generate plausible captions but lack visual understanding of compositional image patterns. HIP addresses this by parsing images into hierarchical regions and instances, refining multilevel features for caption generation.
- Image captioning generates natural-language sentences reflecting an image's visual content and has potential applications in robotic vision and assistance for visually impaired people.
- Current neural captioning models often exploit training-data correlations to produce generic plausible captions while lacking visual understanding of compositional patterns.
- HIP parses an image from the whole image to regions and instances, whose foregrounds emphasize discriminative visual content.
- A bottom-up Tree-LSTM enhances image-level, region-level, and instance-level features across the hierarchy.
- HIP outputs rich multilevel representations that can be jointly or separately used and can incorporate semantic relations between regions or instances.
- HIP is designed as a feature refiner that integrates hierarchical structure into typical neural captioning frameworks and is pluggable into any neural captioning model.
2. Related Work
Prior image-captioning work uses sequence models, attention, object-level features, and object relations, while structured scene parsing studies hierarchical visual representations. HIP applies hierarchical parsing as an image encoder for captioning, enhancing features across instance, region, and image levels.
- Image Captioning: Image-captioning methods mainly use CNN-plus-RNN sequence learning to generate sentences word by word, with later work adding attention and object relations.
- HIP: HIP exploits hierarchy from instances through regions to the whole image and is designed to plug into any neural captioning model.
- Structured Scene Parsing: Structured scene parsing seeks deeper, structured scene understanding beyond classification, localization, semantic labeling, or segmentation.
- Structured Scene Parsing: HIP differs from prior parsing and semantic-labeling methods by acting as an image encoder for captioning.
- HIP: Its tree-structured processing enhances instance-level, region-level, and image-level features before they are injected into caption generation.
3. Our Approach
HIP parses an image into a three-level hierarchy of instances, regions, and the whole image, then uses Tree-LSTM to refine representations across this structure. The resulting feature refiner is designed to integrate with neural captioning and other vision tasks.
- Hierarchy representation: HIP represents an image with a three-level hierarchy whose root is the whole image, middle nodes are regions, and leaves are instances.The region and instance sets are extracted from the image and connected into a hierarchical tree.
- Integration and extension: HIP acts as a feature refiner that can plug into neural captioning models and also provide enhanced image-level features for recognition tasks.The architecture can additionally be combined with semantic relations on region- and instance-level features.
- Regions and instances: Faster R-CNN detects the top K = 36 object regions, while Mask R-CNN supplies foreground masks used to derive instance features.Regions are represented by 2,048-dimensional RoI-pooled features, and foreground regions are processed by another Faster R-CNN.
- Hierarchy construction: Regions are organized by descending area and Intersection over Union, with sufficiently overlapping regions attached as finer children and remaining regions attached to the image root.Each detected instance is then attached to its corresponding region node as a leaf.
- Tree-LSTM encoder: Tree-LSTM processes the hierarchy bottom-up, using multiple child hidden states and child-specific forget gates to contextually enhance instance-, region-, and image-level features.This structure captures dependencies that a chain-structured LSTM cannot represent as fully.
4. Experiments
Experiments on COCO evaluate HIP across dataset settings, competing methods, online testing, ablations, qualitative examples, and human evaluation. Results consistently support hierarchical parsing for improving image captioning, with stronger gains when HIP is combined with relation-aware features and CIDEr-D optimization.
- Datasets and Settings: COCO experiments use the Karpathy split, Visual Genome for object detection, COCO-detect for instance segmentation, and specified training hyperparameters.The Karpathy split contains 113,287 training, 5,000 validation, and 5,000 test images; the final vocabulary contains 10,201 words.
- Performance Comparison: GCN-LSTM+HIP consistently outperforms competing attention and non-attention captioning methods across metrics and optimization methods.The comparison includes LSTM, LSTM-A, SCST, ADP-ATT, RFNet, Up-Down, and GCN-LSTM.
- Performance Comparison: 3.2% is the CIDEr-D improvement of GCN-LSTM+HIP over GCN-LSTM with cross-entropy optimization, while CIDEr-D optimization raises GCN-LSTM+HIP to 130.6%.Up-Down+HIP also improves over Up-Down by 3.1% in CIDEr-D under cross-entropy loss.
- Ablation Study: Integrating region, instance, and Tree-LSTM features achieves the highest ablation performance, indicating complementarity among the three feature types.The ablation reports that finer Tree-LSTM features outperform individual original feature choices, while concatenating feature pairs consistently improves results.
- Qualitative Analysis: Qualitative examples show GCN-LSTM+HIP producing more descriptive captions, including “two zebras” instead of the less precise “a group of zebras.”The authors speculate that segmenting the two zebra instances and integrating them into the hierarchy supports this improvement.
- Online Testing: The GCN-LSTM+HIP run optimized with CIDEr-D achieves a performance boost over other published top-performing systems on the online COCO testing server.The submission uses SENet-154 as the Faster R-CNN and Mask R-CNN backbone.
(a) METEOR (b) CIDEr-D
The experiments examine hierarchy construction and recognition generalization, alongside human-oriented caption evaluation. Performance is best at threshold ϵ=0.1, while HIP-derived features improve recognition metrics.
- 76.5%, 73.9%, and 50.7% are the M1 scores for GCN-LSTM+HIP, GCN-LSTM, and LSTM, respectively.
- ϵ=0.1 achieves the best performance for both METEOR and CIDEr-D as the curves vary from 0.05 to 0.5.Both curves generally form an ∧ shape across the tested threshold range.
- 2% and 3.2% performance gains in C-F1 and O-F1 over Up-Down show HIP image features generalize to recognition.The recognition experiment uses COCO’s 80 object categories and compares HIP features with mean-pooled Up-Down features.
5. Conclusions
The paper concludes that HIP integrates hierarchical image structure into captioning encoders through a three-level hierarchy and Tree-LSTM feature enrichment. COCO experiments support its captioning efficacy and potential generalization to recognition.
- HIP models constituent visual patterns at instance, region, and whole-image levels within a three-level hierarchy.
- A Tree-LSTM enriches features across all three hierarchy levels before attention-based sentence decoding.
- COCO experiments report new state-of-the-art captioning performances for HIP-based models with direct or relation-enhanced features.
- HIP evaluations also support potential generalization from image captioning to recognition.