PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Models and Data for Remote Sensing Image Caption Generation

Xiaoqiang Lu, Binqiang Wang, Xiangtao Zheng, Xuelong Li

arXiv:1712.07835v1cs.CV

TL;DR

Remote sensing captioning lacks clear ways to describe overhead image content accurately and flexibly, beyond existing recognition tasks. The paper introduces annotation guidance and RSICD, then benchmarks captioning methods and representations. Experiments report that captioning methods can produce acceptable descriptions, while more work remains necessary for remote-sensing-specific generation.

  • Problem

    Remote sensing studies largely recognize scenes or objects, while comprehensive semantic descriptions of object attributes and relations remain insufficiently addressed.

  • Method

    The paper defines specialized annotation instructions, constructs the RSICD dataset, and evaluates captioning methods, image representations, and sentence-generation methods.

  • Results

    Experiments benchmark methods on RSICD using BLEU, METEOR, ROUGE L, and CIDEr, finding natural-image captioning methods transfer with only acceptable descriptions.

  • Takeaways & Limitations

    RSICD provides a benchmark and data resource for advancing remote-sensing image captioning, but remote-sensing-specific caption generation still needs further work.

Abstract

from arXiv · show

Inspired by recent development of artificial satellite, remote sensing images have attracted extensive attention. Recently, noticeable progress has been made in scene classification and target detection.However, it is still not clear how to describe the remote sensing image content with accurate and concise sentences. In this paper, we investigate to describe the remote sensing images with accurate and flexible sentences. First, some annotated instructions are presented to better describe the remote sensing images considering the special characteristics of remote sensing images. Second, in order to exhaustively exploit the contents of remote sensing images, a large-scale aerial image data set is constructed for remote sensing image caption. Finally, a comprehensive review is presented on the proposed data set to fully advance the task of remote sensing caption. Extensive experiments on the proposed data set demonstrate that the content of the remote sensing image can be completely described by generating language descriptions. The data set is available at https://github.com/201528014227051/RSICD_optimal

I. INTRODUCTION

Remote sensing captioning addresses the gap between recognizing remote-sensing scenes and describing their objects, attributes, and relations in comprehensive sentences. The paper contributes annotation guidance, the RSICD dataset, and benchmark evaluations of captioning methods.

  • Remote sensing captioning aims to summarize image content semantically, beyond scene classification, object recognition, and segmentation labels.
  • RSICD contains 10921 aerial images, each paired with five descriptions, and is presented as the largest remote sensing captioning dataset.
  • The paper evaluates representative encoder-decoder frameworks using varied image representations and sentence-generation methods on RSICD.
  • The paper’s three contributions are specialized annotation characteristics, a large-scale benchmark dataset, and a comprehensive review of captioning methods.
  • Remote sensing images require specialized annotation because scale, category, and rotation ambiguities affect their semantics.

II. RELATED WORK

Prior image-captioning research includes retrieval, object-detection, and encoder-decoder approaches, but remote-sensing images pose additional semantic challenges. The paper motivates specialized datasets and methods for their ambiguous, broad scene content.

  • Natural image captioning methods include retrieval-based, object-detection-based, and encoder-decoder approaches.
  • Retrieval-based methods depend on similar-image retrieval, which can produce grammatically poor or unreadable sentences.
  • Object-detection methods model relations among detected objects, making detection and relation-model quality important to generated sentences.
  • Encoder-decoder methods encode an image into a vector and decode it into a sentence, commonly using CNNs with RNNs or LSTMs.
  • The paper frames a large-scale captioning dataset as needed because prior remote-sensing captioning lacked an accredited dataset comparable to COCO.
  • Remote-sensing captioning is more complex because overhead imagery lacks a fixed viewpoint and contains ambiguous, difficult-to-describe content.

III. DATASET FOR REMOTE SENSING IMAGE CAPTIONING

RSICD is designed to provide flexible, comprehensive descriptions for remote-sensing images whose scale, viewpoint, and land-cover composition complicate annotation. Its annotations and dataset structure support varied descriptions of image content and relations.

  • Earlier datasets include UCM-captions with 21 land-use classes and Sydney-captions with seven classes, both assigning five sentences per image.
  • A prior undisclosed dataset used fixed semantic templates and was described as lacking flexibility and diversity.
  • RSICD is introduced as a remote-sensing captioning dataset intended to address variable scales and rotation invariance.
  • Annotation instructions require describing important image parts while avoiding unsupported vague quantities and directional nouns.
  • RSICD contains 24333 sentences with at least six words, distributed across images described by one to five sentences.
  • The dataset records relations such as near, in two rows, in, on, surrounded by, and in two sides of.

IV. REMOTE SENSING IMAGE CAPTIONING

The paper reviews encoder-decoder captioning by separating image representation from sentence generation. It covers handcrafted and learned visual features, along with neural architectures for mapping image content to language.

  • The encoder-decoder framework first encodes a remote-sensing image into a vector and then decodes that vector into a sentence.
  • The reviewed deep multimodal approach separately represents images and sentences before generating a coherent sentence for each image.
  • Image representations are divided into handcrafted features and learned features.
  • Handcrafted representations use feature encoding techniques including Bag Of Words, Fisher Vector, and Vector of Locally Aggregated Descriptors.
  • The study extracts learned image features from fully connected layers of ImageNet-pretrained AlexNet, VGGNet, and GoogLeNet models.

1) Representing remote sensing images:

The paper presents an encoder-decoder pipeline that represents remote sensing images as features and generates sentences sequentially with recurrent models. It discusses RNN limitations for long-term dependencies and LSTM gating as a remedy.

  • Representing remote sensing images: The encoder-decoder pipeline encodes a remote sensing image into a vector and decodes that representation into a sentence.The image feature is obtained first, then used during sentence generation.
  • Representing remote sensing images: Remote sensing image features may be handcrafted or deep representations produced by a feature representation process.The image feature is denoted e0 and has dimension u.
  • Sentences Generation: RNNs pass information from one state to the next, allowing previous input information to persist during sentence generation.At each step, an input xt produces a state ht through a neural network.
  • Sentences Generation: RNNs cannot well solve long-term dependencies when information needed for the current task is far from the current state.This motivates using LSTM networks for sentence generation.
  • Sentences Generation: LSTM uses forget, input, and output gates to control information passed through the network and filter the cell state.The input gate selects updates, while a tanh layer generates candidate values added to the network state.
  • Sentences Generation: During training, image features and corresponding sentences train an RNN or LSTM to predict words one by one.The image feature is imported only at t = 1, and the model parameters are learned by minimizing a loss function.

B. Attention based method

The attention-based method analyzes different parts of a remote sensing image through an alternative image representation. It includes deterministic and stochastic attention-training manners.

  • B. Attention based method: Attention-based captioning uses a different image representation method to examine different parts of an image.The method is introduced with deterministic and stochastic attention manners.
  • B. Attention based method: The deterministic attention manner is trained with standard backpropagation, whereas the stochastic manner maximizes a lower bound.

1) Representing remote sensing images:

The methods represent remote sensing images with convolutional feature vectors and generate captions by attending to visual information while predicting words sequentially.

  • Convolutional features are extracted as M vectors, each with dimension D and corresponding to a part of the remote sensing image.
  • An annotation vector contains the extracted set of feature vectors used to represent the image.
  • Sentences are encoded as sequences of 1-of-K word vectors, where K is vocabulary size and L is sentence length.
  • The attention-based LSTM uses the former state to decide where to look before predicting the next word.
  • Soft attention assigns weights to image parts, whereas hard attention samples parts and uses reinforcement learning for an overall result.
  • In the attention-based method, the context vector is computed from annotation vectors and supplied with the previous word to the LSTM.

V. EXPERIMENTS

The experiments evaluate remote sensing captioning with standard language metrics and subjective assessment across multimodal and attention-based methods.

  • The study evaluates multimodal and attention-based methods using deep CNN and handcrafted image representations.
  • Multimodal experiments use 256-dimensional word embeddings and hidden states with a learning rate of 0.0001.
  • Attention-based experiments use 512-dimensional word embeddings and hidden states with a learning rate of 0.0001.
  • Evaluation metrics include BLEU, ROUGE L, METEOR, and CIDEr.
  • BLEU measures n-gram co-occurrence, while ROUGE L uses the longest common subsequence.
  • The paper also uses subjective metrics to assess generated-sentence quality and model generalization across datasets.

B. The results of multimodal method

The multimodal experiments compare handcrafted and CNN representations across three captioning datasets and examine how training ratios affect caption metrics.

  • Experiments split each dataset into 80% training, 10% validation, and 10% test data for multimodal captioning.
  • Four handcrafted representations—SIFT, BOW, FV, and VLAD—are evaluated for caption generation.
  • All CNN features outperform handcrafted features on RSICD, with AlexNet best on ROUGE L and CIDEr and VGG19 slightly best on other objective metrics.
  • Training-ratio experiments use VGG16 features, reserve 10% for validation, and vary the training-to-testing proportion.
  • On UCM-captions, all metrics increase as the training ratio rises.
  • On Sydney-captions, metrics initially increase and then remain almost stable, which the paper attributes to dataset imbalance centered on residential-area sentences.
  • On RSICD, performance first improves, then fluctuates between 60% and 80%, consistent with duplicated reference sentences and the CIDEr reference trend.

C. The results of attention based method

Attention-based captioning was evaluated across CNN features, training ratios, datasets, and feature-cluster settings. Hard attention generally performed best, while results varied by dataset and representation.

  • 1) Results based on different CNNs:: Hard attention outperformed soft attention in most conditions, with GoogLeNet-based hard attention best on UCM-captions and RSICD, but VGG16-based soft attention best on Sydney-captions.The comparison used convolutional features from VGG16, VGG19, AlexNet, and GoogLeNet.
  • 2) Results of different training ratios:: Metrics for AlexNet convolutional features became almost stable when the training ratio exceeded 20%.The reported trend was an initial increase followed by near stability.
  • C. The results of attention based method: Most generated captions described remote sensing images accurately, although errors arose from visually similar structures and frequent word co-occurrences.The examples include buildings resembling railway stations and confusion involving frequently co-occurring words.
  • C. The results of attention based method: The experiments also analyzed Sydney-captions imbalance and used subjective evaluation to compare models trained on different datasets.The supplied table captions identify benchmark results for multimodal and attention-based methods across UCM-captions, Sydney-captions, and RSICD.
  • C. The results of attention based method: Metrics almost decreased as the Fisher Vector number of cluster centers increased on UCM-captions and RSICD.On Sydney-captions, metric changes were not significantly related to the number of cluster centers.

2) Generalization capabilities:

Cross-dataset evaluation examined how models trained on UCM-captions, Sydney-captions, and RSICD generalized to other datasets. RSICD showed stronger generalization in one comparison, but substantial unrelated-caption rates and metric limitations remained.

  • 2) Generalization capabilities:: A model trained on RSICD generalized better to Sydney-captions than a model trained on UCM-captions, while UCM-captions generalized better to RSICD than Sydney-captions.These comparisons were based on objective metrics reported in Table X.
  • 2) Generalization capabilities:: Objective metrics were considered unfair for cross-dataset generalization because they compare generated captions with reference sentences from different datasets.Subjective evaluation was presented as a better assessment for this setting.
  • 2) Generalization capabilities:: Subjective evaluation classified generated sentences as related to the image, unrelated to the image, or totally depicting the image.The categories distinguish partial topic capture, no relation to the main topics, and correct depiction of the main topics.
  • 2) Generalization capabilities:: 38% of sentences from the RSICD-trained model described the main topics of UCM-captions test images.This result came from subjective evaluation on UCM-captions.
  • 2) Generalization capabilities:: 40% of sentences from the RSICD-trained model were unrelated to Sydney-captions test images, despite its comparatively better generalization.The authors conclude that remote sensing image caption generation still needs further study.

VI. CONCLUSION AND FUTURE WORK

The paper introduces instructions for comprehensive remote sensing descriptions, constructs RSICD, and benchmarks captioning methods across representations and datasets. Results are acceptable but indicate that further work is needed, while future dataset revisions address duplicated sentences.

  • VI. CONCLUSION AND FUTURE WORK: The paper provides annotation instructions and constructs the RSICD remote sensing image captioning dataset.It also evaluates captioning methods using handcrafted representations and convolutional features.
  • VI. CONCLUSION AND FUTURE WORK: Benchmarks evaluate methods with BLEU, METEOR, ROUGE L, and CIDEr across different experimental settings.The evaluation covers handcrafted and convolutional image representations.
  • VI. CONCLUSION AND FUTURE WORK: Natural-image captioning methods transfer to remote sensing images only with acceptable descriptions.The conclusion states that remote sensing image characteristics require additional work.
  • VI. CONCLUSION AND FUTURE WORK: Future RSICD versions are intended to be more comprehensive because some current sentences duplicate existing sentences.The authors also plan to apply new image-processing and natural-language-processing techniques.
Loading 1712.07835v1…