PaperlayerView exact revision on arXiv

Source-linked AI summary

RSVQA: Visual Question Answering for Remote Sensing Data

Sylvain Lobry, Diego Marcos, Jesse Murray, Devis Tuia

arXiv:2003.07333v2cs.CV

TL;DR

Remote-sensing information is difficult for nonexperts to access because existing approaches are largely task-specific. The paper introduces RSVQA, automatically builds OSM-derived datasets of image/question/answer triplets, and evaluates a CNN–RNN model. The model reaches 79% accuracy on the low-resolution dataset and 83% on the first high-resolution test set, while geographic generalization remains problematic.

  • Problem

    Existing remote-sensing methods are task-specific, limiting generic natural-language access to image information and leaving direct use largely to experts.

  • Method

    The paper automatically constructs two remote-sensing VQA datasets from OSM information and trains an RSVQA model using CNN visual features, RNN language features, and non-spatial fusion.

  • Results

    79% overall accuracy is achieved on the low-resolution dataset and 83% on the first test set of the high-resolution dataset.

  • Takeaways & Limitations

    RSVQA provides an accessible way to extract high-level information and relational dependencies from remote-sensing images through natural-language questions.

  • Takeaways & Limitations

    The five question types cover only part of the questions of interest, leaving distance, segmentation, and higher-level semantic questions for future diversification.

Abstract

from arXiv · show

This paper introduces the task of visual question answering for remote sensing data (RSVQA). Remote sensing images contain a wealth of information which can be useful for a wide range of tasks including land cover classification, object counting or detection. However, most of the available methodologies are task-specific, thus inhibiting generic and easy access to the information contained in remote sensing data. As a consequence, accurate remote sensing product generation still requires expert knowledge. With RSVQA, we propose a system to extract information from remote sensing data that is accessible to every user: we use questions formulated in natural language and use them to interact with the images. With the system, images can be queried to obtain high level information specific to the image content or relational dependencies between objects visible in the images. Using an automatic method introduced in this article, we built two datasets (using low and high resolution data) of image/question/answer triplets. The information required to build the questions and answers is queried from OpenStreetMap (OSM). The datasets can be used to train (when using supervised methods) and evaluate models to solve the RSVQA task. We report the results obtained by applying a model based on Convolutional Neural Networks (CNNs) for the visual part and on a Recurrent Neural Network (RNN) for the natural language part to this task. The model is trained on the two datasets, yielding promising results in both cases.

I. INTRODUCTION

RSVQA applies visual question answering to remote sensing images so users can query image content and object relationships in natural language. The paper introduces datasets and a CNN–RNN model for this task.

  • Remote sensing information supports diverse applications, but access remains largely limited to experts because existing methods are task-specific.This limitation reduces the scale and variety of addressable problems and the number of potential end-users.
  • VQA answers free-form questions about images, enabling remote sensing queries about objects, counts, and spatial relationships.Examples include detecting a thatched roof and determining whether it is right of a river.
  • Existing remote-sensing language systems generate descriptions but do not enable direct user interaction through questions.
  • A VQA model combines visual feature extraction, language feature extraction, multimodal fusion, and answer prediction.The paper uses a non-spatial fusion step in its proposed RSVQA model.
  • The paper contributes an automatic method for generating remote-sensing VQA datasets and an RSVQA model evaluated on two datasets.The second dataset uses very high resolution imagery to examine spatial generalization and remaining challenges.

II. DATASETS

The dataset construction procedure uses OpenStreetMap information tied to image footprints to create real-image question-and-answer triplets. Multiple pairs can be generated for each image.

  • OpenStreetMap provides geo-localized information used to automatically build question-and-answer pairs for real remotely sensed images.The resulting dataset consists of image, question, and answer triplets.
  • Dataset construction first creates questions and then computes their answers from OSM features within each image footprint.Multiple question-and-answer pairs are extracted for each image.

1) Question contruction:

Questions are generated through a structured procedure that selects an element, optionally refines it with attributes or spatial relations, and converts it into natural language.

  • Question contruction: The procedure selects an element category, applies optional attributes, selects a relative position, and constructs the final question.
  • Question contruction: Element categories come from OSM layers for roads, water areas, buildings, and land-use objects.Buildings and land-use objects are further defined using their OSM type fields.
  • Question contruction: Optional attributes describe element shape or size, with size thresholds adapted to the scale of each dataset.Shapes are determined from geometric properties, while sizes are classified as small, medium, or large.
  • Question contruction: Five relative relations are defined in image space: left of, top of, right of, bottom of, and next to.“Next to” means objects are less than 1000m apart.
  • Question contruction: The method randomly selects among count, comparison, presence, area, and rural/urban question types before generating a base question.Pre-defined language templates are randomly selected to increase diversity across question types and wording.

2) Answer construction:

Answers are computed from OSM objects within each image footprint according to the selected question type. The low-resolution dataset is geographically organized into spatially separated training, validation, and test imagery.

  • Answer construction: Answer construction extracts OSM objects matching the selected element and attributes from the image footprint.
  • Answer construction: Count, presence, and area questions use object counts, zero-versus-nonzero comparisons, and summed object areas, respectively.
  • Answer construction: Comparison questions compare object counts for two elements, while rural/urban questions threshold building counts using resolution-dependent density criteria.
  • Answer construction: The LR dataset uses Sentinel-2 imagery over the Netherlands, split into tiles with training, validation, and test samples represented separately.

B. Data

The paper constructs LR and HR remote-sensing VQA datasets from imagery and OSM-derived questions and answers, with dataset-specific resolutions, coverage, splits, and answer quantization.

  • The study constructs two remote-sensing VQA datasets with different characteristics and use cases.The datasets differ in resolution, spatial coverage, update frequency, and question types.
  • Low resolution (LR): 10 m Sentinel-2 imagery forms the LR dataset, covering 772 tiles of 256 × 256 images and producing 77′232 question-answer pairs.The data cover 6.55 km2 per image and are split at tile level into training, validation, and test sets.
  • High resolution (HR): 15 cm USGS aerial imagery forms the HR dataset, producing 10′659 images and 1′066′316 question-answer pairs from 161 tiles.The images cover 5898m2 each, with training, validation, and two test sets; test set 2 uses the unseen Philadelphia region.
  • Dataset differences: HR imagery supports questions about object coverage and small-object counts, whereas LR resolution limits visibility of such objects.Area questions are restricted to HR, while the two datasets receive different question types based on their characteristics.
  • Answer quantization: LR counting answers are heavy-tailed: 26.7% are 0, 50% are below 7, and the maximum reaches 17139.The LR counting range is quantized into answer intervals because distinguishing very large counts at this resolution is generally impractical.
  • Answer quantization: HR area answers are similarly heavy-tailed, with 60.9% equal to 0m2, while unquantized HR counts span 89 possible classes.Area answers use count-style quantization, but pure count answers remain numerical because the maximum object count is 89.

C. Discussion

The discussion examines answer distributions and identifies limitations in the automatically generated datasets, including annotation, temporal, OSM, and question-template constraints.

  • Questions/Answers distributions: 60.9% of HR numerical answers are “0”, producing the strongest distribution imbalance.The imbalance is greatly reduced by the described quantization process.
  • Limitations of the proposed method: Missing or badly registered annotations, unmatched imagery and OSM acquisition dates, and OSM errors can reduce database accuracy.The method cannot determine whether an OSM element’s entry date matches its real-world appearance date.
  • Limitations of the proposed method: Template-based question and answer generation limits the question space to 9 possible LR answers and 98 HR answers.This makes the generated question-answer set more limited than those in traditional VQA datasets.

III. VQA MODEL

The proposed RSVQA model extracts image and question features, fuses them into a joint representation, and predicts answers through an end-to-end learned network.

  • The VQA model comprises feature extraction, multimodal fusion, and prediction based on a joint image-question representation.The fused vector is learned end-to-end and is used as input to the prediction step.
  • Visual part: A pretrained ResNet-152 extracts a low-dimensional visual representation from each remote-sensing image.The architecture uses residual mappings to avoid degradation associated with deeper networks.

2) Language part:

The language pathway represents each question as a 1200-dimensional vector, combines it with visual features, and classifies the resulting representation into an answer.

  • Language part: A skip-thoughts recurrent neural network encodes each question into a vector representing its word sequence.The model is trained to reconstruct adjacent book sentences, encouraging semantic information in the latent representation.
  • Fusion: The image and question vectors are combined by applying tanh and then performing point-wise multiplication.Although the operation is fixed, end-to-end training encourages the two feature vectors to become comparable under this fusion.
  • Prediction: A one-hidden-layer MLP with 256 units projects the 1200-dimensional fused vector into an answer-class space.Each possible answer is treated as a class, so the output dimension depends on the number of possible answers.

D. Training procedure

The model is trained with Adam until convergence, using dataset-specific epochs and batch sizes to accommodate differing image resolutions.

  • Adam training uses a 10^-5 learning rate until convergence, requiring 150 epochs for LR and 35 epochs for HR.Dropout is set to 0.5 in every fully connected layer.
  • HR training uses batches of 70 instances, whereas LR training uses batches of 280 because HR images are four times larger.

IV. RESULTS AND DISCUSSION

The RSVQA model performs well on several question types, but struggles with relational reasoning, counting, geographic transfer, and limited question diversity. Results also reveal strong dependence on question language and diminishing gains from additional data.

  • General accuracy assessment: 79% overall accuracy is achieved on LR, while 83% is achieved on the first HR test set.Three runs are reported with averages and standard deviations to limit stochastic variability.
  • General accuracy assessment: 87.46% LR and 90.43% HR accuracy for presence questions exceed 67.01% and 68.63% for counting questions.The paper relates this gap to counting difficulty and challenges separating connected instances.
  • General accuracy assessment: 85.24% accuracy is obtained for area questions on the first HR test set.Answer quantization is associated with generally good area-question performance.
  • General accuracy assessment: The model performs poorly on questions about relative object positions.The paper attributes these errors to the high semantic level of the questions and the model’s simple fusion scheme.
  • General accuracy assessment: 90% accuracy is reported for rural/urban questions on the LR dataset.The ground truth uses a hard threshold on the number of buildings.
  • Generalization to unseen areas: Approximately 5% accuracy is lost on the second HR test set, indicating problematic generalization to unseen geographic areas.The paper links this domain shift to different sensors and differing urban organization.
  • Language biases: Random-image evaluation yields 73.78% accuracy on LR, 73.78% on the first HR test set, and 72.51% on the second HR test set.The small accuracy drop indicates that the models rely more on questions than images.
  • Importance of the number of training samples: 65% average accuracy is achieved with 1% of HR training samples versus 83% with the full training set, while performance plateaus after 10%.Numerical tasks require larger sample amounts to reach reported performances.

V. CONCLUSION

The study establishes remote-sensing visual question answering as an accessible information-extraction task, supported by two datasets and a CNN–RNN analysis. Results are promising, but broader question coverage and known challenges such as language bias and counting remain to be addressed.

  • V. CONCLUSION: The paper presents two remote-sensing VQA datasets targeting different applications, built from Sentinel-2 imagery and 30cm USGS aerial orthophotos.The dataset construction method is designed to be extendable and adaptable to different data sources.
  • V. CONCLUSION: A deep-learning model combines convolutional and recurrent neural networks to analyze images and associated questions, then selects the most probable predefined answer.
  • V. CONCLUSION: The first analysis reports promising results and identifies language bias and counting as challenges requiring future research.Suggested directions include attention mechanisms for language bias and dedicated counting components in a modular approach.
  • V. CONCLUSION: The evaluation includes high-resolution confusion-matrix subsets grouped by answer type and accuracy measured after training on different high-resolution training-set sizes.
  • V. CONCLUSION: The current database does not yet support a realistic range of questions, motivating more complex dataset construction or human annotation.
Loading 2003.07333v2…