PaperlayerView exact revision on arXiv

Source-linked AI summary

BreakingNews: Article Annotation by Image and Text Processing

Arnau Ramisa, Fei Yan, Francesc Moreno-Noguer, Krystian Mikolajczyk

arXiv:1603.07141v1cs.CV

TL;DR

Many image–language methods use text that directly describes visual content, whereas news articles can have only loosely related, connotative links between text and images. This paper presents BreakingNews and deep-learning approaches for several news-analysis tasks; the authors report promising baseline results while noting room for improvement.

  • Problem

    Many existing image–language datasets use human-written descriptions of visual content, unlike news articles whose text and images may be only loosely related.

    Source: §1 · pp. 2–3

  • Method

    The paper introduces a roughly 100K-article news dataset and deep-learning approaches, including an adaptive CNN for multiple analysis tasks.

    Source: §1 · p. 2

  • Results

    The authors report promising baseline results across the tasks while noting that substantial room for improvement remains.

    Source: §1 · p. 3

  • Takeaways & Limitations

    BreakingNews is presented as a benchmark for studying image–text relations and news tasks such as source detection, geolocation, and popularity prediction.

    Source: §1 · p. 3, §5 · p. 11

    The authors report that the baseline results leave substantial room for improvement.

    Source: §1 · p. 3

Abstract

from arXiv · show

Building upon recent Deep Neural Network architectures, current approaches lying in the intersection of computer vision and natural language processing have achieved unprecedented breakthroughs in tasks like automatic captioning or image retrieval. Most of these learning methods, though, rely on large training sets of images associated with human annotations that specifically describe the visual content. In this paper we propose to go a step further and explore the more complex cases where textual descriptions are loosely related to the images. We focus on the particular domain of News articles in which the textual content often expresses connotative and ambiguous relations that are only suggested but not directly inferred from images. We introduce new deep learning methods that address source detection, popularity prediction, article illustration and geolocation of articles. An adaptive CNN architecture is proposed, that shares most of the structure for all the tasks, and is suitable for multitask and transfer learning. Deep Canonical Correlation Analysis is deployed for article illustration, and a new loss function based on Great Circle Distance is proposed for geolocation. Furthermore, we present BreakingNews, a novel dataset with approximately 100K news articles including images, text and captions, and enriched with heterogeneous meta-data (such as GPS coordinates and popularity metrics). We show this dataset to be appropriate to explore all aforementioned problems, for which we provide a baseline performance using various Deep Learning architectures, and different representations of the textual and visual features. We report very promising results and bring to light several limitations of current state-of-the-art in this kind of domain, which we hope will help spur progress in the field.

1 INTRODUCTION

BreakingNews addresses news articles whose images and text are only loosely related, a setting that differs from datasets built around direct visual descriptions. The paper introduces a large, metadata-rich dataset and learning approaches for analyzing these articles across several tasks.

1 INTRODUCTION

  • An adaptive CNN shares most of its structure across source detection, article illustration, popularity prediction, and geolocation, with task-specific final layers.The paper also compares single-task, multitask, and transfer learning, and combines image and text representations in an LSTM for caption generation.Source: §1 · p. 2
  • BreakingNews contains approximately 100K news articles with one to three images, captions, linguistic features, GPS coordinates, and reader comments.Its articles cover 2014 and come from newspapers and media agencies including BBC News, The Guardian, and The Washington Post.Source: §1 · pp. 2–3
  • Unlike image-caption datasets, BreakingNews links are often indirect: image objects, actions, and attributes may not appear explicitly in article text.The dataset also supports source detection, GPS estimation, and popularity prediction based on reader comments and retweets.Source: §1 · pp. 2–3
  • Across the proposed tasks, the authors report promising baseline results while noting substantial room for improvement.They benchmark deep neural architectures with different text and image representations.Source: §1 · p. 3

2 TASKS DESCRIPTION AND RELATED WORK

The paper surveys text-and-image tasks and datasets, emphasizing that illustrated news differs from conventional visual-description data. BreakingNews is presented as a larger, more heterogeneous resource for studying these tasks and learning settings.

2.1 Text Illustration

  • Text illustration retrieves an image or small image subset for a news story; prior approaches use semantic matching, search, or generated imagery.Earlier retrieval methods typically use keywords or short sentences, while some news systems rely on article titles, narrative keywords, or image-associated text.Source: §2.1 · pp. 3–4

2.2 Geolocation

  • News geolocation combines text and image cues, while cited news-specific approaches have often relied on text alone.One cited exception combines textual descriptors with global image representations for nearly two thousand New York Times articles.Source: §2.2 · p. 4

2.3 Popularity Prediction

  • News popularity is commonly proxied by comment counts or social-media shares, which have been used to predict article relevance or lifespan.Related work also studies popularity prediction for blogs, videos, and tweets.Source: §2.3 · p. 4

2.4 Source detection

  • Source detection identifies the news agency that originally published an article, using language, topic, or sentiment patterns.The authors report no known prior approaches explicitly targeting news-source identification, while noting related authorship and plagiarism work.Source: §2.4 · p. 4

2.5 Caption Generation

  • News caption generation is harder than describing an image because the image–caption relationship can be indirect and subtle.Most RNN-based captioning work instead focuses on accurately describing visual content; the cited BBC News dataset is much smaller than BreakingNews.Source: §2.5 · p. 5

2.6 Image and Text Datasets

  • Unlike datasets whose captions describe pictured objects and actions, BreakingNews includes connotative text–image links and metadata for a wider range of problems.The paper describes it as two orders of magnitude larger than the BBC News dataset.Source: §2.6 · p. 5

2.7 Multi-Task and Transfer Learning on Large Datasets

  • BreakingNews can support independent task development as well as multi-task and transfer-learning studies.The paper notes that these learning approaches can boost generalization in modern deep networks, citing [49],.Source: §2.7 · p. 5

3 REPRESENTATION

The paper represents article text with Bag-of-Words or Word2Vec and represents images using activations from pretrained CNNs. It considers multiple ways of forming article-level text inputs and combining visual features.

3 REPRESENTATION

  • The representation section uses Bag-of-Words and Word2Vec for text, and deep CNN features for images.The stated goal is to represent text and images for news-article analysis.Source: §3 · p. 6

3.1 Text representation

  • Bag-of-Words uses lemmatized training tokens appearing more than L times and TF-IDF weighting; Word2Vec skip-gram performed better than CBOW in the authors’ applications.Word2Vec article representations are formed from the full word matrix or by taking the mean or maximum across each embedding dimension.Source: §3.1 · p. 6

3.2 Image representation

  • Image features concatenate L2-normalized 4,096-dimensional activations from VGG19 and Places to form an 8,192-dimensional representation.The activations are taken from the last ReLU layers of the pretrained CNNs.Source: §3.2 · p. 6

4 LEARNING

The section presents a shared CNN framework for article analysis, with task-specific outputs and losses, and extends caption generation to fixed-feature and end-to-end LSTM models. It also describes multitask and transfer variants that reuse learned representations.

4.1 CNN for article analysis

  • A shared textual CNN feeds task-specific layers for article illustration, source detection, popularity prediction, and geolocation.The architecture distinguishes shared data and layers from task-specific components.Source: §4.1 · p. 7, §4 · p. 7
  • Articles are encoded as padded Word2Vec matrices, convolved with rectangular kernels, max-pooled, and passed through fully connected layers to task outputs.The convolution uses 256 kernels of size Dw × 5; the pooled vectors are transformed to 64 channels before the final output layer.Source: §4.1 · pp. 7–8
  • Source detection uses multinomial logistic loss, while popularity prediction uses the ℓ1 distance between predicted and observed comment counts.Source: §4.1 · pp. 8–9
  • Article illustration uses a CCA loss to maximize correlation between article representations and concatenated VGG19 and Places image features.The image target is an 8,192-dimensional concatenation of ℓ2-normalized activations.Source: §4.1 · p. 8
  • Geolocation prediction outputs latitude and longitude and minimizes Great Circle Distance rather than Euclidean distance.The spherical distance accounts for the closeness of meridians near −π and π.Source: §4.1 · p. 9
  • The multitask CNN shares representations across tasks but keeps task-specific output layers; the transfer CNN replaces and fine-tunes the output layers for a new task.Source: §4.1 · p. 10

4.2 LSTM for caption generation

  • The fixed-feature captioning LSTM uses mean Word2Vec article features, VGG19 and/or Places image features, or their concatenation.The article and image representations are fixed while the LSTM is trained.Source: §4.2 · p. 10
  • The end-to-end captioning model connects language and image CNNs to an LSTM, allowing gradients to propagate from caption training into both CNNs.The CNNs encode article and image information into 500-dimensional vectors; the image CNN is pretrained on ImageNet.Source: §4.2 · p. 11

5 BREAKINGNEWS DATASET

BreakingNews is a multimodal news dataset proposed to benchmark tasks using article text, images, captions, geolocation, and comments. Its images and text are often only indirectly related, and its scale and metadata exceed those of the comparable BBC News dataset.

5 BREAKINGNEWS DATASET

  • BreakingNews was assembled to evaluate CNN and LSTM architectures on varied tasks, including applications such as image suggestion and article promotion.The paper identifies the indirect relationship between images and text as a challenge for learning these connections.Source: §5 · p. 11

5.1 Description of the Dataset

  • BreakingNews contains approximately 100,000 English-language articles from 2014, each with at least one image and covering a wide range of topics.The articles were restricted to English, image-containing stories from selected highly ranked news agencies; the geographic distribution means the dataset is weighted toward English-speaking countries, particularly the United Kingdom.Source: §5.1 · p. 12
  • News images often show close-ups or complex scenes, while their captions use richer vocabulary than Flickr8K and rarely describe the pictures' exact contents.Forty-six percent of BreakingNews pictures contain faces; 8,000 random captions contain 28,028 unique tokens, compared with 8,918 in Flickr8K.Source: §5.1 · p. 12
  • The dataset includes original images, captions, comments, social-network shares when available, and geolocation for both news agencies and article content.Geolocation is taken primarily from RSS summaries and sometimes inferred using heuristics; for one outlet, tweets substitute for unavailable comments and share data.Source: §5.1 · p. 12
  • Articles are additionally annotated with linguistic features including part-of-speech tags, semantic topics, named entities, and sentiment.Source: §5.1 · p. 12

5.2 Comparison to similar datasets

  • The comparable BBC News dataset has 3,361 image-and-caption articles, one source, and lacks geolocation, tags, social shares, and user comments.The authors describe its size as insufficient for training data-hungry models such as LSTM recurrent neural networks.Source: §5.2 · p. 13

6 RESULTS

The experiments evaluate source detection, article illustration, popularity prediction, geolocation, and caption generation using CNN, CCA, SVR, and LSTM approaches. CNNs show advantages in several analysis tasks, while illustration and caption generation remain challenging when news text and images are weakly related.

6.2 CNNs for article analysis

  • CNN source detection produces accuracies more than 10 points above the best shallow feature, and its learned representation separates source classes better.The comparison includes shallow TF-IDF BoW and Word2Vec features; Table 3 reports results for source detection and balanced accuracy because the classes are unbalanced.Source: §6.2 · p. 14, Table 3 · p. 14
  • CNN article illustration only marginally outperforms CCA with Word2Vec mean and combined VGG19+Places features, with median ranks around 400 across more than 24K images.For CCA, combining VGG19 and Places improves over either visual feature alone, and TF-IDF BoW has an edge over mean Word2Vec; the authors hypothesize loose image-text relation limits CNN gains.Source: §6.2 · p. 15
  • Replacing article images with related Internet images reduces median rank from 499 to 161 for the W2V mean/VGG19 combination.The authors interpret this as evidence that original images are weakly correlated with the articles and that additional images improve the correlation.Source: §6.2 · p. 16
  • Caption-based visual representations reduce median rank from 499 to 7 with original captions and 42 with anonymized captions.Anonymization replaces proper names with common nouns; the authors note that substantial improvement remains possible with original images.Source: §6.2 · p. 16
  • For popularity prediction, CNN significantly outperforms SVR with Word2Vec mean on median ℓ1 distance, while mean-distance differences are less pronounced.The authors regard median distance as more stable given the long-tailed comment counts.Source: §6.2 · p. 16, Table 5 · p. 16
  • CNN with Great Circle Distance loss outperforms shallow learners and CNN with Euclidean loss, achieving a median geolocation error of 900 kilometers.The evaluation uses the nearest ground-truth geolocation at test time when articles have multiple locations.Source: §6.2 · p. 16, Table 6 · p. 17
  • Single-task learning tends to perform best, while transfer learning generally outperforms multitask learning and is usually comparable to single-task learning.Transfer from source detection to popularity prediction performs even better than single-task popularity prediction; the authors suggest that learned low- and mid-level representations may generalize.Source: §6.2 · p. 17

6.3 LSTM for Caption generation

  • Caption generation results are poor relative to standard image-captioning models, and fine-tuning CNN features does not improve the end-to-end model.Combining text and visual features, including VGG19 and Places, improves results; the authors call for more suitable treatment and evaluation of news captions.Source: §6.3 · p. 18, Table 8 · p. 18

7 CONCLUSION

The paper introduces adaptive CNN methods and a roughly 100K-article dataset for news tasks involving weakly correlated text and images. Results are promising for source detection, article illustration, and geolocation, while caption generation remains more sensitive to the indirect image-text relationship.

7 CONCLUSION

  • The paper introduces CNN architectures for news illustration, source detection, geolocation, popularity prediction, and automatic captioning, alongside a dataset of approximately 100K articles.The dataset includes metadata such as GPS coordinates and popularity measures, and is about two orders of magnitude larger than similar existing datasets.Source: §7 · p. 18
  • An adaptive CNN shares most of its structure across tasks, with a Great Circle Distance loss for geolocation and Deep Canonical Correlation Analysis for article illustration.Source: §7 · p. 18
  • Overall results are promising for source detection, article illustration, and geolocation, while automatic caption generation is more sensitive to loosely related text and images.The authors identify designing metrics for this situation as future work.Source: §7 · p. 18

Related papers

Loading 1603.07141v1…