PaperlayerView exact revision on arXiv

Source-linked AI summary

Guide Me: Interacting with Deep Networks

Christian Rupprecht, Iro Laina, Nassir Navab, Gregory D. Hager, Federico Tombari

arXiv:1803.11544v1cs.CV

TL;DR

The paper addresses the limited ability to adapt static CNN predictions through human feedback, proposing a guide that modifies a trained network’s activations in response to user input. It reports more accurate segmentation through language-to-vision semantic transfer, while noting that especially noisy initial predictions can be difficult to repair fully.

  • Problem

    CNNs are commonly deployed as fixed models after training, although they may make imperfect predictions and could benefit from human feedback during use.

    Source: §1 · p. 1

  • Method

    A spatio-semantic guide adjusts a pre-trained CNN’s internal activations in response to optional user feedback, including natural-language queries.

    Source: §1 · p. 1, §3.1 · p. 3

  • Results

    The module transfers semantic information from natural language to vision and produces more accurate segmentations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The system updates feature representations on the fly from a user’s hint without further training the network’s parameters.

    Source: §5 · p. 8

    When the initial prediction is particularly noisy, the guide may only partly recover a missing building and leave many spurious regions.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

Interaction and collaboration between humans and intelligent machines has become increasingly important as machine learning methods move into real-world applications that involve end users. While much prior work lies at the intersection of natural language and vision, such as image captioning or image generation from text descriptions, less focus has been placed on the use of language to guide or improve the performance of a learned visual processing algorithm. In this paper, we explore methods to flexibly guide a trained convolutional neural network through user input to improve its performance during inference. We do so by inserting a layer that acts as a spatio-semantic guide into the network. This guide is trained to modify the network's activations, either directly via an energy minimization scheme or indirectly through a recurrent model that translates human language queries to interaction weights. Learning the verbal interaction is fully automatic and does not require manual text annotations. We evaluate the method on two datasets, showing that guiding a pre-trained network can improve performance, and provide extensive insights into the interaction between the guide and the CNN.

1. Introduction

Because a deployed CNN remains fixed after training and may make mistakes, the paper proposes optional user feedback to adjust its activations at test time. The introduction frames natural-language guidance as a way to refine semantic segmentation predictions without manual output repair.

1. Introduction

  • Static CNN predictions may be imperfect, while online adaptation to human feedback could benefit real-world applications.The introduction highlights medical image analysis and computer-assisted diagnosis, as well as labor-intensive segmentation labeling with scarce, expensive annotations.Source: §1 · p. 1
  • The method lets users guide a pre-trained CNN at test time by translating feedback into changes in its internal activations.User input is optional: the network can produce an initial estimate and then update it in response to a text query.Source: §1 · p. 1
  • The system uses an RNN to process natural-language guidance, illustrated by a user correcting confusion between sky and cloud predictions without updating CNN weights.The figure shows the prediction updated after the user indicates that there is no sky.Source: Figure 1 · p. 2
  • The authors report that the module transfers semantic information from language to vision and produces more accurate segmentations without explicit text-region correspondence annotations.They describe this as also offering insights into how CNNs structure inference around natural categories.Source: §1 · pp. 1–2

2. Related Work

Prior work includes interactive vision systems and combinations of language and vision, but much deep-learning segmentation work lacks an inference-time human interface. This paper distinguishes its approach by keeping the output visual and generating textual interaction expressions automatically rather than relying on paired vision-text data.

2. Related Work

  • Earlier semi-automatic segmentation systems accepted seed points, boxes, contours, scribbles, gaze, or yes/no answers, while deep-learning approaches included scribbles, clicks, distance maps, and boxes.Other examples included user-guided colorization and medical-imaging methods using seed maps or a second network.Source: §2 · p. 2
  • Most deep-learning segmentation methods lack an inference-time human interface, leaving the model and attainable performance fixed after training.The passage notes that integrating a human directly into training across thousands of images is challenging.Source: §2 · p. 2
  • Unlike vision-language work that often returns text, this approach returns a visual output in the same domain as the initial prediction.It also simulates interaction with automatically generated textual expressions rather than relying on paired questions and answers or captions.Source: §2 · p. 2
  • The paper focuses on semantic segmentation, using out-of-the-box state-of-the-art models as estimators that its module guides with human input or other hints.The cited estimator models are [12] [41].Source: §2 · p. 3

3. Methods

The method inserts a guide into a fixed CNN to adjust its activations, enabling either back-propagation-based guidance from output-domain hints or learned guidance from natural-language queries.

3. Methods

  • The guide is inserted between a CNN head and tail, where its guiding block adjusts activation maps to improve the final prediction.The information used to modify the guided network is called the hint.Source: §3 · p. 3

3.1. Guiding Block

  • Channel-wise activation control limits over-fitting from element-wise updates, while added spatial parameters allow localized feature-map changes.The guide can emphasize or suppress aspects of the image representation to adapt the prediction semantically.Source: §3.1 · p. 3

3.2. Guiding by Back-propagation

  • For back-propagation guidance, the system optimizes guiding parameters against a masked loss on the hint while leaving the CNN weights unchanged.Hints can be sparse pixel-level class labels, and the mask selects the locations whose loss contributes to optimization.Source: §3.2 · p. 4

3.3. Learning to Guide with Text

  • For language guidance, a recurrent model encodes a query and predicts guiding parameters, which update the feature map and prediction.The query is encoded with word embeddings and a GRU; the word embeddings are frozen during training.Source: §3.3 · p. 4
  • Training queries are generated automatically from prediction errors, combining operations such as find or remove with semantic classes and spatial phrases.The method uses vision-only annotations, samples classes according to potential prediction improvement, and trains the guide to interpret commands from real users at test time.Source: §3.3 · p. 5

4. Experiments

Experiments show that both direct backpropagation and language-guided interaction can improve segmentation from a fixed, pre-trained CNN. They also examine hint types, guiding location and repetition, qualitative outcomes, failure cases, and learned embeddings.

4.1. Guiding by Backpropagation

  • 62.6% to 81.0% mIoU: 20 user questions improve pre-trained FCN-8s on PascalVOC 2012 without further training.The reported DeepLab-v3 leaderboard score is 86.9% mIoU with additional data.Source: Table 1 · p. 5, §4.1 · p. 5

4.2. Guiding with Text Inputs

  • Find hints yield larger performance gains than remove hints, which are more ambiguous because they do not specify what should replace the removed class.Training with both hint types produces average performance between the two.Source: Table 4 · p. 6, §4.2 · p. 7
  • A second hint further improves performance, but three or more hints over-amplify features and decrease performance while retaining gains over the unguided model.The repeated-hint experiment uses find or rmv queries at res5a, although the guide was not trained for successive hints.Source: Table 5 · p. 6, §4.2 · p. 7
  • 30.5% to 36.5% overall score: a single find hint improves the best guided model on COCO-Stuff.The segmentation model remains fixed; only the guiding mechanism is trained on images unseen during pre-training.Source: §4.2 · p. 6
  • Find hints resolve ground/playing-field confusion and spurious predictions, recover banners and occluded classes, but banner predictions can bleed into buildings.The figure's black ground-truth label marks unlabeled pixels, where any prediction is allowed.Source: Figure 3 · p. 7
  • Guides placed too early lack enough high-level information to guide appropriately.The study evaluates mIoU for guides inserted at different network layers using find queries.Source: Table 3 · p. 6, §4.2 · p. 7

4.3. Insights into the Model

  • Very noisy initial predictions are difficult to repair fully, and correct heatmap focus can still fail to overturn a dominant sea label.The authors suggest more precise queries could help in such cases.Source: §4.3 · pp. 7–8, Figure 4 · p. 8, Figure 5 · p. 8
  • The 256-dimensional γ-vector embedding clusters semantically similar classes, while visually similar sand and mud are difficult to guide apart.The class-category groupings shown by color were not used during training.Source: §4.3 · p. 8, Figure 6 · p. 8

5. Conclusion

The system lets a user guide a neural network by updating its feature representations on the fly in response to a hint, without further training its parameters. The paper also describes automatically generated queries and identifies two directions for future work.

5. Conclusion

  • Text queries let users guide a trained network by updating its feature representations on the fly to improve an initial estimate, without further training its parameters.The user sends a query after receiving an initial estimate, and the network returns an updated prediction.Source: §5 · p. 8
  • A specialized algorithm automatically creates the queries; future work could use a second network to generate hints and add image-guided attention to the RNN.Source: §5 · p. 8
Loading 1803.11544v1…