Source-linked AI summary
Photo-Sketching: Inferring Contour Drawings from Images
Mengtian Li, Zhe Lin, Radomir Mech, Ersin Yumer, Deva Ramanan
TL;DR
The paper addresses the gap between boundary detection and contour drawing, whose visual cues and artistic style differ. It collects contour drawings and proposes a learning-based method for handling diverse, imperfectly aligned annotations. The method achieves strong contour-generation results and state-of-the-art salient boundary detection after fine-tuning on BSDS500.
Problem
Existing boundary detectors do not capture the visual cues and artistic style of contour drawings, while prior sketch representations lack suitable edge-level image correspondence.
Method
The authors collect a contour-drawing dataset and use a learning-based model with a novel MM-loss to handle multiple diverse annotations and imperfect alignment.
Results
Fine-tuning the contour model on BSDS500 notably outperforms HED and RCF for salient boundary detection and learns precise pixel alignment.
Takeaways & Limitations
Contour drawings provide an intermediate representation between image boundaries and abstract line drawings, with potential for scalable geometric annotation through drawing interfaces.
Takeaways & Limitations
The formulation assumes a 1-to-1 mapping between the two domains, although the task has multiple different target drawings for each image.
Abstract
from arXiv · showhide
Edges, boundaries and contours are important subjects of study in both computer graphics and computer vision. On one hand, they are the 2D elements that convey 3D shapes, on the other hand, they are indicative of occlusion events and thus separation of objects or semantic concepts. In this paper, we aim to generate contour drawings, boundary-like drawings that capture the outline of the visual scene. Prior art often cast this problem as boundary detection. However, the set of visual cues presented in the boundary detection output are different from the ones in contour drawings, and also the artistic style is ignored. We address these issues by collecting a new dataset of contour drawings and proposing a learning-based method that resolves diversity in the annotation and, unlike boundary detectors, can work with imperfect alignment of the annotation and the actual ground truth. Our method surpasses previous methods quantitatively and qualitatively. Surprisingly, when our model fine-tunes on BSDS500, we achieve the state-of-the-art performance in salient boundary detection, suggesting contour drawing might be a scalable alternative to boundary annotation, which at the same time is easier and more interesting for annotators to draw.
1. Introduction
The paper studies contour drawings as image representations that combine object boundaries, salient inner edges, and artistic style. It introduces a dataset and learning method for generating them, with application to salient boundary detection.
- Contour drawings encode object boundaries, salient inner edges, and background edges that convey 3D perspective, thickness, and depth.
- Unlike traditional edge and boundary detectors, contour drawings include both occluding contours and salient inner boundaries.
- The contour generator can be fine-tuned for salient boundary detection and achieves state-of-the-art performance on the standard benchmark.
- Prior sketch representations often lack edge-level image correspondence, limiting their suitability for generic scene sketch generation.
- The authors collect 5000 contour drawings and propose a loss that resolves annotator diversity while retaining details.
2. Collecting Contour Sketches
The dataset is collected by having workers trace loosely aligned contour drawings over images, with detailed quality control and multiple drawings per image. Its design distinguishes the dataset from prior edge-like and sketch collections.
- Workers trace over faint background images to produce roughly boundary-aligned drawings through a detailed crowdsourcing interface.
- Compared with other edge-like representations, contour drawings provide detailed internal boundaries while retaining image-contour alignment and greater drawing complexity.
- The dataset comparison highlights differences in boundary alignment, multiple-object coverage, image-sketch pairing, vector encoding, and stroke-order annotations.
- Quality control rejects drawings with missing inner boundaries, missing important objects, large misalignment, or other specified defects.
- 5000 high-quality drawings are paired with 1000 outdoor images, with exactly five drawings per image.
3. Sketch Generation
The paper formulates contour sketch generation as image translation with multiple sketches per image, and proposes MM-loss to handle inconsistent supervision without collapsing diverse targets into an invalid mean contour.
- Problem: The task maps each input image to multiple contour sketches drawn by different annotators, making it a 1-to-many prediction problem.The number of target sketches can vary across images.
- Motivation: Existing conditional GAN methods do not work for sketch generation out of the box, motivating a different cGAN with MM-loss.The paper also notes that soft boundary outputs are difficult to threshold consistently across images.
- Method: The model combines adversarial training with an L1 task loss, where the L1 term encourages the sparsity required for contour outputs.The architecture uses a ResNet encoder-decoder, without skip connections, together with a global discriminator.
- Method: MM-loss treats all target modalities equally for the discriminator while allowing the generator to select the minimum-cost target during training.This mean/min aggregation is designed to alleviate conflicting gradients from diverse annotations.
- Results: Training on original drawings with MM-loss outperforms training on consensus drawings, despite inconsistency in non-consensus strokes.The ablation also reports that adversarial training outperforms pure regression.
- Results: The generated drawings generalize to unseen image contents and capture salient contours without content bias while incorporating perturbations found in human drawings.The model’s results were applied directly on the sketch validation set without hyperparameter tuning.
4. Application to Boundary Detection
The contour generator is repurposed for salient boundary detection because contour sketches must identify salient image edges. Fine-tuning on BSDS500 notably outperforms strong boundary-detection baselines, although performance depends on pretraining.
- Task: Contour sketch generation can be repurposed for salient boundary detection because generating a contour first requires identifying salient image edges.The paper evaluates this application using the salient-boundary criteria proposed by prior work.
- Motivation: Existing boundary detectors commonly merge inconsistent annotations before training, while BSDS500 contains ambiguous boundaries labeled differently by annotators.The paper highlights that weak boundaries may be artifacts of segmentation-like annotation protocols.
- Results: Qualitatively, the method more frequently generates complete objects without over-generating texture edges, although all methods fail on one challenging example.The comparison is reported in the salient-boundary results.
- Results: Fine-tuning the contour model on BSDS500 notably outperforms HED and RCF by a sizable margin.Training only on BSDS500 performs worse than HED and RCF, which are pretrained on ImageNet.
- Results: The fine-tuned model learns to generate contours with precise pixel alignment on BSDS500.The evaluation uses the salient consensus-boundary setup associated with BSDS500.
5. Cost-Free Data Expansion
The authors propose a sketch-drawing game that uses real-time rewards, penalties, and automatic quality control to make contour data collection more scalable and potentially free.
- Sketch-based collection is easier and more engaging than precise boundary annotation because it requires only loose alignment.The authors report that annotators described the task as fun and extended the interface into a game.
- The game rewards strokes near sampled points from generated boundaries and penalizes strokes near sampled off-boundary points.Hidden reward and penalty points provide immediate feedback while scoring sketch quality.
- The AI score identifies 90% of unqualified sketches, demonstrating its ability to reject poor submissions.
- The authors plan to release the game publicly and iteratively update generation models to support an ongoing sketch collection loop.
6. Conclusion
The paper introduces contour-drawing data, evaluation criteria, and a generation model, showing that these sketches support both aesthetically oriented graphics applications and salient boundary detection.
- The work introduces a dataset, benchmark criteria, and model for generating contour drawings from images.
- Contour sketches provide an intermediate representation for exploring open contours from geometric occlusion events and learning representations for salient boundary detection.
- The authors identify contour sketches as a potentially scalable alternative for collecting geometric visual annotations through game interfaces.