Source-linked AI summary
Learning deep structured active contours end-to-end
Diego Marcos, Devis Tuia, Benjamin Kellenberger, Lisa Zhang, Min Bai, Renjie Liao, Raquel Urtasun
TL;DR
Building instance segmentation requires accurate object borders, which CNN-based methods often distort or fuse. DSAC combines CNNs with active contour priors and structured end-to-end learning, and it outperforms state-of-the-art methods across three datasets, including a 10% relative improvement on TorontoCity.
Problem
CNN-based building instance segmentation often detects buildings but fails to preserve precise borders, spatial coverage, and geometric correctness.
Method
DSAC uses a CNN to learn local ACM energy terms and geometric priors, including the balloon term, within an end-to-end structured prediction framework.
Results
10% relative improvement over the state-of-the-art on the TorontoCity dataset was obtained among results across three building instance segmentation datasets.
Takeaways & Limitations
Combining CNN feature extraction with ACM geometric constraints is promising for instance segmentation when strong geometric priors exist.
Takeaways & Limitations
DSAC assumes its initial polygon is supplied by an external method, so initialization is not learned.
Abstract
from arXiv · showhide
The world is covered with millions of buildings, and precisely knowing each instance's position and extents is vital to a multitude of applications. Recently, automated building footprint segmentation models have shown superior detection accuracy thanks to the usage of Convolutional Neural Networks (CNN). However, even the latest evolutions struggle to precisely delineating borders, which often leads to geometric distortions and inadvertent fusion of adjacent building instances. We propose to overcome this issue by exploiting the distinct geometric properties of buildings. To this end, we present Deep Structured Active Contours (DSAC), a novel framework that integrates priors and constraints into the segmentation process, such as continuous boundaries, smooth edges, and sharp corners. To do so, DSAC employs Active Contour Models (ACM), a family of constraint- and prior-based polygonal models. We learn ACM parameterizations per instance using a CNN, and show how to incorporate all components in a structured output model, making DSAC trainable end-to-end. We evaluate DSAC on three challenging building instance segmentation datasets, where it compares favorably against state-of-the-art. Code will be made available.
1. Introduction
Building instance segmentation matters for applications requiring precise object extents, but CNN-based methods often produce geometrically inaccurate borders. DSAC combines CNNs with active contour constraints and structured learning to improve polygonal building delineation.
- Motivation: Building instance segmentation provides per-object masks needed beyond semantic segmentation’s aggregate building coverage estimates.The task must determine whether building pixels belong to the same instance.
- Motivation: CNN-based building segmentations detect buildings well but often produce inaccurate spatial coverage, distorted borders, and geometrically incorrect shapes.Buildings require sharp corners and straight walls, which CNNs generally delineate poorly.
- Active contour formulation: Active Contour Models constrain outputs to polygonal curves and optimize image evidence together with priors such as boundary continuity and smoothness.The balloon term can push vertices outward and prevent contour collapse.
- DSAC: DSAC learns locally varying ACM energy terms with a CNN, including geometric priors and the balloon term, rather than relying on fixed global parameterizations.The framework targets stronger penalties on straight regions and weaker penalties near corners.
- DSAC: DSAC formulates ACM energy learning as structured prediction and trains the CNN end-to-end using a Structured Support Vector Machine loss.The contributions explicitly include structured energy learning, the balloon term, and end-to-end learning of guiding features and local priors.
2. Related work
Prior work advances automated building extraction and instance segmentation but generally lacks direct integration of polygonal shape priors. DSAC extends active contours with CNN-learned local energy terms and structured learning for polygonal instance outputs.
- Building footprint extraction: Building extraction has progressed through LIDAR, human-assisted systems, polygonal shape priors, and deep CNNs, but building instance segmentation remains far from solved.Relevant benchmarks include SpaceNet, DSTL, and the OpenAI Challenge.
- Instance segmentation: Instance-segmentation pipelines commonly use pixel-wise CNNs for detection, mask prediction, pairwise grouping, or sequential tracing.These approaches include multi-task CNNs, location-pair likelihoods, and attention-based recurrent tracing.
- Instance segmentation: Pixel-wise methods are not well suited to directly integrating output shape priors, while polygon-producing models remain limited or tailored to different settings.Prior polygonal approaches include recursive polygon generation and CNN-guided ACM boundary directions.
- Active contours: Classical active contours impose geometric constraints but traditionally depend on simple appearance assumptions and globally weighted energy terms.The balloon force was introduced as an additional force, while DSAC expresses it within the energy formulation.
- Structured learning: DSAC learns local per-pixel weights for ACM energy terms with a CNN, enabling curvature priors and potentially other priors such as convexity.A structured loss coordinates the heavily interdependent energy terms during CNN learning.
3. Method
DSAC combines CNN-learned, locally varying active-contour energies with structured loss training so polygonal contours can incorporate image evidence, shape priors, and task-specific IoU optimization. Its ACM inference separates external data and balloon terms from internal length and curvature penalties, while remaining independent of a particular inference algorithm.
- Framework: DSAC uses a CNN to predict data, smoothness, curvature, and balloon parameterizations, followed by active-contour inference to produce each polygon.The CNN outputs D, α, β, and κ, which are passed to ACM inference; training then computes structured loss from the resulting contour.
- Optimization and implementation: The framework does not require a differentiable ACM inference algorithm, and its locally penalized implementation typically runs in under 50 ms per inference on one CPU.This supports repeated inference during CNN training while leaving the ACM inference procedure modular.
- Active contour formulation: The active contour represents a polygon whose energy combines image-dependent data and balloon terms with internal penalties for contour length and curvature.D identifies likely object boundaries, κ controls inflation, α penalizes length, and β penalizes curvature.
- Internal terms: Locally varying β values avoid the scalar-regularization trade-off between oversmoothing corners and undersmoothing other object regions.Unlike globally constant smoothness weights, β can vary by pixel according to the object part beneath the contour.
- Balloon term: DSAC generalizes the balloon term from uniform outward inflation to maximizing the integral of a learned κ map over the polygon’s enclosed region.The region integral is discretized over κ values, and its force is derived from how small node perturbations alter weighted areas.
- Structured loss: Structured prediction trains the CNN by using loss-augmented ACM inference to lower ground-truth energy and raise the energy of erroneous contours under an IoU loss.IoU optimization modifies κ negatively inside the ground truth and positively outside, generating loss-augmented examples for subgradient updates.
4. Experiments
The experiments evaluate DSAC for building footprint extraction under manual and automatic initialization across Vaihingen, Bing huts, and TorontoCity datasets. They compare learned active-contour energies with a CNN baseline and use DWT outputs to initialize contours automatically.
- Experimental settings: DSAC is evaluated with manual initialization on Vaihingen and Bing huts, and with automatic initialization on TorontoCity.Manual initialization uses a single user click near each building center; automatic initialization uses instance segmentation outputs to generate initial polygons.
- Datasets: The Vaihingen dataset contains 168 buildings at 9 cm resolution, with 100 used for training and 68 for testing.
- Datasets: The Bing huts dataset contains 605 huts at 30 cm resolution, with 335 training images and 270 test images, and is challenging because of low contrast and label noise.
- Baselines: DSAC is compared with a CNN using the same architecture and a three-class cross-entropy loss for building, boundary, and background.The user click selects the nearest connected building region as the instance prediction for the baseline.
- Automatic initialization: For automatic initialization, DWT provides initial polygons on TorontoCity, where it has high recall but tends to undersegment objects and lose boundary detail.TorontoCity contains around 28,000 training instances and 12,000 testing instances.
5. Results and discussion
DSAC improves building-instance segmentation and area estimation while learning spatially varying contour behavior. Its results favor smooth, straight edges and sharp corners, although oversmoothing and noisy ground truth remain visible.
- Manual initialization: DSAC significantly improves baseline IoU on both manually initialized datasets, while the ablation supports locally varying κ and β and retaining α as a global value.The analysis also identifies the balloon term as important for contour convergence.
- Manual initialization: Learned priors promote smooth, straight edges while often preserving sharp corners in Vaihingen segmentations.The model produces low D near corners, high κ inside buildings beside corners, and a sharp drop in κ outside.
- Manual initialization: The smoothness term β is close to 0 at corners and high along edges, encoding different contour behavior across local geometry.
- Manual initialization: Bing huts show the largest performance gain in area estimation, although DSAC oversmooths shapes because noisy labels hinder learning corner locations.The method still converges to polygons of the correct size by balancing balloon and membrane terms.
- Automatic initialization: On TorontoCity, DSAC improves both weighted coverage and PolySim over DWT and achieves the best weighted coverage regardless of initialization strategy.The strongest results use raw DWT during training and post-processed DWT polygons during testing.
6. Conclusion
DSAC embeds high-level geometric processes into deep learning for building instance segmentation, using CNN-predicted ACM energy parameters trained end-to-end. Across three datasets, it shows the promise of combining CNN feature extraction with ACM constraints when strong geometric priors exist.
- DSAC uses a CNN to predict ACM energy-function parameters so polygon outputs approach ground-truth building footprints.The model brings ACM inference into CNN training and updates CNN parameters through a structured loss and back-propagation.
- The framework embeds high-level geometric processes into deep learning for object instances with strong shape priors, such as overhead building footprints.
- Examples compare ground truth, baseline, and active-contour polygons across Vaihingen, Bing huts, and TorontoCity test buildings.The TorontoCity examples include shifted ground-truth polygons, marked by red arrows.
- 10% relative improvement over the state-of-the-art on TorontoCity was reported across results from three different datasets.The authors present this as evidence that combining CNN feature extraction with ACM constraints is promising for instance segmentation with strong geometric priors.