PolyWorld: Polygonal Building Extraction with Graph Neural Networks in Satellite Images

1. Introduction

• Segmentation-based polygonization can carry mask imperfections through a separate pipeline, producing irregular polygons.Such vector representations matter in cartography, city modelling and reconstruction, and map generation.Source: §1 · p. 1
• PolyWorld directly detects building corners and learns their connections to form valid polygons end to end.It represents the vertices and their connections through a permutation matrix.Source: §1 · p. 1

1 Graz University of Technology 2 Blackshark.ai 3 VRVis

• PolyWorld uses a GNN to propagate information across vertex embeddings, distinguish descriptors, and refine corner positions.The refinement minimizes combined segmentation and polygonal angle difference losses.Source: §1 · pp. 1–2

2. Related work

• Segmentation-mask vectorization methods can accumulate errors across sequential blocks and require careful hyperparameter tuning.The paper identifies Frame Field Learning as the field's state of the art at the time.Source: §2 · p. 2
• Most prior polygon-prediction methods directly process polygon parameters but can be difficult to train and require multiple inference iterations.The paper also notes difficulties with complex shapes, including curved walls or holes.Source: §2 · p. 2

3. The PolyWorld Architecture

• PolyWorld represents polygon vertices with a permutation matrix subject to one clockwise and one counterclockwise connection per vertex.The clockwise and counterclockwise matrices are transposes; diagonal assignments can be discarded.Source: §3 · p. 3

3.1. Vertex Detection Network

• A CNN produces a feature map and corner-detection mask; NMS selects the highest-confidence peaks for descriptor extraction.The figure caption describes N selected locations and descriptors extracted from the feature map at those locations.Source: Figure 3 · p. 3, §3.1 · p. 3

3.2. Attentional Graph Neural Network

• The attentional graph network uses vertex positions and visual descriptors to learn matching descriptors and positional offsets.The offsets refine vertices by optimizing corner angles and footprint segmentation.Source: §3.2 · pp. 3–4

3.2.1 Vertex Encoder

• A vertex encoder merges each corner's position and visual descriptor into a single representation before graph processing.It applies an MLP to their concatenation.Source: §3.2.1 · p. 4

3.2.2 Self Attention Network

• Self-attention propagates information across vertices, updating their descriptors through a fixed number of layers.The final embeddings are decomposed into matching descriptors and positional offsets.Source: §3.2.2 · p. 4

3.3. Optimal Connection Network

• The optimal connection network scores possible vertex pairs and uses Sinkhorn to produce an assignment matrix.It combines clockwise and counterclockwise connection scores for consistency; Sinkhorn normalizes rows and columns of exp(S) iteratively.Source: §3.3 · p. 5

Detection

• Corner detection is trained as a segmentation task with weighted binary cross-entropy to address the sparse corner-positive labels.The weight ω counterbalances positive samples because foreground pixels are heavily unbalanced.Source: §4 · p. 5

Matching

• The matching branch backpropagates from partial assignments to visual descriptors, using cross-entropy supervision from the ground-truth permutation matrix.Sinkhorn row and column normalization makes minimizing positive-match negative log-likelihood maximize matching precision and recall simultaneously.Source: §4 · p. 5

Positional refinement

• The angle loss penalizes differences between predicted and ground-truth angles at each polygon vertex, with σ controlling its strength.It compares angles formed by three consecutive vertices in each polygon.Source: §4 · pp. 5–6
• Segmentation loss compares the ground-truth mask with rendered predicted-polygon masks, inhibiting unwanted shape changes from angle loss and increasing segmentation scores.Differentiable Polygon Rendering [33] produces soft polygon masks, and the loss uses soft intersection over union; because NMS is nondifferentiable, minimizing these losses relies on offsets t from Equation 6.Source: §4 · p. 6

5. Implementation details

• NMS selects 256 high-confidence vertices, which are sorted during training by nearest ground-truth point to ensure index consistency with the ground-truth permutation matrix.Vertices that do not minimize distance to a ground-truth point are assigned to the permutation-matrix diagonal.Source: §5 · p. 6
• Training first pretrains vertex detection with Ldet, then trains the full architecture using Ldet + Lmatch + Langle + Lseg; inference discards diagonal-assigned vertices.The permutation matrix uses 100 Sinkhorn iterations during training and the Hungarian algorithm on CPU during inference.Source: §5 · p. 6
• 24 ms per 320 × 320 image is the average forward-pass time on an NVIDIA RTX 3090 and AMD Ryzen7 3700X.The implementation uses a Residual U-Net, D = 64 descriptors and intermediate representations, four self-attention layers with four heads each, γ = 0.05, ω = 100, σ = 10, and λ = 103 [33].Source: §5 · p. 6

6. Experiments

• Experiments use CrowdAI, with over 280k 300×300 training images and 60k test images; images are upsampled to 320 × 320 pixels.The evaluation includes IoU, MS COCO AP and AR, and Max Tangent Angle Error for contour regularity.Source: §6 · p. 6
• Complexity-aware IoU balances segmentation overlap with similarity in vertex counts, penalizing both oversimplified shapes and redundant vertices.The qualitative motivation is that raster-derived polygons can contain redundant vertices, while polygonization can reduce segmentation scores in favor of regular footprints.Source: §6 · p. 7
• PolyWorld produces cleaner, more linear geometries than FFL, including more regular footprints around occlusions, curved walls, and inner courtyards.The qualitative comparisons cover urban areas ordered by building complexity.Source: §6 · p. 7, Figure 5 · p. 7
• PolyWorld achieves state-of-the-art precision and recall on CrowdAI, and its vertex refinement improves all reported scores by a considerable margin.It extracts 4.2M test-set vertices versus 5.1M for FFL with ACM polygonization, while achieving better segmentation scores.Source: §6 · pp. 7–8, Table 1 · p. 8
• PolyWorld overtakes all FFL results on IoU, MTA, and C-IoU when vertex position refinement is enabled.The table also reports the ratio of detected to ground-truth vertices.Source: §6 · p. 8, Table 2 · p. 8

7. Limitations and future work

• Evaluation on large-scale datasets with unseen conditions remains future work; the authors identify Inria with Open Street Map annotations as a planned test.Shared corners occur in the planned setting; the authors suggest multiclass vertex detection and repeated descriptor sampling as a possible technical approach.Source: §7 · p. 8
• Buildings with holes are limited because the permutation matrix cannot bind outer and inner rings to one shape, so post-processing may be needed for multi-polygons.Source: §7 · p. 8

8. Conclusion

• PolyWorld extracts building polygons end to end from satellite and aerial images.It uses self-attention graph neural networks to match and refine detected vertices.Source: §8 · p. 8
• Solving an optimal transport problem provides strong, reliable vertex connections and implicitly avoids redundant points.Source: §8 · p. 8
• PolyWorld significantly outperforms existing building extraction approaches, producing accurate and regular footprints for geographic and cartographic applications.Source: §8 · p. 8

Satellite Images

• Additional qualitative examples compare PolyWorld with Frame Field Learning on the CrowdAI test set and show challenging and randomly sampled scenes.Source: Satellite Images · p. 11
• The supplementary material also presents failure cases of PolyWorld.Source: Satellite Images · p. 11
• An ablation study evaluates the individual components of the method.Source: Satellite Images · p. 11

Qualitative results

• Compared with FFL, PolyWorld uses fewer vertices and generates more regular contours on complex CrowdAI scenes.It also handles unusual shapes, severe occlusions, and curved walls, detecting and connecting corners precisely.Source: Qualitative results · p. 11
• CrowdAI's off-nadir images can misalign predicted polygons with building boundaries because annotations follow building bases.This is especially noted for tall buildings.Source: Qualitative results · p. 11

Failure cases

• The optimal connection network can assign corners to another building, discard false-negative corners, or connect corners to a false-positive vertex.These errors can create artifacts or leave a footprint incomplete.Source: Failure cases · p. 11

Qualitative results

• Figure 6 traces vertex detection from probability maps through NMS and the top 256 peaks to the final result; green marks valid and red discarded vertices.Source: Figure 6 · p. 11

Failure cases

• The illustrated connection errors are very rare and have limited impact on segmentation performance.Source: Failure cases · p. 11

Ablation study

• The ablation evaluates models without refinement offsets, using either score matrix alone, without the GNN, and without angle loss.Source: Ablation study · pp. 11–12
• Tables 3 and 4 report the ablation results; their captions identify CrowdAI test-set metrics, including IoU, MTA, and C-IoU.Source: Ablation study · p. 12, Table 3 · p. 12, Table 4 · p. 12
• Removing vertex-refinement offsets causes a noticeable drop in detection and segmentation performance and less regular-looking polygons.Source: Ablation study · p. 12
• Removing angle loss can yield detection scores equivalent to or higher than the full method, but produces less sharp corners and worse MTA than the full method.The resulting building shapes are visually less appealing than those from training with Langle.Source: Ablation study · p. 12
• Without the GNN, meaningful vertex connections remain possible, but missing footprints and wrong matches are not rare.Source: Ablation study · p. 12
• Using only Sclock or S⊤count leads to lower segmentation scores than combining them as S = Sclock + S⊤count.The ablation reports results in Tables 1 and 2.Source: Ablation study · p. 12

• 0.047s per image for PolyWorld is comparable to Frame Field Learning’s 0.04s on CrowdAI using the same GTX 1080Ti configuration.Source: Runtimes · p. 12
• 0.024s per image is PolyWorld’s reported runtime on an RTX 3090.Source: Runtimes · p. 12