PaperlayerView exact revision on arXiv

Source-linked AI summary

Raster2Seq: Polygon Sequence Generation for Floorplan Reconstruction

Hao Phung, Hadar Averbuch-Elor

arXiv:2602.09016v3cs.CV

TL;DR

Raster-to-vector floorplan methods struggle to capture the structure and semantics of complicated real-world layouts. Raster2Seq represents floorplans as labeled polygon sequences generated autoregressively, outperforming existing methods across geometric and semantic metrics while generalizing to challenging real-world data.

  • Problem

    Existing raster-to-vector methods struggle to capture the structure and semantics of complicated real-world floorplans.

  • Method

    Raster2Seq directly represents floorplans as labeled polygon sequences and generates them autoregressively with an anchor-based decoder.

  • Results

    Raster2Seq consistently surpasses existing methods across geometric and semantic metrics and generalizes strongly to challenging real-world datasets.

  • Takeaways & Limitations

    Raster2Seq provides a flexible framework for reconstructing complex floorplans with many rooms and diverse polygon structures.

  • Takeaways & Limitations

    The method can still mislocalize less prevalent doors and windows, producing artifacts such as cross-over windows.

Abstract

from arXiv · show

Reconstructing a structured vector-graphics representation from a rasterized floorplan image is typically an important prerequisite for computational tasks involving floorplans such as automated understanding or CAD workflows. However, existing techniques struggle in faithfully generating the structure and semantics conveyed by complex floorplans that depict large indoor spaces with many rooms and a varying numbers of polygon corners. To this end, we propose Raster2Seq, framing floorplan reconstruction as a sequence-to-sequence task in which floorplan elements--such as rooms, windows, and doors--are represented as labeled polygon sequences that jointly encode geometry and semantics. Our approach introduces an autoregressive decoder that learns to predict the next corner conditioned on image features and previously generated corners using guidance from learnable anchors. These anchors represent spatial coordinates in image space, hence allowing for effectively directing the attention mechanism to focus on informative image regions. By embracing the autoregressive mechanism, our method offers flexibility in the output format, enabling for efficiently handling complex floorplans with numerous rooms and diverse polygon structures. Our method achieves state-of-the-art performance on standard benchmarks such as Structure3D, CubiCasa5K, and Raster2Graph, while also demonstrating strong generalization to more challenging datasets like WAFFLE, which contain diverse room structures and complex geometric variations.

1 Introduction

Rasterized floorplans lose the structured geometry and semantics of their vector representations, while existing raster-to-vector methods struggle to recover these properties in complex indoor spaces. Raster2Seq addresses this by autoregressively generating labeled polygon sequences and demonstrates consistent gains across multiple reconstruction benchmarks and metrics.

  • Motivation: Rasterization strips structured geometric and semantic information from floorplans originally represented as vector graphics.Floorplans define the structure and semantics of indoor spaces but are commonly distributed as rasterized images.
  • Problem: Existing raster-to-vector methods remain challenged in capturing the structure and semantics conveyed by complex floorplans.Prior work has explored converting floorplan images back to vectorized formats, including Transformer-based approaches.
  • Approach: Raster2Seq represents floorplan elements as labeled polygon sequences to transform rasterized images into vectorized formats.The representation jointly captures floorplan structure and semantic attributes through polygon sequences.
  • Approach: Raster2Seq autoregressively outputs polygon sequences, directly modeling spatial structure and semantic attributes without a fixed-query budget constraint.The framework leverages the left-to-right generation bias of masked attention models to model floorplan elements as a sequence.
  • Results: Raster2Seq consistently surpasses existing methods across geometric and semantic metrics on multiple floorplan reconstruction benchmarks and input settings.Experiments consider rasterized RGB images and 2D density maps, with larger performance gaps for floorplans containing more corners and rooms.

2 Related Work

Prior floorplan reconstruction methods progressed from handcrafted, multi-stage pipelines to neural and end-to-end approaches, but often used different modalities or struggled with layout diversity. Raster2Seq instead represents floorplans as semantic polygon sequences and uses seq2seq-style autoregressive prediction with semantic-aware training.

  • Raster-to-vector floorplan reconstruction: Early raster-to-vector systems used handcrafted features and multi-step geometric recovery, while Liu et al. introduced neural corner prediction followed by integer programming.These approaches detected components such as walls or predicted corner representations before recovering geometric primitives.
  • Raster-to-vector floorplan reconstruction: Point-cloud and panorama methods address related reconstruction tasks, while Floor-SP and MonteFloor use instance segmentation with optimization and generalize poorly to diverse layouts.These methods differ from rasterized floorplan-image reconstruction in input modality or rely on multi-stage processing.
  • Semantic integration: Unlike work focused solely on structural prediction, Raster2Seq incorporates semantic labels alongside spatial coordinates to produce structural and semantic vectorized floorplans.The semantic objective improves the utility of vectorized floorplans by providing both types of information.
  • Semantic integration: RoomFormer can lose fine-grained semantics through padding-inclusive averaged corner embeddings, whereas Raster2Graph adds complexity by predicting four neighboring rooms.Both methods also integrate semantics, but the cited limitations distinguish their representations or prediction procedures from Raster2Seq.
  • Sequence-to-sequence modeling: Raster2Seq applies seq2seq modeling to floorplan reconstruction by decoding labeled polygon sequences with image features, learnable anchors, and previously generated tokens.Its anchor-based autoregressive decoder predicts the next token and uses special <SEP> tokens to separate polygons.

3 Method

Raster2Seq reconstructs rasterized floorplans as sequences of semantically labeled polygons rather than through intermediate geometric elements. Its anchor-based autoregressive decoder predicts polygon tokens sequentially from image features, coordinate tokens, and learnable anchors, with geometry, token types, and semantics jointly supervised.

  • Labeled polygon representation: Raster2Seq directly represents rooms and other floorplan entities as sequences of labeled corner tokens encoding spatial positions and semantic probabilities.Each corner token is c_i = (x_i, y_i, p_i), with p_i representing probabilities over C semantic categories.
  • Labeled polygon representation: Multiple polygon sequences are concatenated with <SEP>, while <BOS> and <EOS> delimit the overall floorplan sequence.Each token also has a probability vector over <CORNER>, <SEP>, and <EOS> types; <BOS> is omitted from token-type modeling.
  • Anchor-based autoregressive decoder: The anchor-based autoregressive decoder consumes image features, coordinate tokens, and learnable anchors to predict labeled polygon sequences.Coordinates are represented through quantized token embeddings, while the decoder learns residuals relative to spatial anchors instead of directly regressing continuous coordinates.
  • Anchor-based autoregressive decoder: Its autoregressive block combines masked attention, deformable attention, and a feed-forward layer, with deformable attention sampling sparse regions around reference points.Masked attention enforces left-to-right generation, while deformable attention avoids attending to the entire feature map.
  • Training and inference: Training combines L1 coordinate regression with token-type and semantic cross-entropy losses, while inference generates tokens until <EOS> and assigns polygon labels by majority voting.During training, polygons are ordered top-to-bottom and left-to-right by top-left coordinates to induce geometric structure and topological relationships.

4 Experiments

Experiments evaluate Raster2Seq across four floorplan datasets using geometric and semantic reconstruction metrics, with quantitative, qualitative, robustness, generalization, and ablation analyses. The results show strong performance across datasets and increasing advantages on complex layouts, while qualitative analysis identifies remaining challenges for window and door localization.

  • Experimental setup: Experiments use Structured3D-B, CubiCasa5K, Raster2Graph, and WAFFLE, with WAFFLE evaluated for zero-shot generalization.Structured3D is converted into binary raster images, while WAFFLE is evaluated only for zero-shot generalization.
  • Experimental setup: Evaluation measures geometric Room, Corner, and Angle performance alongside semantic Room and Window & Door predictions.The default reporting uses F1 scores, while WAFFLE uses IoU for room prediction in the segmentation task.
  • Quantitative results: Raster2Seq achieves state-of-the-art structural and semantic performance across the evaluated raster-to-vector datasets.Its token-level semantic optimization avoids the semantic dilution attributed to RoomFormer’s uniform-length sequences.
  • Robustness and generalization: Raster2Seq maintains strong and stable performance across datasets, whereas PolyRoom and FRI-Net vary substantially between simpler and more complex floorplans.The comparison attributes baseline weaknesses to segmentation proposals and line assembly when polygon lengths and shapes become diverse.
  • Robustness and generalization: Raster2Seq is more robust as floorplan complexity increases and demonstrates the strongest generalization across same-dataset and cross-dataset evaluations.RoomFormer and FRI-Net show notable performance drops in complex scenes with over 15 polygons or 150 corners, while Raster2Seq outperforms baselines across settings.
  • Qualitative analysis and ablation: Qualitative comparisons show superior reconstruction quality and robustness across diverse layouts, while ablations find that learnable anchors and polygon generation ordering contribute meaningfully.Raster2Graph often fails to recover complete structures, and the model occasionally mislocalizes windows and doors, producing artifacts such as cross-over windows.

5 Conclusion

Raster2Seq frames raster-to-vector floorplan conversion as sequence-to-sequence prediction of labeled polygon sequences, driven by an anchor-based autoregressive decoder. The approach shows promising generalization to in-the-wild floorplans and motivates future semantic controls and downstream architectural applications.

  • Core framework: Raster2Seq represents vectorized floorplans as labeled polygon sequences predicted by an anchor-based autoregressive decoder conditioned on previously generated corners.The decoder integrates learnable anchors and additional architectural components to guide next-corner prediction.
  • Generalization and future work: Raster2Seq demonstrates promising generalization to in-the-wild Internet data, supporting future modeling of historical buildings from hand-drawn floorplans.The paper identifies appearance-based augmentations as one possible mechanism for improving out-of-distribution performance.
  • Generalization and future work: Open-vocabulary predictions could enable reconstruction of richer semantics and allow room semantic labels to guide decoding toward desired room coordinates.This proposed control mechanism is presented as future work for diverse real-world floorplans.
  • Downstream applications: Accurate vectorized floorplans could support controllable downstream applications such as floorplan-guided 3D generation of large architectural scenes.The paper connects this potential to increasingly powerful generative models and applications beyond traditional analysis and editing.
  • Qualitative results: Qualitative comparisons show more accurate reconstructions than Raster2Graph, which often produces incomplete results, and stronger generalization than RoomFormer on real-world Internet floorplans.The RoomFormer comparison uses models trained on CubiCasa5K and evaluates WAFFLE floorplan images.

A Data preparation … A.3 Raster2Graph

The data preparation converts Structured3D density maps and CubiCasa5K segmentation annotations into polygon-based RGB floorplan representations, while Raster2Graph data follows its existing preprocessing codebase.

  • A.1 Structured3D: Structured3D density maps projected from 3D point clouds are converted into RGB-format floorplan images using accompanying annotations.The conversion is intended to better mimic standard RGB floorplans, typically presented in black-and-white format.
  • A.1 Structured3D: The Structured3D conversion uses annotations associated with the original density maps.These annotations support conversion from the projected density-map representation into RGB images.
  • A.1 Structured3D: Structured3D raster images are synthetically generated by a rendering engine and differ substantially from real-world images in appearance.The RGB conversion is described as a way to better mimic standard RGB floorplans despite this synthetic origin.
  • A.2 CubiCasa5K: CubiCasa5K segmentation maps are converted into polygons whose corners serve as the real-value corners for each room.The dataset was originally proposed for segmentation using pixel-wise segmentation maps.
  • A.2 CubiCasa5K: The CubiCasa5K preparation selects semantic room classes including Outdoor, Wall, Kitchen, Living Room, Bed Room, Bath, Entry, Railing, Storage, Garage, and Undefined.Window and Door are included as two additional classes.
  • A.2 CubiCasa5K: The CubiCasa5K preparation includes polygons derived from the selected room, Window, and Door segmentation classes.These corresponding classes are converted from segmentation maps into polygon representations.
  • A.3 Raster2Graph: Raster2Graph data is preprocessed following the Raster2Graph codebase.The preprocessing procedure is explicitly based on Hu et al. 2024’s implementation.

B Additional Implementation Details

Raster2Seq uses a ResNet-50-plus-transformer image extractor, bilinear coordinate quantization, and an autoregressive encoder–decoder configured for variable-length polygon sequences. Learnable anchors and decoder KV caches support token-aware generation while reducing repeated computation.

  • Image Feature Extractor: The image feature extractor combines a ResNet-50 backbone with a transformer encoder and is fully fine-tuned end-to-end from ImageNet-pretrained initialization.The resulting image feature vector feeds the autoregressive decoder for polygon sequence generation.
  • Bilinear Quantizer: A learnable codebook discretizes 2D continuous coordinates into 1D embeddings, which the decoder uses with encoded image features to regress continuous coordinates.The codebook has size R H_b×W_b×D, where H_b×W_b is the number of quantization bins and D is the embedding dimension.
  • Model configs: The model has 12 total layers split evenly between encoder and decoder, pads variable-length inputs to fixed length L, and uses 256-channel representations with 512 learnable anchors.The causal attention layer with post-fusion precedes deformable attention, and the anchor count matches the input length.
  • Model configs: Decoder KV caches store previous-token keys and values to eliminate recomputation during each autoregressive generation loop.This mechanism is introduced to accelerate generation.
  • Learnable Anchors: Anchors are learnable for corners and special tokens, while special-token positions omit coordinate loss and use token-type classification to control room and generation termination.At inference, special-token coordinates are ignored; their token types identify the end of a room sequence or the end of generation.

C Training details · D Labeled Polygon Sequence Generation

The method uses RoomFormer-compatible training settings, staged pretraining and semantic-loss fine-tuning, and an ordering strategy that appends window and door coordinates after room polygons. During inference, it generates labeled polygon corners autoregressively, while FRI-Net training uses binary occupancy maps derived from room polygons.

  • C Training details: The model adopts RoomFormer’s hyper-parameters for training.Experiments run on a single NVIDIA A6000 GPU and take approximately 1–2 days.
  • C Training details: 1,400 pretraining epochs are used on Structured3D, compared with 500 on CubiCasa5K.These are dataset-specific pretraining schedules.
  • C Training details: 450 additional fine-tuning epochs are used on both Structured3D and CubiCasa5K with semantic loss.Fine-tuning follows the respective pretraining stages.
  • C Training details: The default loss weights are λ_coord = 20 and λ_sem = 1, while λ_token varies by dataset.The coordinate and semantic weights remain fixed by default; the token weight is dataset-specific.
  • C Training details: Appending window and door coordinates after room polygons substantially improves model performance.This ordering reflects pretraining exposure to room coordinates before fine-tuning introduces windows and doors.
  • C Training details: FRI-Net training supervises binary occupancy maps generated from ground-truth room polygons without requiring point clouds.Pixels inside rooms are labeled 1 and pixels outside are labeled 0.
  • D Labeled Polygon Sequence Generation: During inference, Algorithm 1 iteratively predicts the next point in the polygon sequence.The corner generation process is autoregressive over sequence points.

E Additional Results · E.1 Full performance comparison. · E.2 Zero-shot performance on unseen WAFFLE.

The additional results compare Raster2Seq across Structured3D-B, CubiCasa5K, and Raster2Graph, and evaluate zero-shot transfer to WAFFLE. Raster2Seq delivers strong geometric and semantic reconstruction, while achieving the best reported WAFFLE segmentation results.

  • E Additional Results: The section reports comprehensive comparisons on Structured3D-B, CubiCasa5K, and Raster2Graph, alongside zero-shot evaluation on the WAFFLE segmentation benchmark.It also covers standard and noisy density maps, trade-offs with RoomFormer, and speed comparisons against baselines.
  • E Additional Results: Qualitative results are provided for Structured3D-B, CubiCasa5K, Raster2Graph, zero-shot WAFFLE, and VLM-based refinement.The paper directs readers to an accompanying interactive tool for additional visualizations.
  • E.1 Full performance comparison.: Raster2Seq achieves superior geometric performance on Structured3D-B, CubiCasa5K, and Raster2Graph using the Room and Corner metrics.Detailed results appear in Tables 9, 10, and 11, respectively.
  • E.1 Full performance comparison.: Raster2Seq also demonstrates strong semantic floorplan reconstruction across the reported benchmark comparisons.The authors attribute this performance to labeled polygon representation and token-wise classification loss.
  • E.2 Zero-shot performance on unseen WAFFLE.: 73.9 IoU, 81.6 precision, and 88.6 recall are achieved by Raster2Seq on the unseen WAFFLE test set after training on CubiCasa5K without WAFFLE exposure.These are the highest overall scores reported in the comparison.
  • E.2 Zero-shot performance on unseen WAFFLE.: RoomFormer records 65.7 precision and 60.5 IoU on WAFFLE, trailing Raster2Seq and indicating less reliable predictions.The comparison uses interior segmentation on the WAFFLE test set.

E.3 Performance on Structure3D-Density maps · E.4 Runtime comparison

On Structure3D density maps, Raster2Seq generally surpasses baselines on geometric metrics, delivers stronger room semantics, and remains robust under noisy inputs. Its inference speed is comparable to Raster2Graph, while its training throughput is highest among the compared methods.

  • E.3 Performance on Structure3D-Density maps: Raster2Seq generally outperforms existing baselines on key geometric metrics, including Room and Angle, when trained and tested with density-map inputs.The comparison uses the standard Structured3D benchmark.
  • E.3 Performance on Structure3D-Density maps: FRI-Net achieves competitive density-map performance, but its image-input performance is generally lower than with density maps.The passage attributes this difference to FRI-Net’s reliance on disentangled representations of raw images.
  • E.3 Performance on Structure3D-Density maps: Raster2Seq surpasses RoomFormer on room semantic criteria but obtains slightly lower scores for window and door semantics.When semantic room types are included, RoomFormer’s performance drops by 2–5 points.
  • E.3 Performance on Structure3D-Density maps: With 20% dropout masking, Raster2Seq’s RoomF1 drops 1 point from 98.7, compared with RoomFormer’s 2.6-point drop from 97.5.The masking is applied to randomly remove projected density signals during both training and testing.
  • E.4 Runtime comparison: Raster2Seq’s sampling time is comparable to Raster2Graph at 0.52s versus 0.57s, but slower than RoomFormer at 0.04s.The comparison reports sampling time, training throughput, and training time.
  • E.4 Runtime comparison: 63 images/s gives Raster2Seq the highest training throughput, exceeding RoomFormer at 24 images/s and Raster2Graph at 34 images/s.This throughput advantage enables faster training in low-resource settings despite the inference-time trade-off.

E.5 VLM-based refinement · E.6 Downstream application

E.5 shows that VLM-based refinement can enforce geometric constraints using Raster2Seq’s semantic polygon representation, while E.6 demonstrates that vectorized floorplans support controllable 3D scene generation with structural consistency.

  • E.5 VLM-based refinement: Raster2Seq does not directly enforce geometric constraints, and noisy CubiCasa5K annotations with overlapping rooms can produce corresponding artifacts.The limitation motivates the VLM-based refinement experiment.
  • E.5 VLM-based refinement: The method is evaluated across Structured3D, CubiCasa5K, and Raster2Graph test sets, with additional Structured3D evaluation using density-map inputs and refinement compatibility.The supplied tables report these quantitative evaluations, including speed comparison on a single A6000 GPU and refinement results on a CubiCasa5K subset.
  • E.5 VLM-based refinement: The refinement VLM receives labeled polygons, rasterized inputs, visual overlays, standalone vectorizations, and a prediction-derived adjacency graph.The adjacency graph provides relations between room instances for targeted refinement.
  • E.5 VLM-based refinement: Corner and Angle scores increased from 54.0 to 59.0 and 33.0 to 45.1, respectively, on 30 randomly selected CubiCasa5K test samples.Qualitatively, the refined vectorized floorplans exhibit tight gaps between adjacent rooms.
  • E.5 VLM-based refinement: Semantic labels enable room-instance identification and adjacency recognition, while removing them causes approximately a 3-point drop in both Corner and Angle metrics.These semantic cues are therefore important for effective VLM-based refinement.
  • E.6 Downstream application: Vectorized rasterized floorplans enable downstream computational tasks by converting boundary geometry into a coarse 3D volume through vertical extrusion.This volume provides spatial guidance for a pretrained 3D generative model.
  • E.6 Downstream application: Using TRELLIS with the test-time approach from SpaceControl, vectorization helps a 3D generative model reproduce complex architectural layouts from a single RGB image while maintaining global structural consistency.Figure 17 presents examples of this controllable 3D scene-generation application.

F Additional Ablation Studies

Ablations identify 32×32 quantization, sequence length 512, coordinate-loss coefficient 20, and jointly trained anchors as strong configurations. One-stage training remains comparable to two-stage training, while rasterization loss provides only marginal gains and is omitted.

  • Quantization resolution: 32×32 quantization achieves the best overall performance, reaching Room F1 96.3, Corner F1 93.7, and Angle F1 82.6.Both coarser 16 × 16 and finer 64 × 64 resolutions reduce performance.
  • Random vs. Learnable anchors: Jointly trained learnable anchors significantly improve overall measurements, whereas randomly initialized anchors barely improve over the no-anchor baseline.The comparison demonstrates the importance of learning anchor coordinates with the model parameters.
  • Sequence length: Sequence length 512 is the best-performing configuration, with lengths above 256 improving reconstruction quality by 3–10 points across metrics.The results indicate that length 512 balances structural and geometric detail capture effectively.
  • Coordinate coefficient: Setting the coordinate loss coefficient to 20 yields Room F1 96.3, Corner F1 93.7, and Angle F1 82.6, outperforming coefficients 10 and 40.The token loss coefficient is fixed at 1 in this ablation.
  • Training and rasterization loss: One-stage training achieves comparable scores to two-stage training, with RoomSemanticF1 76.9 vs. 76.1, while rasterization loss yields only marginal gains and is omitted.On Structured3D-B, rasterization loss improves CornerF1 by 0.2 and AngleF1 by 1.0; on CubiCasa5K, CornerF1 changes from 59.4 to 59.8 and AngleF1 from 37.4 to 37.9.
Loading 2602.09016v3…