Source-linked AI summary
GeBDA: Building Damage Assessment as Text-Based Sequence Prediction
Olivier Dietrich, Krishna Sapkota, Konrad Schindler, Genady Beryozkin
TL;DR
Building damage assessment needs scalable localization and grading after disasters, yet GeoVLMs have not demonstrated the full joint task with standard autoregressive decoding. GeBDA fine-tunes Gemma to generate quantized building boxes and damage labels from bi-temporal imagery and a prompt, achieving promising localization and classification results while retaining representation and generalization limitations.
Problem
Manual damage mapping is too slow to scale, while existing GeoVLMs simplify dense BDA or delegate localization instead of jointly grading every building.
Method
GeBDA fine-tunes a general-purpose Gemma VLM to autoregressively generate a variable-length sequence of quantized building bounding boxes and damage labels from bi-temporal images and a prompt.
Results
GeBDA is competitive in localization and classification on optical imagery; rasterized localization reaches F_loc1 78.34 on xBD and 81.23 on Bright.
Takeaways & Limitations
The results support using a general-purpose VLM for end-to-end BDA, with strong localization and promising damage classification from post-event optical images.
Takeaways & Limitations
Plain-text coordinate encoding is not token-efficient, compute-heavy fine-tuning limits backbone scaling, and benchmark splits share disaster events.
Abstract
from arXiv · showhide
Conventionally, Building Damage Assessment (BDA) is tackled either with dedicated network architectures or by fine-tuning geospatial image foundation models. In this work, we ask whether a general-purpose Vision-Language Model (VLM) can localize buildings and grade their damage through autoregressive sequence generation alone. We cast BDA as predicting a variable-length set of bounding boxes, each specified by its coordinates and a damage label. Our preliminary implementation, based on the open Gemma model, achieves promising damage mapping results from only bi-temporal satellite images and a suitable text prompt.
1 Introduction
Building damage mapping is important for disaster response, but manual inspection does not scale. GeBDA tests whether a general-purpose VLM can jointly localize buildings and classify damage through single-pass autoregressive prediction.
- Rapid building damage mapping supports disaster response and recovery, while manual inspection is prohibitively slow and unscalable.
- GeoVLMs struggle with dense BDA localization because autoregressive decoders must generate long sequences of precise coordinates and may drift or hallucinate.
- Existing GeoVLM approaches simplify BDA or delegate localization, leaving joint single-pass localization and grading of every building undemonstrated.
- GeBDA fine-tunes a Gemma model to directly produce building locations and damage classes from bi-temporal images and a task-specific prompt.
- Preliminary results suggest GeBDA is competitive in both localization and classification using optical images.
2 Related Work
BDA traditionally uses specialist dense-prediction networks, while GeoVLM research has not demonstrated full joint localization and grading. GeBDA follows an end-to-end generative alternative based on serialized spatial predictions.
- Specialist BDA networks process pre- and post-disaster image pairs to output dense masks that localize and grade buildings.
- Generalist VLM localization commonly uses external decoders, plain-text bounding-box serialization, or dedicated coordinate tokens.
- Sequence-based localization preserves end-to-end generation, but autoregressive formulations share a critical failure mode in dense environments.
- Existing GeoVLMs simplify full BDA through the dense-localization bottleneck rather than jointly localizing and grading every building.
- GeBDA’s formulation transforms building polygons into quantized, serialized bounding-box strings with damage labels and predicts one target sequence.
3 Method
GeBDA reformulates joint building localization and damage assessment as autoregressive instance-sequence prediction. It serializes quantized box coordinates and damage classes, fine-tunes Gemma, and evaluates both instance and rasterized outputs.
- BDA is formulated as instance-based prediction rather than pixel-level classification to match the autoregressive next-token operation of VLMs.
- 3.1 Problem Formulation: Given pre- and post-disaster images and a prompt, the model generates a discrete sequence representing all scene buildings after converting polygons to boxes.
- 3.1 Problem Formulation: Coordinates are uniformly quantized to a 1000 × 1000 grid before serialization into text alongside each building’s damage class.
- 3.1 Problem Formulation: The target sequence concatenates all building strings, tokenizes them with the native tokenizer, and appends an <EOS> token.
- 3.1 Problem Formulation: Plain-text coordinates preserve Gemma’s pretrained numerical representation and implicit geometric priors, but use tokens less efficiently.
- Training and Inference: GeBDA combines a Transformer backbone with a SigLIP 2 vision encoder, freezes the vision encoder, and fine-tunes the remaining model end-to-end.
- Evaluation: Localization matches predicted and ground-truth boxes with Hungarian assignment at IoU 0.5, while classification computes F1 only on correctly matched instances.
- Evaluation: Predictions are rasterized into dense masks for comparison with pixel-based models, while an Oracle quantifies the geometric ceiling from box representation.
4 Experiments and Results
GeBDA is evaluated on xBD and Bright using instance-based and rasterized pixel-based comparisons, with qualitative analyses of localization, classification, and zero-shot generalization. Results show strong localization, competitive optical-image classification, substantial degradation on Bright SAR imagery, and improved Bright classification when jointly trained.
- Datasets: GeBDA is trained and evaluated on xBD and a subset of Bright with available instance labels, excluding unclassified buildings and those smaller than 16 pixels.xBD contains intact, minor-damage, major-damage, and destroyed classes; Bright contains intact, damaged, and destroyed classes.
- Experimental setup: Each 1024×1024 image is cropped into 512×512 patches, limiting sequence length while enabling up to 200 buildings per patch.The configuration uses up to 19 tokens per building and a vision-encoder token budget of 280.
- Localization: Predicted and ground-truth building counts show near-perfect correlation, although dense patches can produce clusters of tiny false-positive boxes and ambiguous boundaries can lead to multiple boxes for one building.These artifacts help explain why qualitative localization appears stronger than instance-based metrics.
- Damage classification: Fcls1 drops from 76.62 on xBD to 48.99 on Bright, which the authors attribute to the modality gap in Bright’s post-event SAR imagery.The frozen RGB-pretrained vision encoder struggles to extract robust discriminative features from the SAR data.
- Cross-dataset training: Joint training on xBD and Bright yields no improvement overall except a +22.6% increase in Bright Fcls1.The authors hypothesize that xBD transfers modality-agnostic priors and grounding in shared optical pre-event imagery to Bright.
5 Discussion & Conclusion
GeBDA frames building damage assessment as autoregressive sequence prediction, showing strong optical-image localization and promising damage classification while exposing dense-scene, SAR, and representation limitations. Future work targets more efficient representations, parameter-efficient scaling, spatially aligned optimization, and cross-disaster generalization.
- GeBDA frames BDA as autoregressive sequence prediction and achieves strong localization and promising damage classification with post-event optical images.The localization capability is tempered by the bounding-box representation.
- A frozen vision encoder fails to extract discriminative features from post-event SAR imagery, motivating vision-encoder fine-tuning.
- Dense patches can trigger coordinate-generation loops, likely because training caps sequences at 200 buildings, while heterogeneous tiles can produce a single predicted damage class.
- Plain-text bounding-box coordinates are token-inefficient, and fully supervised fine-tuning limits backbone scaling; larger models and LoRA are suggested alternatives.The authors identify Gemma4-26B with a 500M-parameter vision encoder as a potentially beneficial but currently impractical scale-up.
- Future work will investigate reinforcement learning for spatial metrics such as bounding-box IoU and test generalization across disasters, whose splits currently share disaster events.