PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-World Knowledge-Guided Change Data Synthesis for Remote Sensing

Yaoyi Qi, Xingxing Weng, Chao Pang, Yongkang Cui, Xiangyu Hao, Xiaokang Zhang, Guibo Zhu, Gui-Song Xia

arXiv:2608.24263v1cs.AIcs.CV

TL;DR

Existing change synthesis relies on handcrafted rules with limited transition coverage and inflexible designs. KnowChange uses pretrained vision-language models to reason about plausible locations and transitions, then combines this simulation with generalizable synthesis models. Its datasets consistently outperform existing synthetic datasets across building and semantic change detection benchmarks, while knowledge-guided simulation also improves existing pipelines.

  • Problem

    Handcrafted change-simulation rules cover limited class transitions and fixed patterns, restricting synthesized-data diversity and flexibility across change scenarios.

  • Method

    KnowChange uses pretrained vision-language models to infer plausible change locations and class transitions, then combines this reasoning with generalizable layout-to-mask and mask-to-image models.

  • Results

    6.64 average IoU gain on four building change detection benchmarks and 6.78 average F1 gain on two semantic change detection benchmarks were achieved over existing synthetic datasets.

  • Takeaways & Limitations

    Knowledge-guided change simulation supports flexible synthesis of diverse change types and can be integrated into existing synthesis pipelines to improve downstream change detection.

Abstract

from arXiv · show

Change data synthesis provides a cost-effective solution for expanding training data and improving the performance of change detection models. However, existing synthesis methods typically rely on handcrafted rules to simulate changes, where limited coverage of class transitions restricts the diversity of synthesized data, while predefined transition designs limit their flexibility in accommodating varied change types. In this work, we introduce KnowChange, a knowledge-guided change data synthesis framework that leverages pretrained vision-language models as knowledge sources to reason about plausible change locations and class transitions from pre-change scenes and desired change types. By integrating knowledge-guided change simulation with generalizable synthesis models, KnowChange enables flexible synthesis of diverse change types within a unified framework. Extensive experiments demonstrate that KnowChange-generated data consistently outperforms existing synthetic datasets in both synthetic-to-real transfer and synthetic data augmentation, despite being generated at a compact scale. Further analyses show that the knowledge-guided change simulation can be seamlessly integrated into existing synthesis pipelines and enhance the downstream utility of synthesized data.

1 Introduction

KnowChange addresses the limited transition coverage and fixed designs of rule-based change simulation by using pretrained vision-language models to infer plausible changes and generalizable synthesis models to generate diverse change data. Its synthetic datasets improve downstream change detection performance over existing synthetic datasets.

  • Limitations of Existing Methods: Existing synthesis methods simulate future changes with handcrafted transition rules and region-level operations such as copy-paste.These rules determine both where changes occur and which class transitions take place.
  • Limitations of Existing Methods: Limited rule coverage misses real-world transitions, such as forest-to-building changes, reducing the diversity of synthesized data.Existing rules may cover bareland, rangeland, and developed land transitioning into buildings but omit other land-cover categories.
  • Limitations of Existing Methods: Predefined transition patterns cannot flexibly accommodate application-specific changes, requiring rule redesign and image-synthesis adaptation for new scenarios.Examples include building construction or demolition for urban monitoring and road-related changes for transportation monitoring.
  • KnowChange Framework: KnowChange prompts pretrained vision-language models to infer plausible change locations and class transitions from pre-change scene context and user-specified change types.The model produces a global layout of the post-change semantic mask, eliminating manually predefined transition rules.
  • KnowChange Framework: A layout-to-mask model refines changed-object shapes and local context compatibility, while a mask-to-image model generates post-change images from synthesized semantic masks.Generalizable training strategies and a multi-category semantic segmentation dataset support diverse changes without additional retraining.
  • Results: 6.64 average IoU gain on four building benchmarks and 6.78 average F1 gain on two semantic benchmarks show that KnowChange data outperforms existing synthetic datasets.Ablations also show that knowledge-guided simulation can improve existing pipelines such as HySCDG and Changen2.

2 KnowChange

KnowChange combines VLM-based reasoning about plausible changes with generalizable semantic-guided synthesis to generate diverse post-change images and masks from pre-change scenes and desired change types. Its pipeline supports actual and pseudo changes, refines change layouts into semantic masks, and synthesizes post-change imagery.

  • Knowledge-Guided Change Simulation: KnowChange prompts a pretrained VLM to infer plausible change regions and post-change categories, producing a global change layout from pre-change scene information.The inputs include a pre-change image, its semantic mask, and desired change categories or textual prompts.
  • Knowledge-Guided Change Simulation: Actual changes use shape-preserving and shape-altering transitions to represent both category changes within existing regions and newly instantiated regions.Shape-preserving transitions replace semantic labels, whereas shape-altering transitions select candidate boxes and assign post-change categories.
  • Knowledge-Guided Change Simulation: Pseudo-change simulation models appearance differences that preserve semantic categories and guides category-preserving reconstruction during post-change image synthesis.The VLM identifies potentially affected categories and estimates region perturbation ratios for pseudo-change regions.
  • Generalizable Semantic-Guided Synthesis: The L2M model refines change layouts into pixel-level post-change semantic masks, while the M2I model synthesizes post-change images conditioned on the resulting masks and pre-change imagery.The semantic mask integrates shape-preserving and shape-altering masks; the re-rendering mask combines actual and pseudo-change regions.
  • Generalizable Semantic-Guided Synthesis: KnowChange trains its synthesis models on a consolidated semantic segmentation corpus to improve generalization across varied semantic categories.The corpus combines OpenEarthMap, FLAIR, Vaihingen, Potsdam, GID, and SkySA; L2M and M2I learn rich object priors.
  • KnowChange: A single trained framework flexibly synthesizes diverse user-specified change types without repeatedly redesigning transition rules or adapting synthesis models.Using 26 semantic categories from OpenEarthMap and FLAIR, KnowChange produces Know-BCD, Know-SEC, and Know-HR.

3 Experiments

Experiments evaluate KnowChange across six change-detection benchmarks, showing strong synthetic-to-real transfer, augmentation benefits, data-scaling effects, and compatibility with existing synthesis pipelines. Ablations further assess knowledge-guided simulation and VLM selection.

  • Evaluation Setup: KnowChange-generated data is evaluated on four BCD and two SCD benchmarks using F1-score, IoU, mIoU, SCS, and SeK.ChangeFormer is used for BCD and Change3D for SCD downstream evaluation.
  • Synthetic Data Augmentation: 3.67-point average IoU and 4.87-point average F1 are the largest gains from Know-BCD augmentation with 5% real training data.Know-BCD consistently outperforms other synthetic datasets across the four BCD real-data settings.
  • Synthetic Data Quality: KnowChange-generated data achieves lower FID and KID scores than existing synthetic data, indicating better alignment with real-world change data.The quality analysis uses SECOND as the reference distribution.
  • Plug-and-Play Integration: Replacing existing change-mask generation with KnowChange simulation yields a 23-point mIoU gain on SECOND and over 21-point IoU gains on LEVIR-CD and WHU-CD.Image-generation models remain unchanged in this plug-and-play integration.
  • Scaling Analysis: Increasing Know-BCD data from 1% to 5% brings the largest gains, while further scaling improves performance on three BCD benchmarks and Know-SEC scaling benefits SCD metrics.LEVIR-CD shows a temporary drop at 75% data before recovering with the full dataset.

4 Conclusion

KnowChange uses pretrained vision-language models to reason about plausible change regions and class transitions, then combines this simulation with generalizable synthesis models. Its datasets improve transfer and augmentation over existing synthetic data, while the simulation also works as a plug-and-play component.

  • 4 Conclusion: KnowChange replaces rule-based simulation with VLM reasoning about plausible change regions and post-change categories from scene contexts and desired changes.The framework combines this reasoning with layout-to-mask and mask-to-image models.
  • 4 Conclusion: KnowChange-generated datasets improve synthetic-to-real transfer and synthetic data augmentation over existing synthetic datasets.The framework produces datasets for building and semantic change detection.
  • 4 Conclusion: Knowledge-guided change simulation can be integrated into existing synthesis methods in a plug-and-play manner.The conclusion reports this alongside ablation evidence on the simulation’s importance.

A Change Data Synthesis Details

The synthesis details define VLM-guided transition modes for preserving or altering object shapes, plus pseudo-change simulation for appearance variation without semantic-category changes. Coarse VLM outputs are refined into masks and used to guide image synthesis.

  • Shape-Preserving Transitions: Shape-preserving prompts provide pre-change imagery and masks, category-color mappings, candidate regions, and desired categories for VLM reasoning.The VLM selects regions and assigns post-change categories, producing region-category pairs.
  • Shape-Altering Transitions: Shape-altering transitions let the VLM select candidate boxes and assign post-change categories, after which layout-to-mask refinement produces context-compatible object shapes.Newly instantiated regions may differ in shape from pre-change regions.
  • Pseudo-Change Simulation: Pseudo-change simulation preserves semantic categories while varying appearance for imaging differences such as illumination, season, atmosphere, or sensor variation.The VLM identifies susceptible categories and estimates region perturbation ratios.
  • Generated Datasets: KnowChange synthesizes examples for Know-BCD, Know-SEC, and Know-HR from OpenEarthMap and FLAIR source images.Know-BCD targets building change detection, while Know-SEC and Know-HR target semantic change detection using different category sets.

B.1 Data Source

The evaluation covers six change-detection benchmarks spanning building and semantic change detection, with varied geographic, seasonal, and change-pattern conditions. Dataset examples are synthesized from OpenEarthMap and FLAIR source images.

  • Evaluation Datasets: The benchmark suite contains four BCD datasets—LEVIR-CD, WHU-CD, DSIFN-CD, and SEC-BCD—and two SCD datasets, SECOND and HRSCD.The datasets span diverse geographic regions, seasonal variations, and change patterns.
  • Dataset Examples: LEVIR-CD contains 2,548 training and 1,392 testing image pairs after cropping to 512 × 512, while WHU-CD is partitioned into 512 × 512 patches.LEVIR-CD focuses on urban/suburban building changes; WHU-CD contains building construction and demolition annotations.
  • Source Data: KnowChange’s generated examples use OpenEarthMap and FLAIR as source benchmarks across Know-BCD, Know-SEC, and Know-HR.The datasets correspond to building and semantic change detection settings.

B.2 Synthetic Datasets for Comparison

The comparison includes representative synthetic datasets for binary and semantic change detection, alongside KnowChange-generated datasets, with differences in source data, synthesis strategy, scale, and supported tasks.

  • Table 8 compares existing and KnowChange-generated synthetic datasets for binary and semantic change detection.The KnowChange datasets are Know-BCD, Know-SEC, and Know-HR.

B.3 Layout-to-Mask Model Training

The Layout-to-Mask model learns category-shape associations and contextual consistency by reconstructing masked semantic regions under textual and spatial guidance. Its training combines category-aware, random-region, and broader masking strategies to support plausible shape generation.

  • The Layout-to-Mask model takes semantic maps, inpainting masks, and textual target-category prompts to generate context-consistent masked regions.Its objective is to learn associations between semantic categories, object shapes, and surrounding context.
  • Category-Aware Instance Masking: Category-aware instance masking uses bounding boxes or exact connected-component shapes to train plausible category-specific shape completion.Bounding-box masking emphasizes shape distributions within rectangular constraints, while connected-component masking emphasizes boundary fidelity.
  • Random Region Masking: Random region masking covers multiple categories and prompts the model with the two dominant categories, encouraging coherent transitions across semantic boundaries.The model must infer category placement from context rather than receiving explicit guidance for every category.
  • Randomized category colors prevent memorization of fixed color-category bindings and encourage reliance on textual semantic guidance.Conditioned on prompts, masks, and visible context, the model completes masked regions using category-specific contour information.
  • The broader synthesis pipeline trains Mask-to-Image reconstruction with instance-, regional-, and global-level masking plus semantic vector dropout.These strategies target category-consistent appearance, boundary coherence, holistic scene relationships, and robustness to incomplete semantic conditions.
  • The semantic masks are converted into pixel-wise CLIP embeddings for M2I control features, while downstream evaluations use ChangeFormer for BCD and Change3D for SCD.Label mappings align source categories with SECOND and HRSCD benchmark taxonomies.

C.1 Analysis of Change Proportions and Category Transition Matrices

The analysis compares synthetic and real change proportions, transition coverage, and JS divergence. Know-SEC most closely matches SECOND in both change density and overall transition distribution, while rule-based datasets have sparser transition coverage.

  • The analysis evaluates category-transition matrices, overall change proportions, and Jensen-Shannon divergence against the real SECOND dataset.All datasets are mapped to a unified SECOND taxonomy for comparison.
  • Change Proportions: 19.94% is SECOND’s real-world change proportion; Know-SEC differs by 1.90 percentage points, compared with 2.01 percentage points for WHU-GCD.Changen2-S9 reaches 25.52%, while FSC-180k is approximately 8.4%.
  • Category Transition Matrices: SECOND contains diverse inter-class and intra-class transitions, whereas Changen2-S9 and WHU-GCD have relatively sparse matrices with several missing transitions.Their change simulation relies on restricted transition designs.
  • Distributional Similarity: Know-SEC achieves the lowest JS distance, 0.1621, relative to SECOND.This indicates the closest overall transition distribution among the compared synthetic datasets.
  • Category Transition Matrices: Know-SEC uses VLM-inferred scene-context transitions rather than a predefined transition matrix, producing broader transition coverage.Its change proportion is 21.84%, differing from SECOND by only 1.90 percentage points.
  • Caveat: The generated distribution is also affected by the source semantic data.

C.2 Effectiveness of In-Domain Data Augmentation:

KnowChange supports in-domain augmentation on previously unseen datasets by using VLM knowledge and a broad semantic vocabulary. The evaluation samples labeled SECOND data at 1% and 5% to synthesize target-domain change samples.

  • KnowChange performs in-domain augmentation on previously unseen datasets using VLM knowledge and a broad semantic vocabulary.The framework differs from methods requiring manually designed transitions and repeated retraining for different change types.
  • The augmentation study randomly samples 1% and 5% of SECOND’s labeled training data, which was unseen during KnowChange training.The sampled images and labels are used to synthesize target-domain change samples.

C.3 Generalization to Additional Remote Sensing Tasks

KnowChange is further examined for remote sensing tasks beyond its main building and semantic change detection experiments. Its semantic-map-conditioned synthesis can generate images without real-image references for downstream applications.

  • The additional-task analysis examines whether KnowChange supports remote sensing tasks beyond change data construction.The main experiments focus on building and semantic change detection, while broader applicability of the synthesis component is less explored.
  • KnowChange generates remote sensing images from semantic maps without requiring real images as references.The synthesis is conditioned on semantic maps and is described as preserving pixel-level content.
  • The framework is visualized for semantic segmentation, building extraction, road extraction, and water detection.

C.4 Image Reconstruction Quality Analysis

Image reconstruction is evaluated by comparing synthesized and real SECOND images with PSNR, SSIM, and LPIPS. M2I outperforms HySCDG and Changen2 across all three metrics, while KnowChange also supports iterative multi-temporal urban evolution synthesis.

  • Image reconstruction quality: M2I reconstructs post-change images from semantic masks while preserving unchanged-region content.Its synthesis capability is evaluated against HySCDG and Changen2 using SECOND images and the metrics PSNR, SSIM, and LPIPS.
  • Image reconstruction quality: M2I consistently outperforms HySCDG and Changen2 across PSNR, SSIM, and LPIPS on the SECOND dataset.
  • Long-term synthesis: KnowChange can iteratively transform a single-temporal scene into a long-term multi-temporal sequence with progressive scene evolution.Each generated image and semantic map becomes the input for the subsequent timestamp, producing multi-category urban changes while maintaining semantic and spatial consistency.
  • Motivation: Rare and diverse change transitions are difficult to cover with real training samples, motivating scalable change-data generation.
  • Related approaches: Fully synthetic rendering offers explicit parameter control but can lack real-world texture diversity, while hybrid insertion methods may introduce visible artifacts.
Loading 2608.24263v1…