PaperlayerView exact revision on arXiv

Source-linked AI summary

From Pixels to Policy: A Multi-Agent System for Intervention and Geo-Spatial Decision Support

Hosam Elgendy, Utkarsh Mall

arXiv:2610.01870v1cs.CV

TL;DR

Geospatial vision largely monitors urban indicators rather than proposing interventions, while digital-twin simulations can be costly. VIDA-GEO coordinates agents to generate policy-grounded image edits and score them with an indicator model. It achieves the highest LLM-judge scores on 7 of 8 tasks.

  • Problem

    Geospatial vision largely monitors urban indicators rather than identifying interventions, while digital-twin simulations are costly to maintain and run.

    Source: §1 · pp. 1–2

  • Method

    VIDA-GEO coordinates agents that plan regions, propose policy-grounded edits, segment masks, generate interventions, and evaluate them with a black-box indicator model.

    Source: §1 · p. 2

  • Results

    VIDA-GEO obtains the highest LLM-judge scores on 7 of 8 tasks.

    Source: §4.4 · p. 6

  • Takeaways & Limitations

    The system is intended to provide decision support to urban planners and policymakers considering changes to physical spaces.

    Source: §5 · p. 9

    The method does not directly encode full zoning law, engineering standards, or construction budgets.

    Source: §5 · p. 9

Abstract

from arXiv · show

Urban environments are shaped by design choices with long-term implications for health, safety, and quality of life, yet evaluating proposed interventions remains costly, time-consuming, and often impractical. Existing geospatial vision methods largely focus on monitoring urban indicators from aerial and street-view imagery, rather than proposing interventions and estimating their effects on such indicators. Moving beyond recognition, we introduce the problem of discovering interventions that improve target indicators for a given aerial or street-view image. We argue that a black-box indicator model, combined with a generative editing model, can serve as an implicit digital twin for testing intervention hypotheses. We present VIDA-Geo , a multi-agent system that explores this intervention space by coordinating segmentation, diffusion-based inpainting, and indicator scoring models to produce interventions that are both perceptually realistic and aligned with real-world policies. We evaluate our system on 8 indicators across aerial and street-view imagery, measuring changes in factors such as perceived safety and greenery. Our approach outperforms existing baselines in many cases, achieving up to 2X higher perceptual quality and policy alignment scores. Finally, our model provides users with multiple candidate interventions, supporting an expert city-planner-in-the-loop workflow.

1 INTRODUCTION

Geospatial vision has advanced indicator monitoring, but proposing realistic, policy-grounded interventions remains underexplored. VIDA-GEO combines indicator scoring and image editing in a multi-agent framework intended to support intervention decisions.

1 INTRODUCTION

  • Monitoring urban indicators is well explored, but identifying policy-grounded interventions from geospatial images remains underexplored.Digital-twin simulations are costly to maintain and run, and model only intervention types supported by their simulation models.Source: §1 · pp. 1–2
  • VIDA-GEO proposes realistic image interventions that improve a target indicator and offers multiple candidates for decision support.The framework takes an aerial or street-view image and a black-box indicator model as inputs.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • An indicator model combined with a generative editing model can serve as an implicit digital twin for testing intervention hypotheses.The paper argues that indicator models can handle counterfactuals, while generative models provide realistic, controllable editing.Source: §1 · p. 2
  • VIDA-GEO coordinates planning, policy, segmentation, and generation agents, then evaluates candidate edits for indicator improvement and policy grounding.The agents identify regions of interest, propose permissible changes, find masks, generate edits, and use trial history to guide later planning.Source: §1 · p. 2
  • The paper introduces an eight-task benchmark and metrics for intervention realism from image-based and policy-oriented perspectives.It presents VIDA-GEO as a system that proposes, executes, and quantitatively evaluates geospatial interventions.Source: §1 · p. 2

2 RELATED WORKS

Prior geospatial vision work estimates urban indicators, while digital twins and counterfactual methods address simulation or model behavior. VIDA-GEO frames interventions as multimodal decisions that also account for physical realism and policy feasibility.

Urban Indicator Estimation

  • Overhead and street-view imagery have been used to estimate urban indicators such as safety, mobility, vegetation, liveliness, and perceived qualities.The paper uses established indicator models to measure and guide interventions that improve scores at a given place.Source: §2 · p. 3

Intervention and Decision Making

  • Digital twins support evidence-based decisions but are domain-specific and costly to simulate, while counterfactual methods often explain model predictions rather than propose real-world actions.Prior work also uses diffusion models to generate counterfactual explanations for black-box models.Source: §2 · p. 3

Agents for Intervention and Interpretability

  • Unlike related agent and generative systems, VIDA-GEO formulates urban intervention as a multimodal decision problem optimized for indicators, physical realism, and policy feasibility.Related work includes remote-sensing vision-language models, agents for experiment design, and generative street-view interventions.Source: §2 · p. 3

3 METHODOLOGY

VIDA-GEO decomposes geospatial intervention into specialized agents that select regions, constrain permissible changes, generate edits, and score candidate outcomes. Quality checks and multi-epoch feedback support localized, varied proposals.

3 METHODOLOGY

  • VIDA-GEO produces grounded proposals comprising a region, an edit, and a textual description through specialized agents and multi-epoch feedback.The decomposition localizes changes and the accumulated history steers planning away from repeated failure modes.Source: §3 · p. 3

3.1 PLANNING AGENT

  • The Planning Agent ranks multiple candidate regions of interest and supplies coordinates, bounding boxes, and semantic cues for downstream segmentation.Later planning epochs also use summaries of previous attempts.Source: §3.1 · p. 3

3.2 POLICY AGENT

  • The Policy Agent classifies regions as free, constrained, or skip and preserves functional street-scene elements in edits.Constrained regions permit edits only under specified rules; skip regions contain critical infrastructure that must remain unaltered.Source: §3.2 · p. 4

3.3 SEGMENTATION AGENT

  • The Segmentation Agent tries text-referred and open-vocabulary segmentation before point-prompted fallback, then QC checks mask coverage, precision, and fragmentation.Failed masks and QC descriptions feed back into revised segmentation prompts or strategies.Source: §3.3 · p. 4

3.4 GENERATION AGENT

  • Policy annotations guide prompted inpainting; realism, coherence, prompt adherence, and policy compliance are checked, with failed edits revised and mask-free generation used when inpainting is infeasible.The policy checks rely on the LLM’s knowledge; incorporating city regulation rulebooks is identified as future work.Source: §3.4 · pp. 4–5

3.5 SCORING AND MULTI-EPOCH ORCHESTRATION

  • Accepted edits are ranked by their black-box indicator-score change, and candidate histories feed later epochs to encourage diverse edits and avoid unsuccessful strategies.The system retains prior candidates and delta scores across cycles.Source: §3.5 · p. 5

4 RESULTS

The evaluation tests VIDA-GEO on aerial and Street View imagery across eight indicators, using perceptual-quality and policy-alignment measures. VIDA-GEO leads on judged quality in most tasks, while sequential edits can raise indicator scores and expert comparisons provide a separate human assessment.

4.1 BENCHMARK

  • The benchmark samples high-resolution MapBox aerial and Google Street View images, selecting 100 diverse images per indicator after filtering out images already scoring above 90.The evaluation covers eight indicators and prioritizes careful assessment over dataset size because multi-agent and generative-model interactions are repeated.Source: §4.1 · p. 5

4.2 EVALUATION METRICS

  • Interventions are evaluated on perceptual quality and policy alignment, using FID-Proxy, three LLM-judge criteria, and pairwise human comparisons on a subset.The LLM-judge criteria are visual quality, realism, and policy preservation; twelve recruited evaluators include seven graduate students and five urban-planning or policy-design experts.Source: §4.2 · p. 6

4.4 RESULTS AND DISCUSSION

  • VIDA-GEO has the highest LLM-judge scores on 7 of 8 tasks, often by 15pp, with DIFFusion sometimes making larger indicator changes but receiving lower perceptual-quality and policy-alignment scores.Table 1 reports task performance, perceptual quality, and policy alignment; the prose reports the overall comparison.Source: §4.4 · pp. 6–7, Table 1 · p. 7
  • URSimulator is limited to Street View and is more conservative, while NB2.5 (ZS) can produce larger indicator changes that do not consistently translate into high-quality or realistic edits.NB2.5 (ZS) is comparable to VIDA-GEO on greenery, a task described as relatively simple because vegetation can be added to empty spaces.Source: §4.4 · pp. 7–8
  • Experts preferred VIDA-GEO over DIFFusion on perceptual attribute alignment 61.3% of the time, and over LANCE on perceptual attributes 86.7% and feasibility 79.3% of the time.Expert agreement with LLM judges averaged 88.5% on realism and 91.6% on policy preservation; real-world-like interventions increased black-box beauty and safety scores by averages of 24.3% and 12.46%, respectively.Source: §4.4 · p. 8
  • Compared with DIFFusion, VIDA-GEO's examples are described as more realistic, interpretable, and actionable; its aerial proposals include greener parks, cycling bridges, and safer roads.The examples contrast DIFFusion edits such as removing a traffic light or increasing green-channel saturation with practical proposals such as bollards and tactile pavement; aerial interventions cover larger areas and receive relatively lower scores than Street View tasks.Source: §4.4 · p. 8, Figure 3 · p. 8
  • Sequential edits can improve an indicator more than a single edit: the boring score falls from 53% to 17%, versus 40% at best for a single edit.Goal-aligned scores rise with iterations while LLM-judge metrics decrease slightly; Figure 4 also reports a beauty-score increase from 29.2% to 92.4%.Source: §4.4 · p. 9, Figure 4 · p. 9
  • Policy constraints contribute most to policy-aligned outputs, while Edit QC and Mask QC provide substantial gains and the suggestor gives smaller but consistent improvements.Source: §4.4 · p. 9

5 CONCLUSION, LIMITATIONS, AND FUTURE WORK

VIDA-GEO is presented as an early decision-support approach for testing geospatial interventions, with a multi-agent pipeline that plans, constrains, segments, generates, and checks localized edits. The authors note important implementation limits and position the system as a starting point rather than an implementation tool.

5 CONCLUSION, LIMITATIONS, AND FUTURE WORK

  • The benchmark covers 8 tasks across aerial and street-view images using 800 images, and VIDA-GEO is presented as a multi-agent method for grounded interventions evaluated perceptually and policywise.The conclusion frames the system as potential decision support for urban planners and policymakers considering physical-space changes.Source: §5 · p. 9

A VIDA-GEO ALGORITHM AND FALLBACKS

  • VIDA-GEO's region-level pipeline coordinates planning, policy assignment, segmentation, generation, quality control, and multi-epoch feedback to avoid full-image edits that can cause artifacts or policy violations.Fallbacks revise segmentation prompts, switch tools, or smooth masks; generation failures can trigger revised prompts and regeneration without restarting the pipeline or changing the ROI and objective.Source: A VIDA-GEO ALGORITHM AND FALLBACKS · p. 15, Figure 5 · p. 16, Figure 6 · p. 17

B DETAILS ON FID-PROXY

  • The per-image domain-typicality score uses InceptionV3 image features, and a lower score indicates an edit more typical of real aerial or street-view imagery.The reference Gaussian is estimated from held-out real images; the per-image formulation supports comparisons among individual edits and models.Source: B DETAILS ON FID-PROXY · pp. 15–16

C.1 EVALUATORS BACKGROUND

  • Seven graduate students each reviewed 30 image pairs per baseline comparison, while five recruited participants had urban-planning and policy-design expertise.Student participants were not members of the author team; participation was voluntary and no personal information was collected.Source: §C.1 · p. 17

C.2 HUMAN EVALUATION RESULTS

The supplementary analyses describe human comparisons, sequential editing, and qualitative counterfactual behavior. They report that later edits trade some perceptual quality and policy alignment for further target-indicator progress, while counterfactual score changes are described as logical.

C.2 HUMAN EVALUATION RESULTS

  • The human-evaluation tables summarize VIDA-GEO preferences over DIFFusion and LANCE across perceptual alignment, realism, and urban-planning feasibility, but the supplied passages give no preference totals.The student table also breaks down preferences by targeted perceptual attribute, and the expert table reports the same evaluation questions.Source: §C.2 · p. 17, Table 2 · p. 18, Table 3 · p. 18

D SEQUENTIAL EDITING ARTIFACTS

  • Across Street View tasks, the first edit yields the largest target-indicator improvement and generally the highest average LLM-Judge scores; later edits slightly reduce perceptual-quality and policy-alignment scores.Final-iteration average LLM-Judge scores remain above 5.0; editing stops when QC rejects all candidates or the target is reached, and stopped inputs retain their last accepted result in later cohort averages.Source: D SEQUENTIAL EDITING ARTIFACTS · p. 18, Figure 7 · p. 19

E USING GENERATIVE MODELS + BLACK BOX INDICATOR PREDICTIONS AS IDTS

  • For counterfactual street-view edits, removing vegetation and making a scene less maintained corresponds to progressively lower predicted beauty scores, while relative score changes are described as logical.The passages offer qualitative evidence that the black-box predictors respond meaningfully to counterfactual edits, although exact scores are hard to interpret.Source: E USING GENERATIVE MODELS + BLACK BOX… · pp. 19–20, Figure 8 · p. 20

F IMPLEMENTATION AND EXPERIMENTAL SETUP DETAILS

The implementation combines local segmentation and editing services with cloud-based agents, and specifies constraints and scoring criteria through prompts. The benchmark retains inputs without accepted edits rather than excluding them.

F IMPLEMENTATION AND EXPERIMENTAL SETUP DETAILS

  • The pipeline uses FLUX-Fill, LISAt, SAM, and SAM3 as local services, with GPT-5.1 for planning and quality control and Gemini 2.5 Flash (NanoBanana 2.5) for semantically constrained edits.The LLM-judge QC threshold is 0.7, each region is limited to three mask/edit trials per epoch, and FLUX uses 30 steps and guidance 20.Source: F IMPLEMENTATION AND EXPERIMENTAL… · p. 20

G PROMPTS USED WITHIN VIDA-GEO

  • Masked editing gives the editor the original image and a binary mask, with only white pixels eligible for change; if it fails, maskless editing uses the original image and same instruction while keeping unrelated scene content unchanged.The reported LLM Judge Average is the arithmetic mean of the realism, policy-preservation, and visual-quality scores.Source: G PROMPTS USED WITHIN VIDA-GEO · pp. 21–22

H HANDLING OF FAILURE CASES

  • VIDA-GEO retains inputs with no accepted edit in the benchmark and flags them as “No Successful Edit”; the reported pipeline error rate on Street View images is 1.7%.No accepted edit can occur when the original indicator score is relatively optimal or no candidate satisfies acceptance criteria.Source: H HANDLING OF FAILURE CASES · p. 22

I ANALYSIS OF THE BEST OUTPUT FOR EACH MODEL

The best-output analysis reports that VIDA-GEO outperforms LANCE and DIFFusion when each model's best output is selected, and a separate analysis examines results under a different LLM judge.

I ANALYSIS OF THE BEST OUTPUT FOR EACH MODEL

  • VIDA-GEO still outperforms LANCE and DIFFusion when the best output for each input is isolated and evaluated for perceptual quality and policy alignment.Table 6 covers best edited satellite and Street View images across intervention tasks, with human-aligned quality, intervention strength, and a distributional-realism proxy.Source: I ANALYSIS OF THE BEST OUTPUT FOR EACH… · p. 22, Table 6 · p. 23

J ANALYSIS OF DIFFERENT LLM JUDGE

  • The Qwen3-VL judge analysis reports consistent VIDA-GEO results across perceptual-quality and policy-alignment metrics for best outputs.The supplied passage does not provide the underlying values or individual task comparisons.Source: J ANALYSIS OF DIFFERENT LLM JUDGE · p. 24

K BENCHMARK DATA STATISTICS

The benchmark excludes images where meaningful intervention is infeasible and spans five regions. The supplied statistics describe indicator-score distributions and average visual composition for satellite and Street View images.

K BENCHMARK DATA STATISTICS

  • The benchmark selection automatically excludes cases where meaningful intervention is infeasible, and its images span North America (43%), Europe (18%), Asia (17%), South America (14%), and Oceania (8%).Examples of excluded cases include extremely dense urban satellite images and Street View images dominated by surrounding cars.Source: K BENCHMARK DATA STATISTICS · p. 24
  • Table 9 summarizes the indicator-score distributions for selected images, but the supplied passage does not state their values.The selection process is described as an automated procedure.Source: Table 9 · p. 24, K BENCHMARK DATA STATISTICS · p. 24
  • Satellite images average 57.6% urban, 41.1% vegetated, 14.6% road, and 18.9% building composition; Street View images average 39.9% sky and 17.4% buildings, among other categories.Street View averages also include 15.5% vegetation, 14.1% road, 4.5% open ground, and 3.3% sidewalk or path.Source: K BENCHMARK DATA STATISTICS · p. 24

L MORE QUALITATIVE EXAMPLES

The qualitative examples contrast VIDA-GEO with DIFFusion, show a single intervention workflow, and compare generated edits with real-world changes. The supplied table captions describe evaluation measures and layouts but do not provide table values.

L MORE QUALITATIVE EXAMPLES

  • A greenery example follows segmentation, a planner prompt for adding trees, and FLUX inpainting with mixed tree species, increasing the black-box score by 7%.Table 11 is captioned as qualitative examples of counterfactual edits and indicator changes.Source: L MORE QUALITATIVE EXAMPLES · p. 24, Table 11 · p. 29
  • Tables 7 and 8 cover perceptual quality, realism, policy preservation, and FID-Proxy across satellite and Street View tasks; Table 8 marks statistically separated best judge scores.The captions specify average 95% confidence-interval half-widths for the reported columns.Source: Table 7 · p. 25, Table 8 · p. 26
  • For aerial safety and greenery examples, VIDA-GEO makes more realistic and policy-practical changes, while DIFFusion increases green saturation; Table 10 reports LLM-Judge, FID-proxy, and goal-gain scores.Table 10 marks minimization tasks in red and maximization tasks in green.Source: L MORE QUALITATIVE EXAMPLES · p. 24, Table 10 · p. 27

M COMPARISON WITH REAL-WORLD INTERVENTIONS

  • VIDA-GEO produces edits resembling observed real-world changes, including improved sidewalks, a bike lane, a pedestrian crossing, and a theater marquee.The comparisons target beautifulness and safety increases.Source: M COMPARISON WITH REAL-WORLD… · p. 29

N ABLATION STUDIES DETAILS

The ablation examines the policy restrictor, prompt suggestor, and quality-control agents on a uniformly sampled subset of images, reporting failures to improve and score drops when QC agents are disabled.

N ABLATION STUDIES DETAILS

  • Disabling QC agents drops average LLM-as-a-Judge scores from 77.7% to 67.1% for Generation Edit QC and 64.2% for Segmentation Mask QC; across disabled-agent outputs, 38.3% fail to improve on baseline.The ablation uses 10 uniformly sampled images per task and focuses on the policy restrictor, prompt suggestor, Generation QC, and Segmentation QC agents.Source: N ABLATION STUDIES DETAILS · p. 29

O MORE DETAILS ON THE USER INTERFACE FOR HUMAN EVALUATION

The appendix describes evaluator instructions and interface questions, alongside broader-impact cautions and image-licensing terms. It also identifies the row and metric encodings of the ablation table.

O MORE DETAILS ON THE USER INTERFACE FOR HUMAN EVALUATION

  • The interface asks which paired image has greater indicator change, looks more perceptually real, and aligns better with policy.Figure 11 shows the questions presented to human annotators.Source: O MORE DETAILS ON THE USER INTERFACE… · p. 29, Figure 11 · p. 32
  • Figure 9 places the original scene, observed real-world intervention, and VIDA-GEO edit side by side, with examples of crossing, sidewalk, marquee, and bike-lane changes.Source: Figure 9 · p. 30
  • Table 12 compares ablations by disabling one module per row; its LLM-judge metrics include Visual Quality, Realism, Policy Preservation, and their mean, Judge Avg.FID-Proxy measures domain typicality, with lower values better.Source: Table 12 · p. 30

P BROADER IMPACT OF THIS WORK

  • VIDA-GEO is framed as planning support, not a replacement for community engagement, expert review, or detailed engineering studies; pretrained models may carry biases, and plausible edits may be infeasible.The text also cautions against optimizing superficial visual improvements instead of deeper structural needs.Source: P BROADER IMPACT OF THIS WORK · p. 30

Q LICENSES FOR IMAGES

  • The benchmark does not redistribute Google Street View, MapBox, or third-party aerial imagery; it releases independently curated task coordinates instead.Users may obtain imagery from sources whose licenses permit their intended use.Source: Q LICENSES FOR IMAGES · p. 30

O MORE DETAILS ON THE USER INTERFACE FOR HUMAN EVALUATION

Related papers

This paper cites

Loading 2610.01870v1…