Source-linked AI summary
Scene-Aware Urban Design: A Human-AI Recommendation Framework Using Co-Occurrence Embeddings and Vision-Language Models
Rodrigo Gallardo, Oz Fishman, Alexander Htet Kyaw
TL;DR
The paper addresses the limited accessibility of everyday urban design decisions by introducing a human-in-the-loop system for scene-grounded micro-scale recommendations. It combines object co-occurrence analysis with vision-language reasoning and user refinement to generate placeable, context-specific interventions. The framework produces statistically grounded complements and additional recommendations beyond fixed pairings, while remaining constrained by 2D spatial estimates, dataset bias, and broader social and regulatory factors.
Problem
Residents’ ability to shape everyday public-space interventions is constrained, while conventional participatory tools offer limited access to object-scale decisions.
Method
The system detects urban objects in filtered ADE20K scenes, derives co-occurrence suggestions, uses a vision-language model to propose additions, and keeps users in control of selection and refinement.
Results
Across varied urban scenes, the pipeline produced placeable suggestions, including context-specific items that predefined co-occurrence lists could not recommend.
Takeaways & Limitations
The framework positions AI as a collaborator for situated, object-scale urban design grounded in statistical patterns and resident interpretation.
Takeaways & Limitations
The system estimates co-occurrence from 2D pixel distance and uses geographically and culturally biased data, limiting precise site-specific inference and generalization.
Abstract
from arXiv · showhide
This paper introduces a human-in-the-loop computer vision framework that uses generative AI to propose micro-scale design interventions in public space and support more continuous, local participation. Using Grounding DINO and a curated subset of the ADE20K dataset as a proxy for the urban built environment, the system detects urban objects and builds co-occurrence embeddings that reveal common spatial configurations. From this analysis, the user receives five statistically likely complements to a chosen anchor object. A vision language model then reasons over the scene image and the selected pair to suggest a third object that completes a more complex urban tactic. The workflow keeps people in control of selection and refinement and aims to move beyond top-down master planning by grounding choices in everyday patterns and lived experience.
1 Introduction
The paper frames micro-scale urban interventions as consequential to lived experience but difficult for residents to shape, motivating a human-AI decision environment for situated participation.
- Micro-scale decisions such as benches, bike racks, and shade structures shape lived urban experience, while residents face regulatory, resource, and knowledge barriers to making them.
- It aims to support situated resident recommendations through AI-generated complements anchored to specific objects and everyday surroundings.
- The proposed approach uses machine learning to surface possibilities while retaining human interpretation, judgment, and intent.
- The framework asks how computational suggestions can preserve local spatial knowledge and rituals while redistributing design agency in civic processes.
2 State of the Art
Prior work supports interactive and vision-based engagement with urban environments, but existing interfaces are better suited to macro-scale planning than everyday object-scale decisions.
- Tangible interfaces connect physical manipulation to simulations for accessible zoning and land-use exploration, but offer limited usability for everyday object-scale decisions.
- Extended reality, computer vision, and language models make physical urban objects responsive, queryable elements within augmented design interactions.
- Vision-language systems have also been used to combine real-world data and sustainability metrics within design pipelines.
3 Methods
The method combines object detection, empirical co-occurrence embeddings, vision-language reasoning, and text-to-3D generation in a staged human-in-the-loop workflow for urban scene interventions.
- 3.4 Mesh Generation: The generated descriptions are converted into prompts for text-to-3D mesh generation, with real-world scale, ground-level pivots, and low-poly AR assets.Failed meshes are regenerated with tightened prompts, and the model layer can be replaced for benchmarking.
- 3 Methods: The pipeline lets users choose an anchor, select a statistically co-occurring object, and iteratively accept, reject, or re-prompt recommendations before committing them.This two-level interaction preserves user selection and interpretation rather than autonomously replacing them.
- 3.1 Dataset Analysis: Scenes are filtered from ADE20K using scene categories and a threshold of at least five detected people, yielding approximately 900 images for urban-object analysis.Grounding DINO provides open-vocabulary detection, but cluttered or low-resolution scenes can produce inconsistent detections and false positives.
- 3.2 Co-Occurrence Aggregation and Embedding: The co-occurrence representation records unordered object pairs across qualifying scenes, then row-normalizes counts into empirical probability vectors for each object.These vectors act as contextual embeddings derived from observed spatial data rather than backpropagation-trained representations.
- 3.3 Vision Language Model: The vision-language model receives the scene, crops and normalized boxes for the anchor pair, and a compact summary of scene type, palette, materials, and depth.It returns five candidates with material, size, color, placement, and justification fields, then filters proposals using ground type, clearance, access, and other scene constraints.
4 Results
Across varied urban scenes, the pipeline produced placeable recommendations: co-occurrence embeddings supplied likely object complements, while the vision-language stage generated context-specific additions beyond fixed pairings.
- Across varied urban scenes, the pipeline produced placeable suggestions using co-occurrence-based object pairs and vision-language recommendations.Examples included benches paired with trees, trash cans, signs, or lamp posts.
- The vision-language model generated contextually specific items such as bus stops, kiosks, and wayfinding signage that predefined co-occurrence lists could not recommend.The model combined visual context, spatial cues, and object semantics to adapt suggestions to each scene.
- The system exhibited weak detections in cluttered scenes, near-duplicate or policy-sensitive Layer 2 suggestions, and text-to-3D generation errors.Reported mesh failures included inaccurate chess-table patterns and omission of a drinking fountain’s lower basin.
5 Limitations and Future Work
The framework is limited by 2D image-based spatial estimates, geographic and cultural dataset bias, and VLM gaps in practical urban constraints; future work proposes participatory evaluation and civic feedback.
- 2D pixel-distance co-occurrence estimates do not capture true 3D spatial relationships and may reduce precision for site-specific interventions.
- Geographic and cultural bias in the dataset limits the framework’s generalization.
- Future directions include 3D scene understanding, continuous AR use, more diverse datasets, post-deployment feedback, and civic reporting links.
- VLM recommendations may overlook social, cultural, legal, spatial, accessibility, infrastructural, budgetary, and installation constraints.
- Participatory evaluation will let users edit proposed scenes to measure alignment with user intent and refine regeneration fidelity and interpretability.
6 Conclusion
The framework combines co-occurrence embeddings, semantic reasoning, and human-in-the-loop AR interaction to support resident-led micro-scale interventions grounded in statistical patterns and situated knowledge. It presents AI as a collaborator in everyday spatial authorship and civic processes.
- The framework operationalizes co-occurrence embeddings and semantic reasoning within a human-in-the-loop AR interface for resident-led micro-scale interventions.
- It treats benches, trees, and lights as consequential urban elements while addressing their limited accessibility within participatory design at scale.
- By placing computational intelligence at the site, the framework frames urban design as dialogue among people, objects, and algorithms.
- The framework reframes AI as a collaborator in everyday spatial authorship and embeds lived experience into civic processes.
A.1 Co-Occurence Matrices
The co-occurrence analysis uses approximately 300 pedestrian-dense urban scenes to identify recurring object associations. These associations correspond to recognizable public-space configurations such as benches paired with trees and trash cans.
- Approximately 300 high-pedestrian-density urban scenes were processed to aggregate object co-occurrence statistics into a global matrix.
- Bench most often co-occurred with tree, trash can, sign, and lamp post, while planter frequently appeared with fence, balcony, and sidewalk.
- These recurring pairings align with common public-space configurations, supporting co-occurrence vectors as representations of real-world design patterns.
A.2 Embeddings
Table 1 presents the top five co-occurrence embeddings.
- Table 1 presents the top 5 co-occurrence embeddings.
A.3 System Prompt
The system prompt asks for five scene-grounded third-object options based on an urban image, an anchor object, and a co-occurrence object. Each option must be delivered as a two-column CSV containing an object name and a detailed, independently generable description.
- The input consists of an urban scene image, an anchor object, and a co-occurrence object used to propose five third-object options.
- Each proposed option must include the object’s name and a detailed description suitable for text-to-3D generation.
- Descriptions must ground appearance, materials, scale, colors, style, placement relationships, geometry, components, dimensions, and function in the scene context.
- The generated object must remain independent, avoid background placement instructions, unsafe elements, brand names, and features that obstruct primary circulation.
- The response must contain only a CSV with exactly two columns: Object and Description.
A.4 Additional Examples and Scene Outputs from VLM-Generated Recommendations and Descriptions for 3D Generative AI.
This section presents additional VLM-generated urban object descriptions and ranked combinations across pilot scenes, alongside materials addressing reproducibility, ethics, limitations, and responsible use. Together, these examples and documentation cover concrete scene interventions and the broader methodological and societal scope of the recommendations.
- A.4 Additional Examples and Scene Outputs from VLM-Generated Recommendations and Descriptions for 3D Generative AI.: Figures 7–11 identify pilot interfaces in Urban Scenes 2 through 6.
- A.4 Additional Examples and Scene Outputs from VLM-Generated Recommendations and Descriptions for 3D Generative AI.: VLM outputs provide detailed specifications for urban interventions including chess tables, drinking fountains, bike racks, seating, planters, sculptures, kiosks, lamps, shelters, vendor carts, and wayfinding signs.Descriptions specify materials, dimensions, accessibility features, durability, installation, and visual integration with surrounding urban contexts.
- A.5 VLM Suggestions: Across anchors, VLM suggestions repeatedly combine circulation, access, furniture, greenery, lighting, signage, and street-infrastructure objects into ranked recommendation sets.The outputs include combinations such as crosswalks with sidewalks, traffic lights, bicycle racks, planters, benches, kiosks, lamps, fences, and trash cans.
- A.5 VLM Suggestions: The VLM produces varied scene-level combinations of urban objects, including sidewalks, signs, trees, traffic lights, planters, kiosks, seating, bicycle infrastructure, and street art.The examples span two-object anchors and longer configurations containing multiple complementary objects.
- NeurIPS Paper Checklist: The checklist identifies dataset bias, limitations of 2D co-occurrence representations, and hardware-access constraints, while noting possible future improvements.The paper does not contain formal theorems or proofs and does not release a new dataset or model.
- NeurIPS Paper Checklist: The checklist reports that the paper describes its methods and experimental settings sufficiently for reproduction, including data filtering, detection settings, co-occurrence computation, prompts, splits, and model parameters.It also notes that anonymized code and preprocessing scripts are provided in the supplemental material, with compute requirements and variability reporting documented.
- NeurIPS Paper Checklist: The checklist describes potential benefits for community-driven design alongside risks including biased associations and privacy concerns from video capture, with transparency about datasets, model licenses, and societal impacts.The system uses publicly available datasets with controlled prompts and does not release high-risk generative models or datasets.