Source-linked AI summary
DUDA-Bench: Benchmarking LLM Agents on Multimodal Data-Driven Urban Diagnosis
Yizhi Song, Hang Ni, Weijia Zhang, Hao Liu
TL;DR
Expert-led urban diagnosis relies on labor-intensive, case-specific workflows, while existing benchmarks leave open whether agents can sustain evidence-driven diagnosis across heterogeneous urban observations. DUDA-Bench links atomic operations to workflow tasks spanning problem framing, urban profiling, diagnostic reasoning, and validation and synthesis, grounded in multimodal data from 12 cities and an interactive agent environment. A substantial gap separates isolated analytical operations from end-to-end diagnosis, and agent-system benefits vary across backbones.
Problem
Expert-led urban diagnosis relies on labor-intensive, case-specific workflows, while existing benchmarks leave open whether agents can sustain evidence-driven diagnosis across heterogeneous urban observations.
Source: §1 · p. 1Method
Results
A substantial gap separates isolated analytical operations from end-to-end diagnosis, and agent-system benefits vary across backbones.
Source: §1 · pp. 2–3Takeaways & Limitations
The benchmark provides a framework for evaluating complete diagnostic workflows through hierarchical tasks and outcome- and process-level assessment.
Source: §1 · p. 2Geographic coverage and source-data biases may limit generalization, and diagnoses should not inform consequential planning without expert review and local validation.
Source: ETHICS STATEMENT · p. 10
Abstract
from arXiv · showhide
Urban diagnosis integrates heterogeneous observations to identify urban problems, localize affected areas, and investigate contributing factors, informing evidence-based urban planning and management. However, its reliance on labor-intensive, case-specific expert workflows limits scalability and reuse, motivating the exploration of agent-based execution. To evaluate this capability, we introduce DUDA-Bench, a hierarchical and interactive benchmark that formalizes data-driven urban diagnosis as a multi-stage agent workflow. It comprises 86 atomic and 22 workflow tasks spanning four analytical stages, grounded in multimodal data from 12 cities covering five urban problem types. Evaluations of seven backbone models and five agent systems reveal a substantial gap between isolated analytical competence and end-to-end diagnosis, with system benefits varying across backbones. Trajectory analysis shows that unresolved evidence gaps propagate across stages, while successful recovery involves revising assumptions and actions using feedback. These findings highlight limitations in coordinating analytical capabilities across stages, particularly adaptive planning, evidence integration, and verification. More broadly, DUDA-Bench provides a framework for translating expert analytical workflows into hierarchical agent tasks and process-aware evaluation, supporting systematic assessment of end-to-end analytical capabilities.
1 INTRODUCTION
Urban diagnosis depends on expert workflows for integrating heterogeneous evidence, but sustaining that process across observations remains an open challenge. DUDA-Bench evaluates complete diagnostic workflows and reveals a gap between isolated analytical competence and end-to-end diagnosis.
1 INTRODUCTION
- Urban diagnosis relies on expert evidence selection and iterative interpretation, making analyses labor-intensive to reproduce across regions and problem types.Existing benchmarks leave open whether agents can sustain evidence-driven diagnosis across heterogeneous urban observations.Source: §1 · p. 1
- DUDA-Bench combines atomic operations and workflow tasks to distinguish individual analytical competence from coordination into a complete diagnosis.Workflow tasks span problem framing, urban profiling, diagnostic reasoning, and validation and synthesis.Source: §1 · p. 2
- The benchmark pairs multimodal cases and an interactive environment with hierarchical outcome assessment and stage-aligned checkpoints.Its resources include domain tools, reusable skills, and code execution, while checkpoints help locate successes or failures in evidence acquisition, reasoning, and execution.Source: §1 · pp. 2–3
- A substantial gap separates isolated analytical operations from end-to-end diagnosis, with agent-system benefits varying across backbones.Unresolved evidence gaps persist into downstream reasoning, while successful recovery involves revising assumptions and actions in response to feedback.Source: §1 · pp. 2–3
2 RELATED WORK
Prior work establishes foundations in data-driven urban diagnosis and evaluates complementary urban, geospatial, and agent capabilities. DUDA-Bench formulates diagnosis across problem types and regions as a unified agent task.
2 RELATED WORK
- Existing data-driven urban studies typically address individual problems in specific regions rather than a unified task across problem types, regions, and heterogeneous data sources.DUDA-Bench builds on this foundation by formalizing diagnosis as a common, evaluable agent task.Source: §2 · p. 3
- Prior work leaves diagnosis across problem types, regions, and heterogeneous data sources unformulated as a unified task.DUDA-Bench addresses this gap by requiring agents to integrate urban evidence and execute multi-stage analyses interactively.Source: §2 · p. 3
- Urban and agent benchmarks cover complementary capabilities, including interactive decision making, multimodal understanding, executable spatial analysis, and multi-step execution.DUDA-Bench combines heterogeneous urban evidence, interactive multi-stage analysis, and outcome- and process-level evaluation.Source: §2 · p. 3
3 DUDA-BENCH
DUDA-Bench formalizes urban diagnosis as a tool-assisted process over multimodal evidence, pairing atomic analytical operations with complete workflows and assessing both outcomes and intermediate progress.
3 DUDA-BENCH
- DUDA-Bench combines a hierarchical task taxonomy, a multimodal urban dataset, and an interactive agent environment with outcome and stage-aligned evaluation.Its construction spans four stages, from literature-grounded task design through hierarchical evaluation.Source: §3 · p. 4
3.1 TASK FORMULATION
- Urban diagnosis is formulated as a multi-step, tool-assisted task in which agents use intermediate evidence to revise actions and produce a localized, evidence-grounded diagnosis.The task specifies a diagnostic objective and spatiotemporal scope, available multimodal data, and an executable environment.Source: §3.1 · p. 4
3.2 TASK TAXONOMY
- Two experts translate studies of five urban problem types into 22 workflow tasks and 86 atomic operations spanning four stages and three complexity levels.Atomic tasks test individual operations; workflow tasks require agents to select and coordinate capabilities through open-ended, multi-turn analysis.Source: §3.2 · p. 4
3.3 MULTIMODAL URBAN CASE DATASET
- The dataset standardizes provider-collected observations at city and study-region scales, covering 12 cities and 22 regions during 2015–2024.It draws on eight sources, including OSM, WorldPop, GHSL, VIIRS, Landsat, and Sentinel-2.Source: §3.3 · pp. 4–5
3.4 AGENTIC ENVIRONMENT
- The shared interactive environment gives agents data, domain tools, reusable skills, and code execution for iterative urban-diagnosis analysis.Agents can inspect evidence, compute indicators, validate intermediate results, and revise their analysis; systems use the same task inputs and interfaces.Source: §3.4 · p. 5
3.5 EVALUATION PROTOCOL
- Workflow evaluation scores issue recognition, location coverage, and diagnostic-evidence precision and recall, while atomic tasks use task-specific validators.Fourteen stage-aligned checkpoints assess trajectories as pass, partial, or fail; GLM-5.3 achieved 90.4% accuracy against adjudicated human labels. Scores are averaged across repeated runs and then macro-averaged across tasks or workflow targets.Source: §3.5 · pp. 5–6
4 EXPERIMENTS
Experiments compare backbone models and agent systems, then trace workflow failures and capability bottlenecks in multimodal urban diagnosis. The analyses examine overall performance, execution dynamics, and capabilities underlying end-to-end results.
4.1 EXPERIMENTAL SETUP
- The experiments evaluate seven backbones under ReAct, compare five agent systems on two Qwen backbones, and measure Atomic PR and four workflow outcome metrics.Ablations remove code execution, skill routing, or domain tools from the Qwen3.8-Max ReAct configuration.Source: §4.1 · pp. 6–7
4.2 OVERALL PERFORMANCE AND AGENT SYSTEM EFFECTS (RQ1)
- Qwen3.8-Max achieves the highest Workflow Score (0.277), while remaining models range from 0.071–0.182 and Atomic PR ranges from 0.419–0.558.The benchmark comparison also shows that the best-performing agent system depends on the backbone: ReAct for Qwen3.8-Max and LAMBDA for Qwen3.8-27B.Source: Table 2 · p. 7, §4.2 · p. 7, Table 3 · p. 7
- Backbone–system effects reverse under LAMBDA, and Qwen3.8-Max has the strongest cost-normalized performance; a larger interaction budget does not ensure a higher Workflow Score.Workflow episodes use 9.9–20.3 turns and 177.9K–322.3K tokens.Source: Table 3 · p. 7, Figure 2 · p. 8, §4.2 · p. 8
- Code enables finer-grained spatial comparisons but brings more execution failures, while high-level tools are more stable but tend to remain at aggregate statistics.Code frequency alone does not predict success; the evidence must support the diagnostic objective.Source: §4.2 · pp. 7–8
4.3 FAILURE PROPAGATION AND RECOVERY PATTERNS (RQ2)
- Agents can plan plausible analyses yet fail to turn required indicators into usable observations, allowing evidence gaps to propagate from profiling into diagnosis.Maintaining alignment among the target problem, collected evidence, and downstream diagnostic claim is a central difficulty.Source: §4.3 · p. 8
- Valid recovery uses feedback to revise assumptions and actions, while successful tool execution can still yield invalid evidence that requires analytical checks.Runtime errors can guide recovery, but invalid outputs may go unnoticed without checks on category meaning, coordinate systems, and data validity.Source: §4.3 · pp. 8–9
4.4 CAPABILITY BOTTLENECKS AND INTERACTIVE AFFORDANCES (RQ3)
- Workflow success requires tracking unresolved subgoals, preserving valid data and execution state, and integrating intermediate evidence into consistent downstream explanations.Stronger quantification or temporal analysis does not compensate for weaknesses in localization, visual interpretation, or evidence grounding.Source: §4.4 · p. 9
- Interactive feedback can help weaker backbones revise assumptions, while ablations show that code, skill routing, and domain tools support workflow execution.Removing code reduces Workflow Score from 28.5% to 13.7%; disabling skill routing in the no-code setting lowers it to 12.4%, and removing domain tools reduces performance to zero.Source: §4.4 · p. 9
5 CONCLUSION
DUDA-Bench evaluates end-to-end urban diagnosis and exposes gaps between isolated analytical competence and workflow-level performance. The authors also identify limits on generalization and caution against consequential use without expert review and local validation.
5 CONCLUSION
- DUDA-Bench combines hierarchical tasks, multimodal urban data, an interactive environment, and outcome- and process-level evaluation across four diagnostic stages.Experiments reveal a substantial gap between isolated analytical competence and end-to-end diagnosis, with recurring failures in evidence acquisition, state maintenance, recovery, and cross-stage integration.Source: §5 · p. 9
- Verifiable interaction, code execution, skill routing, and domain tools can help agents better realize their analytical capabilities.The conclusion reports this alongside recurring end-to-end failures.Source: §5 · p. 9
- The authors report using generative AI for text refinement, literature discovery, research discussion, code implementation, debugging, evaluated agents, and AI-assisted judging.They state that the underlying urban observations came from real-world sources, not generative AI, and describe a planned public release subject to source-specific redistribution conditions.Source: AI USE STATEMENT · pp. 9–10, REPRODUCIBILITY STATEMENT · p. 10
- Geographic coverage and source-data biases may affect benchmark results and limit their generalization.The authors caution that agent-generated diagnoses should not inform consequential planning decisions without expert review and local validation.Source: ETHICS STATEMENT · p. 10
A.1 TASK CONSTRUCTION
DUDA-Bench reframes urban analysis from performing a specified operation to carrying out open-ended diagnosis. Its task taxonomy includes workflow tasks and constituent atomic operations.
A.1 TASK CONSTRUCTION
- Urban diagnosis requires inspecting spatial patterns, selecting analytical steps, integrating tool outputs, and producing a localized, evidence-grounded diagnosis.This contrasts with predefined tasks that ask an agent to perform a specified calculation.Source: §A.1 · p. 15, Figure 3 · p. 15
- DUDA-Bench comprises 22 workflow tasks categorized by problem complexity and urban problem type, and 86 atomic tasks categorized by capability type and urban problem type.Table 4 summarizes the task composition and distribution.Source: §A.1 · p. 15, Table 4 · p. 15
A.2 DETAILED STATISTICS AND COLLECTION OF DATASET
The dataset brings together urban cases, data families, and source details, while the appendix also documents trajectory-evaluation materials and scoring.
A.2 DETAILED STATISTICS AND COLLECTION OF DATASET
- 12 city settings and 22 study-region cases span 2015–2024; the dataset contains 1,507,693 records across five modalities.The dataset statistics and data-family composition are summarized in Table 5.Source: §A.2 · pp. 15–16, Table 5 · p. 16
- Landsat 8/9 thermal observations have a native resolution of 100 m and are resampled to a 30 m product grid.The listed raster resolutions refer to selected source products before benchmark-specific processing.Source: §A.2 · p. 16
- Table 6 details each data source’s analytical use, spatial level, temporal coverage, resolution, and access point.Its scope is data sources, contents, analysis scales, and spatial resolutions.Source: §A.2 · p. 16, Table 6 · p. 16
- Trajectory checkpoint verdicts use Pass (1.0), Partial (0.5), or Fail (0.0), based on completion and support from the recorded trajectory.Table 8 covers agreement on three-level labels across 716 checkpoint verdicts from 56 trajectories.Source: §A.2 · p. 17, Table 8 · p. 17
A.3 EXTENDED EVALUATION DETAILS AND HUMAN VERIFICATION
The appendix specifies checkpoint-based trajectory scoring, human verification procedures, and agreement results for an LLM judge.
A.3 EXTENDED EVALUATION DETAILS AND HUMAN VERIFICATION
- Table 7 defines 14 checkpoints for trajectory evaluation, scored as Pass (1.0), Partial (0.5), or Fail (0.0) according to completion and trajectory support.The rubric is used by both the LLM-as-a-Judge and human annotators.Source: §A.3 · p. 17
- Two annotators independently labeled applicable checkpoints in 56 sampled ReAct workflow trajectories, with disagreements resolved by a third reviewer.The annotation packets withheld judge verdicts and included task specifications, checkpoint definitions with reference content, compacted observable trajectories, and final outputs.Source: §A.3 · p. 17
- DUDA-Env includes domain skills and representative executable operations, as summarized in Table 9.Source: Table 9 · p. 18
- Annotators A and B agreed on 93.6% of 716 checkpoint verdicts; against the adjudicated gold standard, Annotator A reached 97.8% accuracy and GLM-5.3 reached 90.4%.When partial and fail are merged, GLM-5.3 accuracy rises to 94.1%; the passage attributes most remaining errors to distinguishing partial completion from complete failure.Source: §A.3 · p. 18
A.4 DOMAIN SKILLS
DUDA-Env provides domain skills for common urban-analysis operations and a geospatial sandbox for analyses beyond predefined operations.
A.4 DOMAIN SKILLS
- Domain skills guide agents through case-data discovery, urban-indicator computation, map-feature queries, imagery inspection, and spatial-pattern localization.Table 9 summarizes the skills and representative executable operations.Source: §A.4 · p. 18
- The geospatial sandbox supports task-specific analyses beyond the predefined operations.Source: §A.4 · p. 18
- Representative executable operations accompany the domain skills summarized in Table 9.Source: §A.4 · p. 18
B EXTENDED RELATED WORK
Prior research examines urban problems with heterogeneous evidence and benchmarks individual urban, geospatial, and agent capabilities. DUDA-Bench frames urban diagnosis as a shared multi-stage agent task.
Data-Driven Urban Diagnosis
- Prior data-driven studies use heterogeneous spatial, temporal, and socioeconomic evidence to examine urban problems across individual regions and settings.Examples include sprawl and land fragmentation in Phoenix, shrinkage and inequality in Detroit [Xie et al., 2018], heat islands in Valencia, and accessibility in Seoul.Source: B EXTENDED RELATED WORK · p. 18
- DUDA-Bench abstracts common analytical structure from problem-specific studies into a unified agent-oriented formulation.Source: B EXTENDED RELATED WORK · pp. 18–19
Benchmarks for Urban and Geospatial Intelligence
- Urban and geospatial benchmarks evaluate capabilities such as multimodal understanding, spatial-temporal reasoning, navigation, prediction, and executable GIS analysis.Source: B EXTENDED RELATED WORK · p. 19
LLM Agent Benchmarks
- General and scientific agent benchmarks assess multi-step tool use and trajectory-level execution, a process-aware paradigm DUDA-Bench applies to multimodal urban diagnosis.Source: B EXTENDED RELATED WORK · p. 19
C AGENT SYSTEM ENVIRONMENT CONFIGURATION
The environment standardizes inputs and data access while comparing agent systems with distinct execution and orchestration designs. Their prompts range from adaptive interaction to fixed plans and multi-role review.
C AGENT SYSTEM ENVIRONMENT CONFIGURATION
- All five agent systems receive the same task input, public data card, benchmark data and tool interfaces, and evidence-grounding policy, while retaining system-specific execution and orchestration.Unless specified otherwise, systems are restricted to the case bundle and cannot access external web resources.Source: C AGENT SYSTEM ENVIRONMENT CONFIGURATION · p. 19
C.1 REACT SYSTEM PROMPT
- The ReAct system takes at most one environment action per step, then uses the resulting observation to choose its next action.It receives sandbox API details, domain-skill instructions, and available skill directories.Source: §C.1 · p. 20
C.2 DIRECT TOOL USE SYSTEM PROMPT
- Direct Tool Use permits at most one round of environment actions, all selected before observing results, and does not support adapting actions afterward.Source: §C.2 · p. 20
C.3 PLAN-AND-EXECUTE SYSTEM PROMPT
- Plan-and-Execute requires a complete ordered action plan before observations, and the agent cannot revise that plan after execution begins.The synthesis role answers using only the evidence obtained from the fixed plan.Source: §C.3 · p. 21
C.4 MAF SYSTEM PROMPTS
- The MAF baseline assigns environment actions to an Analyst, read-only verification to a Reviewer, and coordination and final response to a Manager.All three roles use the same backbone model.Source: §C.4 · p. 22
C.5 LAMBDA SYSTEM PROMPT
- The benchmark-facing LAMBDA adaptation iteratively selects an analytical function, observes its execution result, and adapts subsequent actions within the common DURA environment.Source: §C.5 · p. 23