Source-linked AI summary
AutoMARS: Automatic and Scalable Multi-Agent Large Language Model Framework for Supply Chain Risk Assessment
Yinzhu Quan, Zefang Liu, Frederick Benaben, Benoit Montreuil
TL;DR
Supply-chain risk assessment must compare locations despite heterogeneous risk information and gaps in automation and scalability. AutoMARS coordinates agents and tools to automate cross-domain analysis; its reported evaluations show consistent assessments and stable ranking structures across repeated runs and spatial scales.
Problem
Comparative location assessment is constrained by manual coordination, limited integration of heterogeneous risk information, and computational demands across many regions.
Source: §3 · p. 4Method
AutoMARS plans and orchestrates multi-agent workflows, integrates cross-domain information through external tools, and supports parallel execution and comparison-based sorting.
Source: §4.1 · p. 5, §6 · p. 17Results
Repeated-run analyses showed strong consistency in pairwise comparisons and rankings, while multi-scale rankings were stable with variability mainly among locations with similar risk profiles.
Source: §6 · p. 17Takeaways & Limitations
AutoMARS offers a practical foundation for structured, scalable risk-aware location evaluation through comparison-based reasoning.
Source: §6 · p. 18The framework relies on manually designed LLM-agent prompts, making results sensitive to prompt formulation and potentially affecting consistency across runs and settings.
Source: §6 · p. 18
Abstract
from arXiv · showhide
We propose AutoMARS (Automatic Multi-Agent Risk Assessment), an agentic framework for automatic and scalable comparative risk assessment in supply chain networks. The framework enables end-to-end execution of analytical workflows by automatically planning tasks, coordinating specialized agents, invoking external analytical tools, and synthesizing domain-specific results into integrated risk evaluations. AutoMARS further supports scalable risk comparison across multiple locations by parallelizing independent analytical tasks. The framework adopts a multi-domain, multi-agent, and multi-tool architecture that integrates heterogeneous risk factors from socioeconomic, disaster, environmental, and transportation domains. Through coordinated agent interactions and tool integration, AutoMARS enables structured, systematic, and interpretable comparative analysis to support supply chain risk-aware location evaluation.
1. Introduction
Supply chains face diverse disruption risks, while existing approaches leave gaps in end-to-end automation, scalability, and stable comparative assessment. AutoMARS proposes an automated, multi-domain framework for assessing and comparing risk across locations.
1. Introduction
- Disruptions from hazards, environmental events, transportation accidents, and socioeconomic vulnerabilities can interrupt material flows and destabilize supply chains.Organizations therefore face a risk-management challenge in maintaining supply continuity and operational resilience.Source: §1 · p. 1
- Existing risk research includes optimization, simulation, and multi-agent approaches, while MARS integrates structured and unstructured factors for logistic hub selection.The cited examples include optimization-based models, simulation methods, and multi-agent coordination.Source: §1 · p. 1
- Prior multi-agent systems may require human intervention, and sequential interactions create computational overhead in large-scale assessment.The introduction also identifies instability associated with relying on absolute scores.Source: §1 · pp. 1–2
- Relative risk ordering is presented as more robust than absolute scores when small LLM output variations can affect assessment results.The paper associates relative ordering with improved consistency and interpretability across repeated assessments.Source: §1 · p. 2
- AutoMARS plans and decomposes high-level requests, coordinates specialized agents and external tools, and synthesizes domain analyses into unified risk evaluations.Its architecture integrates socioeconomic, disaster, environmental, and transportation risk factors.Source: §1 · p. 2
- Parallel execution of independent tasks supports scalable risk comparison across multiple locations and reduces computational time.The framework is intended for large-scale supply chain location decisions.Source: §1 · p. 2
2. Related Work
The related work covers supply-chain risk automation, LLM-based multi-agent orchestration, and LLM-as-a-Judge evaluation. Across these areas, the review notes limitations in workflow integration, scalability, stability, and evaluation reliability.
2.1. Automation in Supply Chain Risk Management
- Automation research addresses evolving-risk modeling and multi-source logistics data, but remains fragmented and offers limited end-to-end workflows and regional comparison.Simulation and digital-twin systems can automate monitoring but rely on predefined models, limiting adaptability.Source: §2.1 · p. 3
2.2. Automated LLM-Based Multi-Agent Orchestration
- LLM-based agents use natural-language coordination, role assignment, and task decomposition, while orchestration systems provide workflow and interoperability support.The review also notes continuing challenges in stability, evaluation, alignment, safety, and controllability.Source: §2.2 · p. 3
2.3. LLM-as-a-Judge in Multi-Agent Collaboration
- LLM-as-a-Judge research evaluates agent outputs, reasoning, tool use, and task completion, including through benchmarks and domain-specific applications.The review cites work on multi-step evaluation (Xu et al., 2023) and agent benchmarks (Tan et al., 2024).Source: §2.3 · pp. 3–4
- LLM judges can scale evaluation, but may show context sensitivity, inconsistent judgments, bias amplification, and limited standardization.Suggested robustness strategies include multi-judge aggregation, cross-model evaluation, structured criteria, and hybrid human-AI evaluation.Source: §2.3 · p. 4
3. Problem Definition
The problem is to compare supply-chain location risks across regions with heterogeneous conditions and information sources. The proposed assessment must coordinate domain analyses, integrate their results, and support scalable, adaptive evaluation.
3. Problem Definition
- Risk comparison across locations matters because regions differ in socioeconomic conditions, disaster exposure, environmental quality, and transportation safety.The section frames these differences as relevant to supply-chain operational stability and resilience.Source: §3 · p. 4
- The framework is tasked with planning domain-specific analyses, coordinating agents and tools, integrating results, and using reflection feedback to revise assessments.Independent analytical tasks can run in parallel to improve scalability.Source: §3 · p. 4
- The proposed response combines dynamic task orchestration, parallel execution, and iterative reflection for comparative risk evaluation.The stated objective is to integrate heterogeneous risk information across multiple locations.Source: §3 · p. 4
4. Methodology
The methodology section introduces AutoMARS’s framework, agent architecture, and execution strategies. It identifies parallelism, caching, and sorting as efficiency mechanisms.
4. Methodology
- The methodology covers system architecture, workflow, specialized-agent design, and efficient execution strategies.It is organized around structured multi-agent coordination and cross-domain risk analysis.Source: §4 · p. 4
- Structured multi-agent coordination is a central part of the methodology’s workflow.The section overview identifies this alongside the framework and agent architecture.Source: §4 · p. 4
- The execution strategies named for improving efficiency are parallelism, caching mechanisms, and a sorting algorithm.The overview does not specify their implementation in this passage.Source: §4 · p. 5
4.1. Framework
AutoMARS coordinates specialized agents and external tools to produce comparative risk assessments across supply-chain locations. A Reflection Agent checks the integrated assessment and can send it back for revision.
4.1. Framework
- The framework combines a Manager Agent, Domain Agents, external tools, and a Reflection Agent in a comparative risk-assessment workflow.The Manager Agent coordinates domain-specific analyses and integrates their results.Source: §4.1 · p. 5
- Domain Agents retrieve and process heterogeneous risk information, which is synthesized into a comparative risk evaluation.Source: §4.1 · p. 5
- The Reflection Agent checks assessment completeness and consistency, returning deficiencies to the Manager Agent for revision.The process repeats until a stable and reliable assessment is obtained.Source: §4.1 · p. 5
- Figure 1 depicts the AutoMARS framework for supply-chain risk assessment.Source: Figure 1 · p. 5
4.2. Agents
AutoMARS assigns workflow coordination, specialized risk analysis, and final validation to distinct agents. The Manager Agent plans and coordinates analyses, while the Reflection Agent checks the integrated comparative assessment.
4.2. Agents
- The system separates coordination and integration by the Manager Agent, specialized analysis by Domain Agents, and final-result validation by the Reflection Agent.Source: §4.2 · p. 5
4.2.1. Manager Agent
- The Manager Agent plans domain analyses and parallel execution, gathers evidence while adapting the plan, then revises the synthesis if Reflection Agent feedback identifies deficiencies.The workflow algorithm executes selected worker agents in parallel and repeats reflection until the assessment is accepted.Source: §4.2.1 · pp. 5–6
4.2.2. Domain Agents
- Census, Disaster, Environment, and Traffic Agents analyze socioeconomic conditions, hazard exposure, environmental quality, and transportation safety, respectively.Their specialized tools retrieve and analyze domain-specific data.Source: §4.2.2 · pp. 7–8
4.2.3. Reflection Agent
- The Reflection Agent checks cross-domain coverage, logical consistency, and whether the assessment clearly identifies a higher-risk location, routing deficiencies back for revision.Source: §4.2.3 · pp. 8–9
4.3. Efficient Execution
AutoMARS combines hierarchical parallel execution, caching, and comparison-based sorting to support efficient multi-location risk assessment.
4.3.1. Parallel Execution
- Parallel execution dispatches independent Domain Agent subgraphs concurrently and runs tool calls asynchronously, reducing idle waiting while supporting modular extension.Outputs are merged through synchronized associative updates before Manager Agent integration.Source: §4.3.1 · p. 9
4.3.2. Caching Mechanism
- Separate caches reuse structured external-tool outputs and LLM responses to avoid repeated data retrieval and model inference.Tool results are keyed by deterministic location-based identifiers; LLM cache keys derive from normalized prompts and context.Source: §4.3.2 · p. 9
4.3.3. Sorting Algorithm
- Location rankings use pairwise comparative assessments rather than predefined scalar scores or fixed global weighting schemes.The Timsort-based procedure has O(n log n) average and worst-case complexity and O(n) best-case complexity.Source: §4.3.3 · p. 9
5. Experiments
The experiments evaluate AutoMARS for effectiveness, stability, and scalability using pairwise assessments and multi-location rankings.
5. Experiments
- The experimental evaluation examines effectiveness, stability, and scalability of comparative risk assessment.Source: §5 · p. 10
- Pairwise experimental results are used to demonstrate the workflow and comparative performance.Source: §5 · p. 10
- Ranking experiments analyze consistency and robustness in multi-location ranking scenarios.Source: §5 · p. 10
5.1. Experimental Setup
Experiments cover three spatial scales in metropolitan Atlanta, with GPT-5 and LangGraph supporting agent reasoning and workflow orchestration.
5.1. Experimental Setup
- Experiments span Atlanta Urban Core (five counties), the ARC region (11 counties), and Metro Atlanta (29 counties).The paper uses these scales to evaluate the framework across different levels of spatial aggregation.Source: §5.1 · p. 10
- All AutoMARS agents use GPT-5 at temperature 1, with standardized prompts and API-based invocation.Source: §5.1 · p. 10
- LangGraph coordinates the Manager, Domain, and Reflection Agents, maintains shared state, and supports parallel branches and iterative reflection routing.Source: §5.1 · p. 10
5.2. Pairwise Experimental Results
Pairwise experiments illustrate AutoMARS’s cross-domain assessment and evaluate its stability across repeated runs. In the case study, the convergent evidence identifies Cobb County as higher-risk than Paulding County.
5.2.1. Case Study
- The Manager Agent plans parallel domain tasks, while Domain Agents produce aligned county-level evidence for cross-domain synthesis.The plan compares risk direction and relative differences using magnitude, consistency, and reinforcement.Source: §5.2.1 · p. 11
- Cobb County has greater socioeconomic exposure, including a larger population and more residents with multiple vulnerability factors.Its economic structure is more centralized and inward-commuting, while Paulding is smaller and more outward-oriented.Source: §5.2.1 · p. 11, Table 2 · p. 11
- Cobb County has higher modeled multi-hazard risk and expected losses and lower resilience, while Paulding has more recent federal weather-disaster declarations.The counties’ observed severe-weather economic losses are broadly comparable, but event frequency and fatalities differ.Source: §5.2.1 · p. 11, Table 3 · p. 12
- Cobb County has observed air-quality measurements and a larger industrial and waste-related facility footprint; Paulding lacks county-coded air observations for the study period.Source: §5.2.1 · p. 11, Table 4 · p. 12
- Cobb County has many more reported crashes, with severe or fatal incidents observed there but not in Paulding during the study period.Average severity levels are broadly similar.Source: §5.2.1 · p. 11, Table 5 · p. 12
- Reflection identified inconsistencies in hazard interpretation and unnormalized single-year traffic data; revision removed categorical NRI interpretations and excluded traffic indicators from primary decision weighting.The revised assessment retained stronger socioeconomic, environmental, and hazard evidence for the final comparison.Source: §5.2.1 · pp. 12–13
- Cobb County is accepted as higher-risk than Paulding County, based on convergent socioeconomic, modeled hazard, and environmental evidence.The final validation notes higher modeled hazard risk and expected losses, greater chronic environmental burden, and a larger, more vulnerable population in Cobb.Source: §5.2.1 · pp. 12–13
5.2.2. Comparison Stability
- Most repeated comparisons achieve 100% agreement, and reversing county order does not change the overall conclusions across three representative pairs.The experiment runs each ordered pair 20 times and repeats the assessment in reverse order; Table 6 summarizes the results.Source: §5.2.2 · pp. 13–14, Table 6 · p. 14
5.3. Ranking Experimental Results
Across Atlanta’s three spatial scales, AutoMARS produces largely stable county rankings, with variation concentrated among counties with similar risk levels. The experiments also examine model configurations, parallel execution, and caching.
5.3.1. Ranking Stability
- Across Atlanta Urban Core (5 counties), ARC (11), and Metro Atlanta (29), rankings remain largely stable; Metro Atlanta’s extremes persist, with variation concentrated among middle-ranked counties with similar risk profiles.Source: §5.3 · p. 14, §5.3.1 · p. 14
5.3.2. Ranking Statistics
- County ranking statistics show small run-to-run variation across scales, including zero variance for Fulton County in Atlanta Urban Core.In Metro Atlanta, highest- and lowest-risk groups have very small standard deviations, while middle counties form two subgroups with differing variation.Source: §5.3.2 · pp. 14–15, Figure 4 · p. 15
5.3.3. Model Configurations
- Across GPT-5, GPT-5 mini, and GPT-5 nano configurations, county rankings remain broadly consistent, with many standard deviations below 1.0 and some at zero.Fulton remains highest-risk, while Clayton tends to rank comparatively low; minor ordering differences appear across configurations.Source: §5.3.3 · pp. 15–16, Table 7 · p. 16, Figure 5 · p. 16
5.3.4. Computational Efficiency
- Parallel execution significantly improves runtime efficiency over sequential execution, with stable standard deviations across repeated experiments.The authors attribute the improvement to concurrent independent domain agents reducing idle waiting time and improving resource utilization.Source: §5.3.4 · p. 17, Table 8 · p. 17
- Caching reduces average invocation time from 76.68 seconds to near-zero for Air Quality Tool and from 37.98 seconds to 0.58 seconds for Environmental Facility Tool.The comparison uses the same query parameters executed five times in caching and non-caching settings.Source: §5.3.4 · p. 17, Table 9 · p. 17
6. Conclusion
AutoMARS combines multi-agent orchestration, cross-domain risk information, external tools, and comparison-based assessment for supply chain locations. Experiments report stable comparative rankings, while the authors identify prompt sensitivity and county-level boundaries as limitations or cautions.
6. Conclusion
- AutoMARS coordinates specialized agents, integrates heterogeneous risk information, invokes external tools, and uses planning and reflection-based validation for comparative assessment.It supports parallel execution and comparison-based sorting without predefined scoring schemes.Source: §6 · p. 17
- Experiments report consistent comparative assessments across repeated runs and spatial scales, with ranking variability mainly among locations with similar risk profiles.The authors also report that pairwise case studies synthesize cross-domain evidence.Source: §6 · p. 17
- County-level analysis can produce sharp local contrasts between neighboring counties, so localized ranking differences warrant caution.The authors attribute these contrasts to administrative boundary granularity and heterogeneous domain indicators.Source: §6 · p. 18
- Manually designed prompts make results sensitive to prompt formulation and may affect consistency across runs and settings.The authors suggest automatic prompt tuning as future work.Source: §6 · p. 18
- The authors present AutoMARS as a unified, extensible framework for automated multi-domain risk assessment in complex supply chain environments.They describe it as a practical foundation for risk-aware location evaluation.Source: §6 · p. 18