PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluating the Health of Open-Source Smart City Platforms

Rodrigo Bravo Simões, Fernando Brito e Abreu, Vasco Amaral

arXiv:2609.18801v1cs.SEcs.CY

TL;DR

Limited OSS support and scarce municipal technical staff make it important to assess ecosystem health when comparing open-source SCPs. The paper evaluates six active open-source SCPs using repository metrics, with special emphasis on documentation quality assessed with LLMs. Orion-LD leads the comprehensive ranking with a 4.8 average rank, followed by CIVITAS/CORE at 3.8; FogFlow, OpenRemote, and UDP tie at 3.2, while Snap4City ranks lowest at 2.6.

  • Problem

    Limited OSS support and scarce municipal technical staff make it important to assess ecosystem health when comparing open-source SCPs.

    Source: §I · p. 1

  • Method

    The paper evaluates six active open-source SCPs using repository metrics, with special emphasis on documentation quality assessed with LLMs.

    Source: §I · p. 1, §II · p. 2

  • Results

    Orion-LD leads the comprehensive ranking with a 4.8 average rank, followed by CIVITAS/CORE at 3.8; FogFlow, OpenRemote, and UDP tie at 3.2, while Snap4City ranks lowest at 2.6.

    Source: A. Consolidation · p. 5

  • Takeaways & Limitations

    The results provide an overview of open-source SCP health, not a recommendation to adopt a specific platform.

    Source: §V · p. 6

    The health priorities were not based on interviews with municipal staff; the authors imagined stakeholder needs when devising user stories.

    Source: §V · p. 6

Abstract

from arXiv · show

To manage the complexity of smart cities, a variety of smart city platforms (SCPs), both proprietary and open-source, have been proposed. Typically acting as middleware between IoT devices at a lower layer and smart services at a higher layer, these platforms simplify the management of smart cities by reifying a variety of requirements common to different municipalities. Open-source software (OSS) is typically free of charge, making open-source SCPs an attractive option for municipalities with smaller budgets that still wish to improve efficiency and quality of life for residents and visitors. Furthermore, using an OSS SCP promotes digital sovereignty, since municipalities can control where the data will reside and will be less prone to vendor lock-in. A lack of support for OSS hinders its adoption by municipalities' IT teams, often with scarce technical resources. An active (aka "healthy") ecosystem, including well-orchestrated developers and users, the availability of installation and user manuals, non-vulnerable and documented code, test batteries, and other artifacts, is an important decision factor before adopting an OSS SCP. In this paper, we evaluate different open-source SCPs using the concept of "OSS health", covering several of the aforementioned characteristics.

I. INTRODUCTION

Open-source smart city platforms can offer municipalities lower-cost infrastructure and greater control over data, but limited OSS support and technical staffing make ecosystem health important to adoption. This paper evaluates six open-source SCPs, emphasizing repository-based health metrics and documentation quality.

I. INTRODUCTION

  • SCPs typically act as middleware between lower-layer IoT devices and higher-layer smart services, helping manage common smart-city requirements.Source: §I · p. 1
  • Open-source SCPs can help municipalities with smaller budgets while supporting control over data location and reducing exposure to vendor lock-in.Source: §I · p. 1
  • Limited OSS support and scarce specialized staff make community activity and high-quality documentation important adoption considerations for municipal SCPs.Documentation can help smaller municipal IT teams deploy and operate platforms that require significant technical knowledge.Source: §I · p. 1
  • The paper evaluates the health of six open-source SCPs, mostly using source-code repository metrics and giving special attention to documentation quality analyzed with LLMs.Source: §I · p. 1

II. RELATED WORK

Prior work compares SCP requirements and surveys platform technologies, but the paper identifies OSS health as an unexplored dimension in this domain. It positions repository metrics as a broader basis for evaluating ecosystem health than activity and popularity measures alone.

II. RELATED WORK

  • Earlier SCP studies compare platforms against requirements, synthesize platform technologies, or present a reference architecture,,.Source: §II · p. 1
  • The authors identify OSS health as an unexplored SCP dimension and use Aveloxis to gather GitHub and GitLab metrics from active platforms.The literature review describes 107 health characteristics across 15 themes, while neither it nor CHAOSS prescribes a selection framework,.Source: §II · p. 2
  • A prior study of open-source IoT platforms compares activity and popularity, whereas this paper gathers a wider variety of metrics but gives less detail on platform features.Source: §II · p. 2

III. ANALYSIS

The analysis identifies SCP repositories from prior surveys and a technical report, then filters for recent project activity. For FIWARE, it uses a context broker as the representative repository because the project lacks one obvious core repository.

A. Gathering Repositories

  • Repository candidates were compiled from SCP surveys and a technical report, excluding unnamed platforms before searching for repositories.Searches used each platform name with “source” or “repository,” adding smart-city or IoT terms when initial results were irrelevant.Source: A. Gathering Repositories · p. 2
  • From 15 GitHub and GitLab entries, the authors selected six repositories after excluding projects with no commits or only trivial commits over the preceding two years.Source: A. Gathering Repositories · p. 2
  • Because FIWARE has no obvious single core repository, the analysis represents it with Orion-LD, the only context broker developed by the FIWARE Foundation.The authors chose a context broker because it is a core, mandatory component of FIWARE-powered platforms and can serve as an SCP core.Source: A. Gathering Repositories · p. 2

B. Overview

The section describes the six platforms and the repository characteristics used to frame their comparison. The platforms range from integrated component stacks to focused IoT and context-management systems.

B. Overview

  • Table I describes the included SCPs in alphabetic order, with repository, producer, origin, stars, two-year commit counts, code-line totals, and main language.Platform descriptions were manually compiled from official documentation.Source: B. Overview · p. 2, Table I · p. 3
  • CIVITAS/CORE is a UDP fork that reuses components to provide a pre-packaged platform described as easier to install and configure than assembling them manually.Its repository is primarily declarative YAML, while most Python files serve testing purposes.Source: B. Overview · p. 2
  • FogFlow provides cloud-edge orchestration and IoT service development tools, while OpenRemote offers device integration, automation, analytics, and smart-city application tools.FogFlow is not a complete SCP by itself; OpenRemote supports multiple protocols and provides dashboards and interface components.Source: B. Overview · p. 2
  • Orion-LD adds NGSI-LD linked-data capabilities to Orion, including temporal and geographical representations that may improve interoperability across platforms.Source: B. Overview · pp. 2–3
  • Snap4City combines a Node-RED and JavaScript development environment with IoT support, dashboards, analytics, decision support, and ready-to-use AI modules.Source: B. Overview · p. 3
  • UDP integrates tens of components into stacks such as identity, context, API, and data management, with much of its codebase in YAML configuration.The Jinja templating language is frequently used.Source: B. Overview · p. 3

C. Open-source licenses

The platforms use different license models, which affect customization and redistribution. Municipalities should assess the obligations of each license before deployment.

C. Open-source licenses

  • FogFlow uses BSD 3-Clause, while the other platforms use GNU AGPL or EUPL v1.2 copyleft licenses.The EUPL uses variable copyleft; the AGPL requires modifications to be published even when offered only as a network service.Source: C. Open-source licenses · p. 3
  • The EUPL may allow integration with proprietary software under specific circumstances, while the AGPL imposes strong copyleft.Both licenses require publication of derivative works under certain conditions.Source: C. Open-source licenses · p. 3
  • Municipalities planning to customize and deploy a platform should assess its legal obligations and seek appropriate legal guidance.Licensing directly affects deployment, customization, and redistribution.Source: C. Open-source licenses · p. 3

D. Documentation Quality

The documentation assessment uses binary questions answered by three LLMs, with prompts designed to ground answers in the supplied documentation. Orion-LD and Snap4City had the same majority-vote score, but Orion-LD had higher inter-rater agreement.

D. Documentation Quality

  • The assessment uses 18 binary Yes/No questions targeting 21 documentation-quality aspects and practical configuration, deployment, and interoperability issues.Some questions target more than one aspect.Source: D. Documentation Quality · p. 3
  • Three LLMs from different providers answered prompts containing the questions, instructions, and each platform’s documentation.The selected models had large context windows and similar pricing, used as an indicator of quality.Source: D. Documentation Quality · p. 3
  • Documentation was assembled as Markdown, with Snap4City represented by its 189-page Technical Overview PDF converted to Markdown.Using the converted text gave all models the same input.Source: D. Documentation Quality · p. 3
  • Models were instructed to answer Yes only when unambiguous and to rely on documentation rather than factual knowledge about the platforms.They also supplied explanations and supporting documentation sections, making positive answers easier for a human to check; negative answers are harder to verify.Source: D. Documentation Quality · pp. 3–4
  • Orion-LD and Snap4City received the same majority-vote documentation score, but Orion-LD had higher inter-rater agreement.Across all platforms, Fleiss’ κ was 0.368, considered fair agreement; per-platform agreement indicates how unambiguous the documentation is.Source: D. Documentation Quality · p. 4, Table II · p. 4

E. Code complexity

The code-complexity analysis derives a repository-level CC/LoC measure from file-level complexity and code-line counts. The results differ by platform type: integration-oriented platforms have near-zero values, while Orion-LD has the highest value among the others.

E. Code complexity

  • Aveloxis collected repository metrics from GitHub and GitLab over the two years from June 2024 to June 2026.The resulting data are presented in Tables III and IV.Source: E. Code complexity · p. 4
  • CIVITAS/CORE and UDP have near-zero CC/LoC because they primarily integrate tools rather than being typical software applications.The other platforms have similar values, with Orion-LD highest.Source: E. Code complexity · p. 4
  • Repository CC/LoC is calculated by dividing the sum of file-level cyclomatic complexity by the sum of lines of code.The per-file metrics are collected by scc within Aveloxis.Source: E. Code complexity · p. 4

F. Maintainability

The section assesses SCP maintainability and security alongside the maturity of contributor communities. It reports both the limits of available issue data and the scope of its community assessment.

F. Maintainability

  • MTTR includes only bug-labeled issues closed within the study period; GitLab permissions blocked analysis for CIVITAS/CORE and UDP, and no active bug labels may reflect weak reporting.The authors also note that some bugs remained open or were closed after the period.Source: F. Maintainability · p. 4

G. Security

  • OpenSSF Scorecard rates security checks from 0 to 10 and aggregates them using its repository weighting system.Checks cover vulnerability presence, security practices, dependency management, and organizational diversity.Source: G. Security · p. 4

H. Community maturity

  • OpenRemote has the most mature, balanced observed contributor community; Orion-LD has a smaller team but broad issue reporting, while CIVITAS/CORE and UDP lack complete assessments.The assessment concerns observed contributor communities, not overall governance or total user populations.Source: H. Community maturity · p. 5
  • Community maturity combines contributor breadth, development-effort distribution, and review activity; GitLab permissions leave CIVITAS/CORE and UDP issue and reviewer counts unavailable, not zero.Core-team size is estimated as the smallest set responsible for 90% of commits.Source: H. Community maturity · pp. 4–5

IV. DISCUSSION

Across five health dimensions, the six platforms show distinct profiles, with Orion-LD ranking highest overall and Snap4City lowest. Snap4City’s low community-maturity rank reflects how institutional commits were attributed, not evidence that it has only one developer.

A. Consolidation

  • The methodology uses a focused subset of public-data indicators, yet the consolidated metrics reveal distinct profiles for each platform.Source: A. Consolidation · p. 5
  • Orion-LD leads with a 4.8 average rank, CIVITAS/CORE follows at 3.8, FogFlow, OpenRemote, and UDP tie at 3.2, and Snap4City ranks lowest at 2.6.The ranking aggregates metrics across five health dimensions.Source: A. Consolidation · p. 5, Table V · p. 6
  • Snap4City is strong in documentation quality, while its low community-maturity rank reflects attributing all commits to an institutional account, not a single developer.Figure 1 maps platform ranks across the five dimensions.Source: Figure 1 · p. 5, A. Consolidation · p. 6

B. Threats to validity

The documentation-quality assessment has limits: LLM analysis may be less accurate than manual review, and text-only evaluation excludes graphical support.

B. Threats to validity

  • LLM-based documentation analysis is not guaranteed to match the accuracy of human manual review.The authors asked the LLMs to provide reasoning, and made their responses available as supplementary material.Source: B. Threats to validity · p. 6
  • Text-only evaluation could not assess the graphical-support quality of documentation.Source: B. Threats to validity · p. 6
  • Answers contained in omitted figures, such as architectural diagrams, could be missed and produce false negatives.Source: B. Threats to validity · p. 6

V. CONCLUSION AND FUTURE WORK

The paper presents an overview of health characteristics across open-source SCPs, not a recommendation to adopt a particular platform. It identifies platform features and stakeholder-informed priorities as important parts of adoption decisions that it did not fully assess.

V. CONCLUSION AND FUTURE WORK

  • The study presents health characteristics of open-source SCPs and does not recommend adopting a specific platform.Source: §V · p. 6
  • The analysis does not assess platform features in depth, although those features are critical when deciding which platform to adopt.The paper reports only brief platform descriptions.Source: §V · p. 6
  • The authors suggest prioritizing health characteristics through user stories, ideally gathered by interviewing municipal staff rather than imagining stakeholder needs.Source: §V · p. 6
Loading 2609.18801v1…