Source-linked AI summary
Ontology-supported AI Model and Dataset Management
Jan Novacek, Ali Ahari, Tobias Müller, Sebastian Reiter, Alexander Viehl, Oliver Bringmann
TL;DR
Industrial AI asset exchange requires discoverable, semantically consistent descriptions of models and datasets. The paper presents AIMDEP, a collaborative exchange platform incorporating the AIMDEO ontology, and demonstrates its use in a safety-critical timing scenario. The platform supports asset registration, metadata enrichment, search, and model use for software-timing analysis.
Problem
Industrial collaboration requires interoperable descriptions because AI models and datasets exchanged across organizations may lack structured metadata and common formats.
Method
The paper develops AIMDEP, a collaborative AI model and dataset exchange platform incorporating AIMDEO, an ontology for asset description.
Results
A use case demonstrates collaborative development and use of a model predicting memory access time for safety-critical software timing.
Takeaways & Limitations
The platform supports asset registration, ontology-based metadata enrichment, search, and use of models and datasets in the development workflow.
Abstract
from arXiv · showhide
Recently, there has been a great deal of research into improving AI methods and their application. The main focus is on tracking progress, enabling transparent comparisons, and fostering a more profound understanding of AI. In that process, different organizations generate and use plenty of assets that need to be tracked, traced and managed. Moreover, it is important to discover assets relevant for the task at hand. This paper presents research aiming to contribute to answering the question of what is required to exchange and manage AI models and related assets effectively without semantic gaps in an industrial context. We introduce a platform for AI model exchange, which facilitates the usage, exchange, and analysis of AI models and datasets. The platform incorporates an ontology that can foster a more profound common understanding of what is required in these tasks and help tackle the issues mentioned above. Finally, we elucidate the utility of the platform through the illustration of a use case in the context of real-time critical systems.
I. INTRODUCTION
The paper addresses collaborative AI model development and use in industrial settings, where discovering suitable assets and sharing them is hindered by missing structured metadata and incompatible formats. It presents a platform and ontology for describing, exchanging, and using AI models and datasets, illustrated through a use case.
- AI assets require characterization covering model descriptions, data, quality, and context to support discovery and trusted use.
- Industrial collaboration is hindered by unstructured or incompatible metadata when models and related assets move between organizations.
- The paper asks what is required to develop and use an AI model collaboratively in an industrial context.
- AIMDEP provides a platform for collaborative AI model development and use, incorporating AIMDEO to describe machine learning assets.
- The paper contributes an AI model and dataset ontology, a collaborative development platform, and a use case demonstrating applicability.
II. RELATED WORK
Related work covers AI asset management and AI asset characterization as the bases for comparing the presented approach.
- The related-work review addresses both AI asset management and AI asset characterization.
A. AI Asset Management
Existing tools and models support parts of machine-learning asset management or characterization, but the paper distinguishes AIMDEO by emphasizing exchange-oriented, semantically explicit descriptions.
- AI Asset Management: MLflow supports asset registry and exchange without third-party data sharing, whereas its metadata description does not use ontologies.
- AI Asset Management: MLEM provides a deployment interface and model registry, but its registry is limited and does not collect metadata.
- AI Asset Characterization: AIMDEO targets AI model and dataset exchange and includes provenance, model type, model and dataset parameters, and intended tasks and subtasks.
- AI Asset Characterization: EMMM models machine-learning asset types, relationships, and version-control structures, but captures element metadata through arbitrary key-value mappings lacking common semantics.
III. EXCHANGE PLATFORM
AIMDEP centralizes AI models and datasets, enriches them with AIMDEO metadata, and supports registration, semi-automatic feature description, search, and deployment-oriented operations.
- AIMDEP provides a central registry that makes AI models and datasets visible and accessible while capturing metadata through AIMDEO.
- The platform supports online deployment through a standard platform-independent interface.
- A client-server architecture with a web interface and REST API supports platform operations and custom client interaction.
- Asset registration uploads physical files, identifies required data handlers or machine-learning frameworks, and supports common dataset formats and popular model exports.
- Input and output features are detected semi-automatically, verified by users, stored as AIMDEO parameters, and supplemented with configuration and evaluation metadata.
B. Asset Operations
AIMDEP supports visualization, deployment, search, and download operations for registered datasets and models, using metadata to aid analysis, inference, and asset discovery.
- Dataset Operations: Datasets can be explored through statistical tables, plots, and numerical feature analyses to support later model training and evaluation.Feature analyses identify the importance of each feature.
- Model Operations: Registered models can be deployed online through platform-independent REST API or graphical interfaces generated from model tasks and features.Inference is available through REST and Gradio interfaces.
- Search and Download: Metadata-powered search returns relevant models and datasets, while downloads include original files and asset attributes packaged as micro-ontologies.Assets can be downloaded for offline use or integration into custom workflows.
IV. ONTOLOGY
AIMDEO is an OWL ontology designed to capture concepts needed for collaborative AI model development and use. Its concepts were identified from collaboration scenarios and organized as essential ontology elements.
- Ontology: AIMDEO captures essential concepts involved in collaborative AI model development and use as an OWL ontology.The paper reports overall ontology metrics in Table 1.
- Ontology Metrics: Table 1 presents the overall metrics of the AIMDEO ontology.
- Ontology Development: The ontology’s required concepts and relationships were identified by examining different collaboration scenarios.These concepts are modeled as OWL classes and properties.
- Collaboration Scenarios: Figure 3 illustrates the AI model collaboration scenarios used to understand the concepts required for collaborative development and use.
1) Internal Use:
Internal use keeps models and datasets with their creator while allowing models to be used within development environments and industrial tiers, with results and feedback exchanged across environments.
- Internal Use: AI models and datasets remain with the creator during internal use.
- Internal Use: AI models can be used from an AI development environment or within an automotive-industry tier, with results stored in product development and feedback returned.
- Shared Data: When data cannot be captured internally, supply-chain participants may exchange development data for use in the AI development environment.This collaboration form requires data descriptions and well-defined access interfaces.
- Model Sharing: Another cooperation form shares trained models instead of sharing data from the product development environment.
4) Model Services:
The paper describes service-based cooperation and an ontology-supported exchange workflow in which stakeholders develop, register, discover, and apply models for safety-critical software-timing analysis.
- 4) Model Services:: In model-service cooperation, users send requests to a service that delegates them internally to an AI model instead of exchanging the model directly.
- IV. ONTOLOGY: Ontology construction models concepts and relationships identified from usage scenarios as OWL classes and properties.Essential concepts are listed in Table 2.
- V. EVALUATION: The use case involves domain experts, AI experts, and end users collaborating on a model that predicts memory access time for safety-critical software timing.
- V. EVALUATION: A domain expert registers a dataset with cache characteristics in AIMDEP, giving the AI expert knowledge to design a suitable model.The described characteristics include replacement strategy and size.
- V. EVALUATION: The registered model is annotated through the ontology so an end user can search for a suitable model and assess software timing with a described memory configuration.The assessment supports evaluating whether test hardware should be purchased.
- V. EVALUATION: The use case demonstrates the exchange workflow with the participating stakeholders.
A. Dataset Exchange
AIMDEP supports dataset publication by automatically recognizing features, guiding metadata entry, and storing dataset information through AIMDEO for export.
- A. Dataset Exchange: AIMDEP semi-automatically recognizes supported dataset features during upload, reducing errors from omitted features.Users can then add descriptions and value ranges through the platform’s input mask.
- A. Dataset Exchange: Dataset metadata and configuration parameters, including memory replacement strategy and memory size, are stored using AIMDEO.The resulting dataset representation can be exported as OWL files for download.
B. AI Model Exchange
AI Model Exchange uses AIMDEO to describe model inputs, outputs, evaluation information, configuration parameters, and model metadata, while AIMDEP supports collaborative development and tool integration. Compatibility with other metamodels remains future work.
- B. AI Model Exchange: AI models are described through input and output parameters, feature data types, optional value ranges and distributions, and configuration parameters.The ontology also captures evaluation information, framework, model description, publication date, and the assessment dataset.
- B. AI Model Exchange: AIMDEP was used successfully for collaborative AI model development, while AIMDEO can separately help ensure required information is available and specify how it is provided.The platform’s metadata specification and export functionality can help integrate tools in the development process.
- B. AI Model Exchange: AIMDEO compatibility with EMMM and ITO is identified as future work.One proposed direction is storing AIMDEO metadata in EMMM’s generic key-value pairs.