Source-linked AI summary
AllMetrics: A Unified Python Library for Standardized Metric Evaluation and Robust Data Validation in Machine Learning
Morteza Alizadeh, Mehrdad Oveisi, Sonya Falahati, Ghazal Mousavi, Mohsen Alambardar Meybodi, Somayeh Sadat Mehrnia, Ilker Hacihaliloglu, Arman Rahmim, Mohammad R. Salmanpour
TL;DR
Machine-learning evaluation is hindered by inconsistent metric implementations, reporting formats, and insufficient validation. AllMetrics addresses these gaps with a unified, configurable library spanning multiple tasks, and comparisons report alignment with established tools alongside remaining scalability and coverage boundaries.
Problem
Method
Results
AllMetrics resolves evaluation inconsistencies across five task families, with regression outputs exactly matching Scikit-Learn and Matlab and matching TorchMetrics to seven decimal places for tested R² cases.
Source: §3.2 · pp. 5–6Takeaways & Limitations
AllMetrics offers a unified framework for reproducible and trustworthy machine-learning model evaluation across diverse task types.
Source: §5 · p. 9Extremely large datasets may require further memory optimization, and future versions could add metrics for current and specialized tasks.
Source: §3.3 · p. 7
Abstract
from arXiv · showhide
Machine learning (ML) models rely heavily on consistent and accurate performance metrics to evaluate and compare their effectiveness. However, existing libraries often suffer from fragmentation, inconsistent implementations, and insufficient data validation protocols, leading to unreliable results. Existing libraries have often been developed independently and without adherence to a unified standard, particularly concerning the specific tasks they aim to support. As a result, each library tends to adopt its conventions for metric computation, input/output formatting, error handling, and data validation protocols. This lack of standardization leads to both implementation differences (ID) and reporting differences (RD), making it difficult to compare results across frameworks or ensure reliable evaluations. To address these issues, we introduce AllMetrics, an open-source unified Python library designed to standardize metric evaluation across diverse ML tasks, including regression, classification, clustering, segmentation, and image-to-image translation. The library implements class-specific reporting for multi-class tasks through configurable parameters to cover all use cases, while incorporating task-specific parameters to resolve metric computation discrepancies across implementations. Various datasets from domains like healthcare, finance, and real estate were applied to our library and compared with Python, Matlab, and R components to identify which yield similar results. AllMetrics combines a modular Application Programming Interface (API) with robust input validation mechanisms to ensure reproducibility and reliability in model evaluation. This paper presents the design principles, architectural components, and empirical analyses demonstrating the ability to mitigate evaluation errors and to enhance the trustworthiness of ML workflows.
1. INTRODUCTION
AllMetrics addresses inconsistent metric computation and reporting, weak interoperability, and inadequate input validation across machine-learning evaluation tools. It unifies metric implementations and validation across five task families through a modular, reproducible API.
Motivation
- Missing values, imbalanced classes, and mismatched dimensionalities further undermine evaluation when libraries lack rigorous input validation.These issues can affect metric computation across machine-learning tasks.Source: §1 · p. 1
- Existing evaluation libraries produce unreliable comparisons because metric implementations and reporting conventions vary across platforms.These inconsistencies are identified as implementation differences (ID) and reporting differences (RD), which hinder reproducible research and benchmarking.Source: §1 · p. 1
Related work
- Existing tools remain fragmented across task and language ecosystems, with specialized libraries offering limited interoperability and no unified metric-validation framework.Examples include Scikit-Learn, TorchMetrics, Scikit-Image, SciPy, OpenCV, caret, PyCM, and TensorFlow.Source: §1 · pp. 1–2
Contributions
- AllMetrics standardizes metric computation across regression, classification, clustering, segmentation, and image-to-image tasks while adding automated checks for common edge cases.The library's modular, task-agnostic design is intended to support reproducible integration across evaluation workflows.Source: §1 · p. 2
- AllMetrics combines a standardized API, integrated anomaly detection and handling, and empirical analysis of inconsistencies in existing libraries.Its validation framework enforces criteria such as handling missing regression values and non-binary segmentation masks before computation.Source: §1 · p. 2
2. PROGRAMMING PREPARATION CRITERIA
AllMetrics organizes metrics by task and provides discovery tools alongside a modular validation framework. Shared and task-specific checks can be run through configurable interfaces, supporting thorough evaluation, efficient processing, and custom extensions.
2.1 Metric organization and accessibility
- AllMetrics organizes independent modules for regression, classification, clustering, segmentation, and image-to-image translation, covering metrics such as R², F scores, Dice, IoU, SSIM, and PSNR.The package is installable through PyPI with “pip install allmetrics”.Source: §2.1 · p. 2, §2 · p. 2
- Metric discovery uses list_of_metrics(), while get_metric_details() exposes parameter specifications and usage guidance for transparent metric selection.Table 1 compares AllMetrics metric availability with other major libraries; checkmarks indicate support and dashes indicate no native implementation.Source: §2.1 · p. 2, Table 1 · p. 3
2.2.1 Core validation functions
- Core validation checks verify data types, structures, dimensionality, missing or infinite values, length consistency, variance, logarithmic non-negativity, and statistical outliers.Optimized sampling maintains validation efficiency for large datasets.Source: §2.2.1 · p. 2
2.2.2 Task-specific validation extensions
- Task-specific validators extend shared checks to detect regression multicollinearity, classification label and probability issues, clustering dimensionality and separation problems, segmentation mask and spatial errors, and image channel inconsistencies.The extensions also address class imbalance, binary-input requirements, empty masks, and pixel-value normalization.Source: §2.2.2 · pp. 2–3
2.2.3 Comprehensive validation interface
- The unified validate_all() interface runs relevant checks with configurable activation, balancing validation thoroughness and computational efficiency.The modular architecture also supports custom validation rules for specialized applications.Source: §2.2.3 · p. 3
- The library demonstrates direct metric use through simple task-specific calls, such as computing IoU from ground-truth and predicted segmentation masks.The example instantiates arrays and passes them directly to iou_score.Source: §2.2.3 · p. 4
3. EVALUATION
AllMetrics was evaluated across diverse ML tasks and datasets against established libraries, with experiments targeting accuracy, efficiency, and robustness. Results show standardized outputs, strong cross-library agreement, configurable reporting, validation of edge cases, and identified scalability and coverage limits.
3.1 Experimental setup
- AllMetrics was benchmarked across regression, classification, clustering, segmentation, and image-to-image translation using established datasets, models, and libraries.Comparisons used each library's default parameters to isolate implementation differences rather than parameter tuning effects.Source: §3.1 · p. 4, §3.2 · p. 5
3.2 Benchmark results
- AllMetrics resolves implementation and reporting inconsistencies across five ML tasks through standardized implementations and configurable reporting options.The evaluation explicitly targets both implementation differences and reporting differences identified in prior metric-evaluation work.Source: §3.2 · p. 5
- AllMetrics preserves per-class classification results, matching PyCM while other frameworks often return aggregated, micro-averaged, or macro-averaged values.For binary classification, AllMetrics reports separate values for classes 0 and 1; for multi-class classification, it returns per-class results by default.Source: §3.2 · p. 5, Table 3 · p. 5, Table 4 · p. 5
- AllMetrics exactly matches Scikit-Learn and Matlab for tested regression R² values and matches TorchMetrics to seven decimal places.The comparison used the Weather Dataset and reported agreement across all tested cases.Source: §3.2 · p. 6, Table 5 · p. 6
- AllMetrics aligns with Scikit-Learn for Rand Score, Silhouette Score, and Davies-Bouldin Index, supporting compatibility with existing clustering workflows.The reported agreement covers both similarity and clustering-quality measures.Source: §3.2 · p. 6
- AllMetrics produces segmentation results closely aligned with MedPy on 240 lung CT tumor segmentations from eight independent multi-center datasets.The comparison is presented as evidence of implementation accuracy and compatibility with clinical and research tools.Source: §3.2 · p. 6, Table 6 · p. 6
- AllMetrics achieved the second-fastest SSIM computation among four libraries on the Prostate Cancer image-to-image translation benchmark.SSIM results varied across libraries because implementations used different window-size choices, while AllMetrics supports a user-adjustable window size.Source: §3.2 · p. 7, Figure 1 · p. 7, Table 6 · p. 7
- Robust validation detects empty segmentation masks, severe classification imbalance, mismatched dimensions, missing regression values, and non-binary masks before or during metric computation.The validation mechanisms are intended to maintain reliable computation across diverse scenarios while preserving computational efficiency.Source: §3.2 · p. 7
4. DISCUSSION
AllMetrics addresses implementation and reporting differences in ML metrics through configurable, task-specific standardization and rigorous validation. Its discussion emphasizes flexibility across classification, regression, segmentation, and image-to-image evaluation while targeting reproducible comparisons.
Standardization
- Configurable standardization targets both implementation and reporting differences, enabling more consistent metric evaluation across ML frameworks.AllMetrics uses standardized implementations with configurable computation parameters and an explicit average parameter for classification reporting.Source: §4 · p. 8
Classification
- Classification reporting supports macro, micro, weighted, and class-wise outputs through the average parameter.The get_metric_details() function documents metric parameters and usage guidelines.Source: §4 · p. 8
Regression
- R² evaluation supports both traditional linear and adjusted formulations through explicit computation-method control.The formulations align with Scikit-Learn and TorchMetrics or with StatsModels and Yardstick, respectively.Source: §4 · p. 8
Segmentation
- Segmentation metrics use direct Euclidean distances between foreground boundary points, support HD95, and return ∞ for empty masks.This differs from MONAI’s distance-transform approach and is described as more geometrically accurate and interpretable for sparse or noisy segmentation tasks.Source: §4 · p. 8
Image-to-image translation
- AllMetrics makes SSIM window dimensions configurable, using a 7×7 default instead of the fixed 11×11 window used by TensorFlow and TorchMetrics.The window_size parameter allows smaller windows for localized detail or larger windows for global structure preservation.Source: §4 · p. 8
Reliability
- Rigorous validation complements standardized computation by proactively identifying input-quality problems and supporting reliable, reproducible evaluations.The design combines mathematically precise implementations with intuitive interfaces and comprehensive validation.Source: §4 · pp. 8–9
5. CONCLUSION
AllMetrics provides a unified, open-source framework for standardized and reliable ML evaluation across diverse tasks. Its modular architecture and automated validation are presented as supporting reproducible and comparable benchmarking in research and production.
Conclusion
- AllMetrics unifies task-specific metrics and automated validation across regression, classification, clustering, segmentation, and image-to-image translation.The framework checks for empty masks, class imbalances, and dimensional inconsistencies while providing an intuitive API.Source: §5 · p. 9
- The library’s open-source availability, optimized implementations, and modular architecture support use in research and production environments.The source code, documentation, and usage examples are freely available through the project repository.Source: §5 · p. 9
- AllMetrics is presented as improving the reproducibility and comparability of ML evaluations by resolving implementation and reporting discrepancies.The conclusion frames this as supporting more rigorous and transparent benchmarking.Source: §5 · p. 9