Source-linked AI summary
The Accuracy Paradox: Empirical Diagnostic of Default Decision Thresholds in Multi-Label Enzyme Commission Prediction [With Code]
Bilal Ahmad, Rajed Mehmood
TL;DR
Fixed 0.50 decision thresholds are commonly used for multi-label enzyme prediction despite uneven label prevalence. This study diagnoses their behavior across six EC targets and finds that high mean accuracy masks poor macro metrics and complete EC6 collapse, motivating target-specific thresholding and post-hoc calibration.
Problem
Fixed 0.50 thresholds are used for multi-label enzyme targets despite nonuniform label prevalence and multi-functional co-activation.
Source: §1 · p. 2Method
The study empirically evaluates fixed-threshold predictions across six EC targets in 14,096 compounds and examines molecular-feature correlations.
Source: §1 · p. 3, §2.3 · p. 6, §3.1 · p. 6Results
77.16% mean accuracy coexists with macro F1 of 0.3976 and macro recall of 0.3872; EC6 has zero true positives despite ROC-AUC of 0.5857.
Source: §5 · p. 10Takeaways & Limitations
The paper recommends target-specific threshold optimization and post-hoc conformal calibration as safeguards for biological multi-label pipelines.
Source: §4 · p. 10
Abstract
from arXiv · showhide
Automated prediction of Enzyme Commission (EC) numbers plays a central role in functional annotation and computational drug discovery. However, standard multi-label machine learning pipelines frequently rely on default decision thresholds (t=0.50), assuming balanced prior distributions across target heads. In this study, we present a systematic empirical diagnostic of uncalibrated fixed decision boundaries operating under severe class imbalance across N = 14,096 annotated compounds categorized into six primary EC classes (EC1-EC6). Our results highlight a pronounced Accuracy Paradox: while the multi-label system achieves a deceivingly high mean accuracy of 77.16%, the macro F1-score (0.3976) and macro recall (0.3872) reveal severe predictive breakdown. Majority target classes suffer from hyper-sensitivity and over-prediction, whereas minority classes exhibit sharp recall decay, culminating in a total decision boundary collapse for EC6 (Recall = 0.00%) despite underlying discriminative power (ROC-AUC = 0.5857). Feature correlation analysis further reveals high linear redundancy among topological indices relative to fingerprint density metrics. Ultimately, this diagnostic study demonstrates that standard point predictions mask critical errors in bioinformatics workflows. We establish target-specific threshold optimization and post-hoc conformal calibration as essential, open-source post-processing safeguards for reliable applied machine learning and deep learning architectures.
1. Introduction
Multi-label enzyme prediction must contend with uneven label prevalence and co-activation, while fixed 0.50 thresholds are often used as if label priors were balanced. This study diagnoses the resulting decision-boundary failures and examines feature redundancy and post-processing recommendations.
1. Introduction
- EC numbers classify biological catalysts into six functional classes, and accurate computational annotations support drug discovery, metabolic reconstruction, and industrial biotechnology,.Virtual screening and molecular docking also depend on precise functional assignments.Source: §1 · p. 2
- Imbalanced label prevalence and multi-functional co-activation create interdependent targets that standard multi-output predictors struggle to model.Highly prevalent reactions dominate training, while specialized pathways remain sparse.Source: §1 · p. 2
- Fixed 0.50 thresholds assume balanced label priors, but on skewed biological data they can mask majority-target over-prediction and minority-target recall collapse –.Source: §1 · pp. 2–3
- The study empirically diagnoses fixed-threshold decision boundaries across 14,096 annotated compounds rather than proposing an incremental state-of-the-art predictor.Source: §1 · p. 3
- The contributions cover descriptor redundancy, threshold-induced target failures, and recommendations for target-specific threshold optimization and post-hoc conformal calibration.Source: §1 · p. 3
2. M ethodology
The study formulates enzyme prediction as six parallel binary target tasks, evaluates them on an imbalanced dataset, and analyzes molecular descriptors and fixed-threshold performance. It uses topological and fingerprint-density features and reports macro-averaged metrics across targets.
2.1. M ulti-Label Framework
- Each compound is represented by a molecular feature vector and classified into a six-element binary vector of EC targets.Source: §2.1 · p. 3
2.2. Dataset
- The dataset contains 14,096 compounds, with EC1 and EC2 more prevalent than EC5 and EC6.Active ratios are 66.75% and 80.09% for EC1 and EC2, versus 14.46% and 15.10% for EC5 and EC6.Source: §2.2 · p. 4, Figure 2 · p. 4
2.3. Feature Analysis
- Molecular inputs include global topological and connectivity descriptors alongside local Morgan fingerprint-density measures.Source: §2.3 · p. 5
- Linear relationships among structural features and targets are measured with Pearson correlation.Source: §2.3 · p. 5
- Predicted probabilities are converted to binary labels at τ = 0.50, with accuracy, precision, recall, F1, and ROC-AUC evaluated per target and macro-averaged across targets.Source: §2.3 · p. 6
3. Results and Discussion
Under the fixed 0.50 threshold, high mean accuracy coexists with weak macro F1 and recall, and target-level results show distinct majority- and minority-class failures. Feature analysis also finds strong topological redundancy and sparse co-activation among some classes.
3.1. Exploratory Data Analysis
- Topological descriptors are strongly correlated, while Morgan fingerprint densities have near-zero linear correlation with EC targets; EC1–EC2 co-activate in 7,151 samples versus 245 for EC5–EC6.Chi1 correlates with Chi2v at r = 0.971 and ExactMolWt at r = 0.960; fingerprint-density correlations with targets have |r| < 0.025.Source: §3.1 · p. 6
3.2. M ulti-Label Classification Performance & The Accuracy Paradox
- 77.16% mean accuracy contrasts with macro F1 of 0.3976 and macro recall of 0.3872; mean ROC-AUC is 0.6451.Source: §3.2 · p. 7
3.3. Target-by-Target Failure Dynamics
- The target-level diagnostic contrasts majority-class over-prediction with minority-class false negatives and complete EC6 collapse under τ = 0.50.Source: §3.3 · p. 7
3.3.1 M ajority Class Over-Prediction (EC1 & EC2)
- EC2 achieves 100.00% recall but only TN = 1, while EC1 has 3,034 false positives despite an F1-score of 0.7946.Source: §3.3.1 · p. 9
3.3.2 M inority Class Sensitivity Decay & Target Collapse (EC3–EC6)
- Recall falls to 16.34% for EC3, 18.25% for EC4, and 10.55% for EC5; EC6 has zero true positives despite ROC-AUC of 0.5857.For EC6, all 2,129 active test instances are misclassified as inactive.Source: §3.3.2 · p. 9
4. M ethodological Implications & Recommendations
The results indicate that fixed-threshold point predictions are insufficient for imbalanced multi-label biological targets. The paper recommends target-specific thresholds and post-hoc conformal calibration as deployment safeguards.
4. M ethodological Implications & Recommendations
- Fixed-threshold point predictions are insufficient for imbalanced multi-label biological targets because raw accuracy masks errors in minority categories.Source: §4 · p. 10
- Target-specific thresholds, such as those optimized for Precision-Recall AUC or Fβ scores, are recommended to recover sensitivity on collapsed classes such as EC6.Source: §4 · p. 10
- The paper recommends post-hoc probability calibration with class-conditional Mondrian conformal prediction to bound coverage errors across EC categories.Source: §4 · p. 10
5. Conclusion
Across six EC target heads, default uncalibrated thresholds yielded high mean accuracy alongside low macro F1 and recall. Target-level failures and feature redundancy sharpen the diagnostic.
5. Conclusion
- 77.16% mean accuracy contrasted with macro F1-score 0.3976 and macro recall 0.3872, revealing diagnostic failures under default thresholds.The evaluation covered 14,096 compounds across EC1–EC6.Source: §5 · p. 10
- EC2 (80.09% active) reached Recall = 100.00% but TN = 1, while EC6 (15.10% active) had TP = 0 and F1 = 0.0000 despite ROC-AUC = 0.5857.As prevalence falls below 35%, sensitivity degrades rapidly across EC3–EC6.Source: §5 · p. 10
- Connectivity indices were strongly redundant (r > 0.90), unlike local fingerprint densities (|r| < 0.025); fixed thresholding also failed under nonuniform targets.Source: §5 · p. 10
- Code is available on request by correspondence email.Source: §5 · p. 10