Systematically Optimized CNN-Transformer with Focal Loss for Imbalanced Intrusion Detection on NSL-KDD

• Severe NSL-KDD imbalance makes overall accuracy potentially misleading because majority-class success can coexist with errors on rare R2L and U2R attacks.Source: §1 · p. 2
• The framework combines XGBoost feature selection, a CNN-Transformer, Focal Loss, Optuna optimization, and SHAP explanations for minority-class predictions.The CNN extracts local patterns while transformer attention captures long-range dependencies.Source: §1 · pp. 2–3
• The study examines how Focal Loss and oversampling interact, reporting that tuned Focal Loss alone can outperform combinations in optimized settings and avoid over-correction.The introduction also identifies careful data handling to avoid leakage as part of its methodological contribution.Source: §1 · p. 3
• The model achieves high overall performance and considerably improves R2L and U2R F1-scores versus baselines without specialized imbalance methods.The introduction reports the comparison qualitatively, without metric values.Source: §1 · p. 3

2 Related Work

• The section surveys IDS deep learning, class-imbalance mitigation, and explainability, and positions the work among these areas.Source: §2 · p. 3

2.1 Deep Learning Approaches for IDS

• Hybrid neural models combine feature-learning approaches; the paper pairs CNN local-feature extraction with transformer attention for global context on imbalanced NSL-KDD.Prior work cited includes CNN-LSTM IDS and transformer-based IDS studies.Source: §2.1 · pp. 3–4

2.2 Addressing Class Imbalance in IDS

• The work addresses limited systematic evidence on combining data-level oversampling such as SMOTE with algorithm-level Focal Loss, finding tuned Focal Loss most effective in its framework.Source: §2.2 · p. 4

2.3 Explainable AI (XAI) in IDS

• Deep-learning opacity can constrain trust and adoption in security-critical settings; SHAP offers local and global feature attributions but carries post-hoc computational costs for real-time use.Source: §2.3 · pp. 4–5

3 Our Methodology

• The workflow covers dataset preparation, feature selection, model design, imbalance handling, optimization, evaluation, and explainability.It is presented as a systematic approach to reliable and interpretable results.Source: §3 · p. 5

3.1 Dataset Description

• NSL-KDD records contain 41 features and are grouped into normal traffic and DoS, Probe, R2L, and U2R classes, with R2L and especially U2R scarce.The dataset is a cleaned version of KDD Cup 1999 intended to address duplicate records.Source: §3.1 · p. 5

3.2 Data Preprocessing

• One-hot encoding and MinMax scaling expand the original 41 features into a 122-dimensional numerical vector.Numerical features are scaled to [0, 1] using training-set minima and maxima.Source: §3.2 · pp. 6–7

3.3 Feature Selection

• XGBoost ranks features by gain, and the 35 highest-gain features are selected as deep-learning inputs.Gain is the average training-loss or classification-impurity reduction from splits using a feature.Source: §3.3 · p. 7

3.4 Model Architecture

• ACT-IDS uses CNN blocks for local patterns and Transformer encoder layers for global dependencies before a classification head produces five class outputs.The CNN output is adaptively pooled to sequence length 16 before entering the Transformer layers.Source: §3.4 · pp. 7–8

3.5 Imbalance Handling Strategy

• Focal Loss with γ = 1.5 focuses training on hard-to-classify minority samples, while the final model omits oversampling after SMOTE over-corrected in ablations.Balancing approaches were restricted to the training partition to avoid data leakage.Source: §3.5 · p. 9

3.6 Hyperparameter Optimization

• Optuna searched 50 trials and selected top-k = 35, four attention heads, ff dim = 256, learning rate = 0.00042, and batch size = 256.Focal Loss gamma was selected separately through manual ablation.Source: §3.6 · p. 9

3.7 Training details

• Final training used Adam, learning rate 0.00042, Focal Loss with γ = 1.5, and batch size 256, with early stopping after seven validation-loss patience epochs.Training ran for at most 50 epochs and saved weights with the lowest validation loss.Source: §3.7 · p. 9

3.8 Performance Metrics

• Accuracy, precision, recall, and F1-score are used as conventional confusion-matrix performance metrics.F1-score is described as the harmonic mean of precision and recall.Source: §3.8 · p. 10
• Metrics are reported by class and as overall weighted averages on the untouched test set, accounting for class imbalance during evaluation.The weighted averages reflect the test-set class distribution.Source: §3.8 · p. 10

3.9 Explainable AI

• SHAP with GradientExplainer quantifies feature contributions to individual predictions and supports global and local summaries, including U2R cases and false positives.The analysis is used to examine the model's decision logic, including minority-class behavior.Source: §3.9 · p. 10

4.1 Baseline Performance

• 97.89% overall accuracy coexisted with 30.61% U2R precision in the no-oversampling, γ = 0.0 baseline.The authors interpret the high recall and low precision for U2R as indicating many false positives.Source: Table 1 · p. 11, §4.1 · p. 11

4.2 Ablation Study 1: Impact of Focal Loss Gamma

• γ = 1.5 was selected as the best balance across overall metrics, R2L F1, and U2R F1; γ ≈ 3.0 sharply reduced precision.The ablation used SMOTE-balanced training data and evaluated on the untouched test set.Source: §4.2 · p. 11, Table 2 · p. 12

4.3 Ablation Study 2: Impact of Oversampling Techniques

• With γ = 1.5, oversampling produced the best F1-scores for R2L (84.63%) and U2R (69.66%), but oversampling combinations reduced precision, particularly for U2R.The authors nevertheless chose the non-oversampling condition, judging oversampling an apparent over-correction.Source: Table 3 · p. 12, §4.3 · p. 12

4.4 Final Model Performance

• The last model reached 98.73% overall accuracy and 98.70% weighted F1-score; R2L F1 was 84.63% and U2R F1 was 69.66%, making results more balanced than the baseline.The final optimized model used γ = 1.5 with no oversampling.Source: §4.4 · p. 13, Table 4 · p. 13

4.5 Comparison with other Methodologies

• The experiments report a comparison of model performance with other methodologies in Table 5, but this section provides no comparison values.Source: §4.5 · p. 13

5.1 Interpretation of final results

• 98.73% accuracy and 98.70% weighted F1; R2L F1 was 84.63% versus 79.45% baseline, and U2R F1 was 69.66% versus 45.69% baseline.Source: §5.1 · p. 13, Table 5 · p. 14

5.2.1 Optimizing focal loss

• γ = 1.5 was the best reported trade-off between minority recall/F1, reasonable precision, and overall accuracy; γ ≈ 3.0 drastically diminished precision.Source: §5.2.1 · p. 14

5.2.2 Focal Loss vs. Oversampling

• The sampler ablation found degraded performance when Focal Loss was combined with SMOTE, BorderlineSMOTE, ADASYN, SMOTEENN, or SMOTETomek.The discussion concludes that Focal Loss alone was sufficient to handle imbalance in this model.Source: §5.2.2 · p. 14

5.3.1 Global Feature importance

• Global mean absolute SHAP values highlighted service type, destination-host connection patterns, login status, and connection frequency as important features.Source: §5.3.1 · pp. 14–15, Figure 3 · p. 15

5.3.2 U2R-Specific Feature Importance

• For U2R predictions, high logged-in and FTP-data values increased prediction likelihood, while high service-http values decreased it.Low error rates and connection counts also pushed predictions toward U2R.Source: §5.3.2 · p. 15

5.3.3 Local Explanation (U2R False Positive)

• A Normal record predicted as U2R was driven mainly by FTP-data, same-source-port rate, absent S0 flags, and low destination-host error rate.The authors note that the features pushing away from U2R had very little impact.Source: §5.3.3 · pp. 15–16

6 Conclusion

• The framework combines a CNN-Transformer, XGBoost feature selection, and tuned Focal Loss for imbalanced NSL-KDD intrusion detection.The conclusion identifies γ = 1.5 without oversampling as the best-performing configuration in the optimized framework.Source: §6 · p. 16
• 98.73% accuracy was accompanied by R2L F1 of 84.63% and U2R F1 of 69.66%.Source: §6 · p. 16
• SHAP analysis identified overlapping features, including FTP-data and logged-in status, as the primary contributor to U2R precision challenges.Source: §6 · p. 16

Ethical Statement

• The authors report that they performed no studies involving human or animal subjects.Source: Ethical Statement · p. 17

• The manuscript has no associated data.Source: Data Availability · p. 17

• The research did not receive any specific grant from funding agencies.Source: Funding Declaration · p. 17