A Hybrid Intrusion Detection Framework Using Deep Autoencoder and Machine Learning Models

Salam Allawi Hussein, Sándor R. Répás

This study provides a detailed comparative analysis of a three-hybrid intrusion detection method aimed at strengthening network security through precise and adaptive threat identification. The proposed framework integrates an Autoencoder-Gaussian Mixture Model (AE-GMM) with two supervised learning techniques, XGBoost and Logistic Regression, combining deep feature extraction with interpretability and stable generalization. Although the downstream classifiers are trained in a supervised manner, the hybrid intrusion detection nature of the framework is preserved through unsupervised representation learning and probabilistic modeling in the AE-GMM stage. Two benchmark datasets were used for evaluation: NSL-KDD, representing traditional network behavior, and UNSW-NB15, reflecting modern and diverse traffic patterns. A consistent preprocessing pipeline was applied, including normalization, feature selection, and dimensionality reduction, to ensure fair comparison and efficient training. The experimental findings show that hybridizing deep learning with gradient-boosted and linear classifiers markedly enhances detection performance and resilience. The AE–GMM-XGBoost model achieved superior outcomes, reaching an F1-score above 0.94 ± 0.0021 and an AUC greater than 0.97 on both datasets, demonstrating high accuracy in distinguishing legitimate and malicious traffic. AE-GMM-Logistic Regression also achieved strong and balanced performance, recording an F1-score exceeding 0.91 ± 0.0020 with stable generalization across test conditions. Conversely, the standalone AE-GMM effectively captured deep latent patterns but exhibited lower recall, indicating limited sensitivity to subtle or emerging attacks. These results collectively confirm that integrating autoencoder-based representation learning with advanced supervised models significantly improves intrusion detection in complex network settings. The proposed framework therefore provides a solid and extensible basis for future research in explainable and federated intrusion detection, supporting the development of adaptive and proactive cybersecurity defenses.