Machine Learning-Based Classification of Encrypted Network Traffic and Domain Generation Algorithm (DGA) Threats

DOI

https://www.doi.org/10.59256/indjcst.20260503003

Published In

Volume 5, Issue 3 (2026)

Published Online

September-December 2026

Pages

12-17

Article Type

Case Study

Check for updates

Modern cybersecurity landscapes face a dual threat vector characterized by the widespread adoption of payload encryption for covert communications and the proliferation of Domain Generation Algorithms (DGAs) used by advanced malware to dynamically establish Command and Control (C2) infrastructure. Traditional signature-based detection mechanisms and static IP/domain blacklists frequently fail against these evasive techniques. This paper proposes an integrated, machine learning-driven detection framework designed to simultaneously classify encrypted network traffic patterns and identify algorithmically generated malicious domains. By extracting behavioral flow statistics, packet length distributions, and linguistic character properties (such as entropy, lexical length ratios, and N-gram frequency distributions) from network telemetry, we construct a high-dimensional feature matrix. We evaluate and benchmark multiple supervised classifiers—including Random Forest, Extreme Gradient Boosting (XGBoost), and Long Short-Term Memory (LSTM) deep learning networks—to separate benign traffic and legitimate domains from encrypted C2 tunnels and DGA variants. Experimental results demonstrate that ensemble tree models combined with sequence-aware architectures achieve high classification accuracy, minimal false-positive rates, and low processing latency, offering enterprise Security Operations Centers (SOCs) an automated blueprint for proactive threat hunting.