Lightweight CNN-LSTM framework spots malicious traffic in encrypted networks

Modern enterprise networks have become a paradox for their own defenders: the same encryption that protects customer data also blinds the intrusion detection systems trying to spot attackers hiding inside the traffic. A study published in Discover Artificial Intelligence on 24 October 2026 tackles this problem with a deliberately lightweight hybrid deep learning framework that combines convolutional and recurrent neural networks to flag malicious flows in real time, even when most of the traffic is TLS- or SSL-encrypted, severely imbalanced, and drawn from fundamentally different network environments. The work, led by researcher Marzieh Ghavidel, is motivated by the security needs of Software-Defined Wide Area Networks, or SD-WAN, an architecture that decouples network control from data forwarding and has become a foundational component of modern enterprise communication infrastructures. Yet the study is unusually candid about its own scope: no SD-WAN testbed or SD-WAN traffic was used in any experiment, and the framework was evaluated on three heterogeneous benchmark datasets rather than on operational SD-WAN deployments.

The difficulty the framework addresses is well known to network security researchers. Traditional signature-based systems and classical machine learning detectors degrade sharply when confronted with encrypted communications, large-scale traffic volumes, and previously unseen attack behaviors, because encryption hides the payload content on which many detectors rely. At the same time, most published deep learning intrusion detectors are evaluated on a single benchmark dataset under controlled laboratory conditions, which inflates reported accuracy and conceals dataset-specific overfitting. Class imbalance compounds the problem: in datasets such as UNSW-NB15, malicious flows span nine attack categories but many categories contain only sparse samples, making minority attacks statistically easy to miss. The new study argues that the field needs evaluation protocols that test robustness across heterogeneous traffic domains, cross-dataset generalization under controlled feature overlap, and computational efficiency on commodity hardware, rather than yet another architecture chasing a record on one dataset. The framework’s contribution, the author emphasizes, lies in this evaluation protocol rather than in any architectural novelty.

Architecturally, the model is a compact instantiation of an established combination. A two-stage one-dimensional convolutional neural network first extracts local spatial correlations from flow-level feature vectors: the first block applies 64 filters of kernel size three followed by ReLU activation and max pooling, and the second block repeats the pattern with 128 filters. Between the CNN and the temporal stage sits a lightweight channel attention module following the squeeze-and-excitation design. Global average pooling compresses each feature channel into a descriptor, a two-layer bottleneck network with a reduction ratio of four produces channel importance weights, and the original feature map is rescaled channel-wise. The module adds only 8,192 parameters, less than 3.5 percent of the total model, and less than 0.1 milliseconds of per-flow latency overhead. Stacked LSTM layers with 128 and 64 memory units then model sequential dependencies across windows of ten consecutive flows, built with a stride of one, before a softmax output layer performs the final benign-versus-malicious classification.

The evaluation spans three deliberately different traffic domains. CICIDS-2017, created by the Canadian Institute for Cybersecurity, contains over 2.8 million records collected across five days of simulated traffic, described by 78 flow-level statistical features and encompassing denial-of-service, distributed denial-of-service, botnet, brute-force, infiltration, and web attacks. UNSW-NB15, from UNSW Canberra, comprises roughly 2.5 million records with 49 features across nine attack categories and is characterized by significant class imbalance. CERNET-2020, obtained through official academic access channels from the China Education and Research Network backbone, contains over 4.5 million flow-based records captured over ten days, represented through NetFlow/IPFIX-derived statistics reduced to 38 features by chi-square selection, with a substantial proportion of TLS/SSL-encrypted flows including encrypted command-and-control traffic. Crucially, the framework operates exclusively on statistical traffic features such as packet lengths, inter-arrival times, flow duration, and byte counts; it never decrypts payloads. Encryption obscures content but not the underlying statistical and temporal patterns, which malicious encrypted traffic may still betray.

On the most operationally challenging benchmark, the encrypted CERNET dataset, the framework achieved 98.1 percent accuracy, a 97.6 percent F1-score, and an AUC of 0.99, with a false-positive rate of 2.1 percent and a false-negative rate of 2.5 percent. On CICIDS-2017 it reached 99.2 percent accuracy, 98.9 percent precision, 98.7 percent recall, and a 98.8 percent F1-score, with a false-positive rate of just 0.8 percent. On the imbalanced UNSW-NB15 dataset it maintained 94.5 percent accuracy, 94.2 percent recall, and a 93.8 percent F1-score, although its false-negative rate of 5.8 percent was the highest of the three datasets, plausibly reflecting sparse minority attack categories, though the binary evaluation could not verify which categories were involved. All experiments were independently repeated five times and averaged. Against reimplemented baselines including a DNN-LSTM-GRU hybrid, a particle-swarm-optimized approach, a deep reinforcement learning DQN method, and conventional machine learning ensembles, the framework achieved higher within-dataset performance across the evaluated benchmarks while keeping computational overhead low.

Deployment efficiency was measured under deliberately constrained conditions. All experiments ran on an Intel Core i7-10700 processor with 16 gigabytes of RAM and no GPU acceleration, emulating the resource constraints typical of enterprise WAN controllers. The model contains roughly 260,000 to 1.21 million trainable parameters depending on input dimensionality, far smaller than the 5 to 50 million parameters common in published transformer-based or deep recurrent IDS architectures, though those were not reimplemented here. The measured amortized per-flow model-inference latency was 2.3 milliseconds at batch size 64, implying an idealized processing rate of about 435 flows per second. The author is careful to note that this figure excludes traffic acquisition, feature extraction, and window construction, so it is not end-to-end detection latency, and real deployments would face OS scheduling and concurrency overhead that reduce effective throughput. Classical machine learning models were faster, at roughly 1.5 milliseconds per flow, but scored considerably lower on detection, while the optimization-based PSO-D-SEM approach incurred about 4 milliseconds.

The cross-dataset experiments probe how much of what the model learns is transferable. Using 22 manually aligned flow-level features shared between CICIDS-2017 and UNSW-NB15, covering flow duration, protocol, byte and packet counts, timing statistics, and inter-arrival measures, the model trained on CICIDS-2017 achieved 83.5 percent accuracy and an 81.2 percent F1-score when evaluated on UNSW-NB15 without retraining, and 82.1 percent accuracy with 79.8 percent F1 in the reverse direction. Relative to the source models’ within-dataset performance on the same 22 features, accuracy dropped by roughly 12 to 15 percentage points, evidence of partial but real cross-dataset generalization. A constrained transferability probe on CERNET, using only 10 features common to all three datasets because of the fundamental mismatch between CICFlowMeter-style and NetFlow/IPFIX-style representations, yielded 71.3 percent accuracy when trained on CICIDS-2017 and 70.1 percent when trained on UNSW-NB15. The roughly 27-point drop relative to the native-feature CERNET result mixes reduced feature overlap with domain shift, and no within-dataset reference on the 10-feature subset was run to separate the two effects.

An ablation study, run only on CERNET, quantified each component’s contribution. Combining CNN and LSTM stages improved performance substantially over either architecture alone; the LSTM-only variant was actually slower per flow than the CNN-only variant despite fewer parameters, because recurrent computation resists parallelization on CPU hardware. Hybrid resampling combining SMOTE oversampling with random undersampling added about 1.2 points of accuracy and 1.5 points of F1, and channel attention added a further 1.7 points of accuracy and 1.5 points of F1 at negligible latency cost. The study also acknowledges methodological caveats with unusual transparency. Because SMOTE was applied before sliding-window construction, training windows containing synthetic flows are not guaranteed to follow capture-time order, so part of what the model learns from such windows may not reflect real traffic dynamics; no SMOTE-free sensitivity check was performed. Feature selection and normalization were likewise applied before the train-test split, a potential source of information leakage whose quantitative impact was not evaluated. And because all evaluation is binary, fine-grained attack-family information needed for threat attribution and forensics is lost.

The study’s conclusions are correspondingly measured. The comparisons support conclusions only against the baselines evaluated; no Transformer or temporal convolutional network baseline was included, and no superiority over current methods is claimed. The datasets, while collectively capturing encrypted, heterogeneous, and imbalanced traffic characteristics, do not fully represent operational SD-WAN conditions, and CERNET-2020 is not publicly accessible through a centralized repository, which limits independent reproduction. Latency was measured offline on test-set batches without live traffic load, and no percentile-based tail-latency statistics such as p95 or p99 were recorded. Future work identified by the author includes multi-class evaluation using the original attack taxonomies, time-based and host-based evaluation protocols to assess concept drift, systematic benchmarking against contemporary architectures under identical CPU-only constraints, imbalance-aware training strategies such as focal loss that preserve temporal coherence, domain adaptation and automated feature mapping for heterogeneous monitoring platforms, continual learning mechanisms for evolving attacks, and explainable AI techniques to make detection decisions transparent. Within those limits, the framework stands as a candidate for further enterprise WAN evaluation.

Subject of Research: Artificial intelligence, network security, deep learning, intrusion detection

Article Title: A deployment-efficient CNN-LSTM framework for real-time malicious traffic detection across heterogeneous, imbalanced, and encrypted network traffic

Article References: Ghavidel, M. (2026). A deployment-efficient CNN-LSTM framework for real-time malicious traffic detection across heterogeneous, imbalanced, and encrypted network traffic. Discover Artificial Intelligence, 6(1), Article 1622. https://doi.org/10.1007/s44163-026-02550-z

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02550-z

Keywords: deep learning, network security, intrusion detection, CNN-LSTM, encrypted traffic, SD-WAN, malicious traffic detection, channel attention, deployment-efficient, framework, real-time, malicious