Comparing Flow-Based Models for Network Anomaly Detection under Extreme Class Imbalance

Researchers at the Technical University of Košice have shown that even the most impressive-looking scores in machine-learning-based network intrusion detection can hide serious weaknesses in spotting rare attacks, particularly stealthy infiltrations and web-based exploits. In a study published in Mobile Networks and Applications, Eva Chovancová, Dušan Čatloch, and Martin Chovanec compared seven supervised classifiers on the CSE-CIC-IDS2018 benchmark using a demanding seven-class label space, and found that headline accuracy figures near 98 percent coexist with F1-scores as low as 0.21 for the rarest attack category. The work delivers both a carefully documented benchmark comparison and a sobering reminder that benchmark performance should not be confused with deployment readiness.

The team’s starting point was a problem that has dogged the intrusion-detection field for years: flow-based detectors can analyse traffic even when packet payloads are encrypted or inaccessible, but their aggregate performance metrics can conceal catastrophic failures on uncommon attack types. Flow records summarise bidirectional communication between hosts using timing, volume, rate, packet-size, directional, and TCP-flag features, making them suitable for modern encrypted networks. Yet they omit host history, payload content, and sequence context, precisely the kinds of evidence that might distinguish a slow, quiet infiltration from ordinary service use. Prior survey work on CSE-CIC-IDS2018 had already flagged reproducibility concerns, including severe class imbalance, suspiciously high reported scores, and incomplete descriptions of data-cleaning procedures.

To address these gaps, the researchers built a protocol designed to make every analytical choice explicit. They used all ten days of the CSE-CIC-IDS2018 dataset, which contains emulated enterprise traffic captured on Amazon Web Services and converted by the CICFlowMeter tool into bidirectional flow records. Rather than collapsing everything into a binary attack-versus-benign problem, as many earlier studies have done, they consolidated the attack scenarios into seven operationally meaningful classes: Benign, Bot, Brute Force, distributed denial of service (DDoS), denial of service (DoS), Infiltration, and Web Attack. This keeps related tool variants together while preserving a much harder task than binary classification, one in which Web Attack flows are extremely sparse while benign and denial-of-service traffic dominates.

The preprocessing pipeline was equally deliberate. The data were cleaned numerically and split in a stratified 70/15/15 ratio, with 70 percent for training, 15 percent for validation, and 15 percent held out as a test partition of 2,434,942 flows. Low-variance feature selection retained a fixed 65-feature schema, and a sensitivity check confirmed that restricting selector fitting to the training partition did not change the retained features or the reported metrics. Crucially, identifiers such as Flow ID, source and destination IP addresses, and timestamps were excluded because they could allow models to memorise specific hosts, services, or capture schedules. Source port, destination port, and protocol remained, a defensible choice for this input schema but one the authors acknowledge could still encode benchmark-specific shortcuts. Borderline-SMOTE, an oversampling technique that synthesises new examples near the difficult decision boundaries of minority classes, was applied to selected training classes only, so validation and test flows kept their original, natural class distributions.

Seven model families faced off under this fixed protocol: Gaussian naive Bayes, logistic regression, random forest, XGBoost, LightGBM, CatBoost, and a multilayer perceptron, all trained with fixed configurations on an eight-core 2.3-GHz Intel Core i9 with 16 GB of RAM and no GPU. The evaluation combined macro precision, recall, and F1-score, which weight every class equally, with accuracy, weighted F1-score, and the multiclass Matthews correlation coefficient (MCC), which remains informative under imbalance because it considers the full confusion matrix. To quantify uncertainty, the researchers computed 95 percent percentile bootstrap intervals from 10,000 stratified confusion-matrix resamples, an approach that measures sampling variability within the fixed test partition without claiming that the ranking would hold under different groupings, days, or organisations.

The results told a story of two very different scorecards. Random forest achieved the highest macro F1-score at 0.7841, with a 95 percent bootstrap interval of [0.7785, 0.7898], while LightGBM came second on that measure at 0.7683 but led on weighted F1 (0.9825), accuracy (0.9851), and MCC (with random forest’s multiclass MCC at 0.9472). The gap between macro and weighted measures is the study’s central finding: for random forest, macro F1 trails weighted F1 by 0.1974, and for LightGBM by 0.2142. With 83.07 percent of test flows being benign, accuracy essentially measures how well a model recognises normal traffic. When classes count equally, the picture darkens dramatically.

Per-class analysis of the leading random forest made this concrete. Benign, Bot, and DDoS classes all exceeded 0.99 F1, and Brute Force and DoS remained strong at 0.8900 and 0.9295 respectively. But Infiltration reached only 0.1346 recall and 0.2106 F1, with the interval [0.2045, 0.2166], while Web Attack combined a recall of 0.8345 with a precision of just 0.3231, yielding an F1 of 0.4659 with a wide interval of [0.4272, 0.5052] reflecting only 139 Web Attack flows in the test partition. The pattern repeated across model families: XGBoost managed only 0.1893 F1 on Infiltration, LightGBM 0.2146, CatBoost 0.1720, and the multilayer perceptron 0.1959, typically through confusion with benign traffic. The overlapping bootstrap intervals across the stronger tree ensembles mean random forest cannot be considered definitively superior for that class.

The two failing classes fail for opposite reasons, and the authors argue this distinction matters for how defenders respond. Infiltration is difficult because individual flow summaries can resemble ordinary service use when the surrounding host sequence is absent; the remedy may lie in temporal or host-level context rather than better flow classifiers. Web Attack presents the reverse problem: models flag many flows, but too many benign ones get promoted to analyst review, so the remedy is calibrated false-positive control or second-stage verification rather than raw sensitivity. Because these errors are not interchangeable, the authors contend that a single aggregate score cannot determine the deployment response. They also caution that the near-perfect DDoS and Bot scores may reflect separable, scenario-specific signatures shared across the random split rather than genuine resilience to new networks or shifting traffic.

The study went one step further than most benchmark papers by building a proof-of-concept inference pipeline that exercises the saved feature contract outside the training CSV files. The prototype captures rotating packet traces with tcpdump, sends completed captures through CICFlowMeter, aligns the extracted flow columns with the training-time 65-feature schema, applies the saved scaler where required, and reports predicted labels with confidence values. In a demonstration run over nine PCAP files and 66 unlabeled flows processed by a random-forest model, only 27 of the 65 expected features matched and 38 had to be zero-filled; all 66 flows were predicted benign. The authors are candid that this exposes a feature-alignment limitation and establishes nothing about field detection accuracy, since the operational flows carried no ground-truth labels.

The researchers are equally forthright about the limits of their benchmark conclusions. A single dataset with a random stratified split cannot establish transfer across organisations, sensors, unseen days, or attack campaigns, and related flows from the same capture period or scenario may straddle partitions, letting models exploit scenario regularities rather than learning transferable behaviour. The closed-set classifier cannot identify unknown attacks at all, and the study lacks duplicate-flow sensitivity analysis, imbalance ablations, metadata-removal sensitivity tests, calibration analysis, and operational benchmarking. Under the fixed protocol, however, the reported ranking and minority-class weaknesses are fully reproducible, with the retained feature schema, model settings, and software environment documented for independent verification. The authors call for future work on grouped or temporal validation, novelty detection with abstention mechanisms, calibrated thresholds, and models that incorporate temporal host context. For now, their message to the security community is clear: before trusting a detector that reports 98 percent accuracy, ask what it is doing with the attacks you rarely see.