Binary protocols, which encode data directly as compact byte sequences rather than human-readable text, form the backbone of much of the modern digital world. They underpin network devices, industrial control systems, and Internet of Things applications, where efficiency and low overhead are paramount. But their very compactness creates a security blind spot. Unlike text protocols, binary formats lack delimiters, tags, and other redundant markers that make field boundaries obvious. When implementations of these protocols mishandle malformed messages, the consequences can be severe. The researchers point to the infamous EternalBlue vulnerability in the SMB protocol, which arose from improper parsing of binary messages and enabled buffer overflow and remote code execution with global impact, as a stark reminder of what is at stake.

Mutation-based greybox fuzzing has become one of the most widely used techniques for discovering such flaws. Greybox fuzzers occupy a middle ground between blackbox and whitebox approaches: they do not require complete knowledge of a program’s internals, but they do use lightweight runtime feedback, such as coverage information gathered through instrumentation, to guide how inputs are mutated. Tools like AFL, AFL++, and MOPT generate malformed test cases by applying mutation operators such as bit flips and byte substitutions to well-formed seed messages, then monitor the target program for crashes, hangs, and other abnormal behavior. The approach has uncovered countless vulnerabilities, yet the authors argue that existing fuzzers stumble when confronted with structured binary messages.

The team identifies three core challenges. First, protocol parsing is difficult because binary protocols represent data as bit streams with unclear field boundaries and implicit semantics, making it laborious to determine which positions in a message can safely be mutated. Second, evaluating the mutation value of different fields is hard because the effect of changing a field depends on runtime behavior, field dependencies, and the target program’s responses, none of which can be reliably determined statically. Third, and perhaps most damaging, existing fuzzers schedule mutation operators blindly. A bit flip applied to a function code field may drive a program into entirely new logical states, while the same operation on a data field merely alters content. Worse, mutating a length field without adjusting the corresponding payload breaks the message structure, causing the test case to be discarded before it ever reaches deep parsing logic.

MPAuzz tackles these problems in two stages. The first is a feedback-based mutation position exploration module that partitions protocol messages at bit-level granularity rather than the coarser byte level used by most prior tools. This fine granularity matters: in an MQTT message, for example, the high four bits of the header flags field determine the message type while the low four bits serve as flags, a distinction that byte-level analysis cannot capture. The module flips one bit at a time, sends the mutated message to the target, and classifies the result. Normal responses mark a bit as mutable; format anomalies detected with protocol parsing tools such as Tshark mark it as restricted; and mutations of essential control fields such as protocol names and function codes mark regions as immutable. Adjacent bits with similar properties are then merged into continuous regions, giving the fuzzer a map of where mutation is safe, where it must respect constraints, and where it is forbidden.

The second stage is where the ocean comes in. MPAuzz formulates mutation operator scheduling as a multidimensional optimization problem in which each dimension corresponds to the operator choice for one mutation region, and it solves this problem using the Marine Predators Algorithm, a metaheuristic inspired by how marine predators forage. The historically best operator combination plays the role of the predator, while candidate combinations act as prey. The algorithm dynamically switches between Lévy flights, long-distance jumps that enable broad exploration, and Brownian motion, small-step searches that enable fine-grained exploitation. The researchers deliberately chose MPA over alternatives such as multi-armed bandit strategies and particle swarm optimization because mutating one region of a binary message often depends on other regions; treating each region as an independent arm can produce structurally invalid test cases, while PSO’s velocity-based updates risk premature convergence when early coverage gains come from only a few regions.

The scheduling unfolds across three adaptive stages that mirror the fuzzing lifecycle. In the early, high-speed exploration stage, Lévy-distributed random vectors drive wide-ranging tests of operator combinations to avoid premature convergence. In the middle, balanced coordination stage, the population splits: half fine-tunes the elite combination using Brownian perturbations governed by a quadratically decreasing convergence factor, while the bottom-performing half continues exploring with Lévy motion. In the final, low-speed exploitation stage, an enhanced social learning term pulls candidate combinations toward the best-known strategy, changing only one or a few region assignments at a time. Crucially, a candidate combination is retained only if it satisfies region-specific constraints and improves coverage feedback. The framework also includes a repairing step for restricted regions: when a mutable payload region changes size, the associated length field is recalculated automatically, with nested dependencies repaired from the innermost region outward.

To validate the design, the team evaluated MPAuzz on four widely used binary protocol implementations covering MQTT, DTLS, DNS, and CoAP, comparing it against AFLNet and StateAFL, two of the most prominent greybox protocol fuzzers. Each fuzzer ran continuously for 24 hours per target, with each experiment repeated ten times to account for the inherent randomness of fuzzing. The results were decisive. MPAuzz’s valid-test-case ratio exceeded 97 percent on every target, compared with 83.1 percent for a multi-armed bandit variant and 79.5 percent for a PSO variant of the same tool. It reached comparable coverage 3.47 times faster than AFLNet and 2.22 times faster than StateAFL on average, with Vargha-Delaney effect sizes mostly at or above 0.85, indicating a consistent statistical advantage.

The vulnerability discovery results were equally compelling. Instrumenting targets with AddressSanitizer to capture memory-related faults, the researchers found that MPAuzz produced more crashes than both baselines on all four targets. On the Mosquitto MQTT broker, MPAuzz triggered an average of 87.8 crashes per run, compared with 3.9 for AFLNet and 43.2 for StateAFL, and exposed its first crash just 32 minutes into testing, whereas AFLNet failed to crash the target within the allotted time at all. On Tinydtls, MPAuzz found its first crash in 15 seconds. Analysis of the proof-of-concept inputs showed that the anomalies detected in Mosquitto and Libcoap correspond to real, documented vulnerabilities: the medium-severity CVE-2021-28166, a null pointer dereference triggered when an authenticated client sends a mutated SUBSCRIBE message before the server issues a PUBLISH message, and the high-severity CVE-2024-46304. These findings demonstrate that the fuzzer can surface genuine security flaws arising from abnormal protocol-state sequences, not merely superficial parsing errors.

The authors are candid about limitations. MPAuzz currently leans on external parsers such as Tshark and Scapy for protocol-format feedback, so for proprietary or undocumented protocols the precision of restricted and immutable region identification may degrade, though the fuzzer can still operate using runtime responses and coverage feedback alone. The automatic repair mechanism primarily supports length-related constraints; checksums, authentication fields, and state-dependent constraints require protocol-specific rules. The evaluation, while rigorous, covers four protocol implementations, and the team notes that additional targets and longer experiments would strengthen the evidence. Still, the work makes a persuasive case that combining protocol-aware region classification with staged, nature-inspired operator scheduling is a practical path forward for binary-protocol fuzzing, and the researchers plan to reduce reliance on external parsers and extend support for proprietary protocols with complex field dependencies in future work.

Subject of Research: A nature-inspired greybox fuzzing framework for discovering vulnerabilities in binary network protocol implementations

Article Title: Binary protocol greybox fuzzing driven by marine predators algorithm

Article References: Jiang, C., Hong, Z., Zhang, G., Li, Y., & Gu, J. (2026). Binary protocol greybox fuzzing driven by marine predators algorithm. Cybersecurity, 9(1), Article 214. https://doi.org/10.1186/s42400-026-00646-8

Image Credits: AI Generated

DOI: 10.1186/s42400-026-00646-8

Keywords: binary protocols, greybox fuzzing, Marine Predators Algorithm, vulnerability discovery, mutation scheduling, network security, MQTT, DTLS, DNS, CoAP, software testing, cybersecurity