Smart Splitting: New System Slashes Edge AI Data Loads by 80 Percent Without Sacrificing Accuracy

Every time a smartphone recognizes a face, a network camera flags an anomaly, or a wearable device transcribes a voice command, a deep neural network must run its calculations somewhere. Traditionally, that somewhere has been a distant data center, forcing raw data to travel over wireless networks, adding latency, burning bandwidth, and raising privacy concerns. A research team spanning Virginia Tech, Queen’s University Belfast, and University College Dublin now proposes a different path: an inference serving system called SWEET that dynamically tailors both the neural network itself and the split of computation between device and server, squeezing more than 80 percent out of the communication payload while keeping accuracy losses below one percent.

The core insight behind SWEET, described in the journal Frontiers in Complex Systems, is that no single fixed model configuration suits every edge device, every wireless channel, and every application’s accuracy needs. A smartwatch with a modest processor, a network camera with a dedicated accelerator, and augmented reality glasses each present different compute capacities. Channel conditions fluctuate with signal-to-noise ratio, which per Shannon’s capacity formula determines how much data can flow and how quickly. And the tolerance for accuracy degradation varies wildly: medical diagnosis and autonomous driving demand near-perfect predictions, while recommendation engines and social media filters can absorb small errors in exchange for speed and energy savings.

SWEET responds to each inference request by solving a joint optimization problem. When an application on an edge device needs a neural network inference, it sends the server a query specifying the model, the maximum acceptable accuracy degradation, and the device’s available computation. The server then determines two things simultaneously: the layer-wise quantization bitwidths for the model and the partition point at which the network should be split. The first segment of the quantized model is transmitted to the device for local inference; the intermediate output travels back to the server, which completes the remaining computation and returns the final result.

Quantization, the practice of representing a model’s weights and activations with fewer bits than full floating-point precision, is central to the approach. Rather than applying a uniform bitwidth across all layers, SWEET assigns each layer its own precision, chosen from a closed-form solution derived from the optimization. The key theoretical tool is an accuracy degradation metric, adapted from prior work on adaptive quantization, that models how quantization noise at each layer propagates to the final output. Because this metric is linear and additive across layers, and because layer-wise contributions can be treated as independent, the system can compute a per-layer bitwidth that collectively satisfies the accuracy budget while minimizing time, energy, and cost.

To make this tractable in real time, the authors split the computation into two stages. An offline quantization algorithm precomputes optimized bitwidth patterns for every possible partition point and several accuracy tiers, storing them ahead of time. An online serving algorithm then periodically selects the best pattern for current conditions, balancing algorithmic overhead against adaptability to changing workloads and channel states. This two-stage design avoids the expensive forward passes that would otherwise be required to compute adversarial noise thresholds on the fly, making real-time deployment practical.

The system models the full cost of an inference request: local computation time based on the device’s clock rate and the number of multiply-accumulate operations, energy consumption derived from the relationship between processor voltage and frequency, transmission latency governed by channel capacity and signal-to-noise ratio, and the monetary cost of renting server computation. Users can weight these objectives differently—a battery-sensitive device might prioritize energy, while a latency-critical application might emphasize speed—by adjusting coefficients in the objective function.

Simulation results demonstrate the approach’s effectiveness across multiple dimensions. On a controlled six-layer fully connected network, SWEET achieved the lowest overall cost among four compared schemes, including autoencoder-based compression and model pruning baselines, both of which were tuned to match SWEET’s accuracy budget. Layer-wise parameter reductions ranged from 62 to 84 percent, averaging 77 percent, with accuracy degradation held below one percent. Across more demanding benchmarks—SVHN, CIFAR-10, CIFAR-100, and ImageNet with ResNet-18 and ResNet-34 architectures—communication payload compression ratios ranged from roughly 12 to 18 percent of the original model size, with accuracy losses between 0.08 and 0.66 percent.

Real-world trace-driven experiments using Twitter activity data as a proxy for bursty inference workloads showed that SWEET maintains stable latency under fluctuating demand, outperforming a model-switching baseline that must choose among pre-trained variants without adapting the model itself. On a physical testbed pairing a Raspberry Pi 5 with an Nvidia L40S GPU server, SWEET consistently achieved the lowest average latency across four network architectures and varying levels of concurrent devices. At low concurrency, the system routes all computation to the server; as server queues grow, it progressively shifts more layers to the edge, avoiding the unbounded latency growth that plagues purely server-side strategies.

The authors position SWEET as the first inference serving system to optimize layer-wise quantization bitwidth using a theoretical accuracy degradation measurement, rather than relying on heuristic tuning or manual model variants. Because it operates on already-trained models through post-training quantization, no retraining or fine-tuning is needed, making the approach compatible with existing deployed networks. Future work, the team notes, will explore hierarchical global scheduling across multiple devices and servers, extending the framework toward comprehensive resource management in mobile edge computing systems.

Subject of Research: Adaptive neural network quantization and partitioning for efficient edge inference serving

Article Title: SWEET: serving workload-balanced end-to-end efficient and tailored edge inference via quantization and partitioning

Article References: SWEET: serving workload-balanced end-to-end efficient and tailored edge inference via quantization and partitioning. (n.d.). https://doi.org/10.3389/fcpxs.2026.1801157

Image Credits: AI Generated

DOI: 10.3389/fcpxs.2026.1801157

Keywords: edge computing, model quantization, inference serving, neural network partitioning, deep learning, mobile edge computing, accuracy degradation, latency optimization, energy efficiency, wireless communication, post-training quantization, distributed inference