The heart of HEVC’s complexity lies in how it decides to chop each picture into blocks. The standard divides every frame into large coding tree units, which are then recursively split into smaller coding units in a quadtree structure. For each coding unit, the encoder must evaluate a set of prediction unit, or PU, partition modes, including symmetric square partitions of 2N by 2N and N by N, asymmetric partitions, and special modes such as skip and merge. For every candidate, the encoder runs motion estimation, motion compensation, and a rate-distortion optimization loop that weighs the number of bits spent against the distortion introduced. Because the quadtree can nest several levels deep and every node spawns multiple mode candidates, the number of decisions an encoder must make per frame explodes combinatorially. This recursive block partitioning and the associated PU mode decision are, as the authors note, the main factors behind HEVC’s heavy computational load.

The insight behind the new work is that many of these exhaustive evaluations are wasted effort. In real video content, a large fraction of coding units end up choosing the simplest option anyway: the 2N square PU partition, which treats the whole coding unit as a single prediction block. This is especially true for the skip mode, in which no residual data is coded at all, and for merge mode, in which motion information is simply copied from a neighboring block. When the encoder nevertheless tests all the non-square and asymmetric alternatives, it burns cycles on candidates that almost never win. If a smart rule could predict, early in the process, that the square mode will turn out to be optimal, the encoder could skip the expensive evaluations entirely and move on to the next coding unit.

Building such a rule is harder than it sounds, because the cost of a given mode depends on the content of the scene. A static talking head favors big square blocks and skip decisions, while a fast pan across a crowded stadium favors fine partitions with asymmetric shapes. Fixed thresholds tuned on one type of content fail badly on another. The researchers’ answer is an online threshold model, one that adapts to the sequence as it is being encoded rather than relying on constants learned offline. The model is built from statistical analysis of the rate-distortion costs of the best PU partition modes of previously encoded coding units, so it continuously tracks the cost distribution that the current sequence is actually producing.

The mechanism works as follows. As the encoder processes coding units, it records the rate-distortion cost of the winning partition mode for each one. From this running history, the scheme establishes a threshold that characterizes when a 2N square PU partition is likely to be optimal for the unit currently under evaluation. When the cost of the square mode for the current coding unit falls below that threshold, the encoder concludes that the square partition is the right answer and skips the evaluation of all non-square PU partition modes entirely. This covers the full family of square decisions, including both the merging mode and the advanced motion vector prediction, or AMVP, mode, as well as the skip mode. If the cost does not fall below the threshold, the encoder falls back on the standard exhaustive search, so the scheme never gambles more than the evidence supports.

The word online in the model’s name is doing important work. Because the threshold is derived from the costs of previously encoded units within the same encoding session, it automatically adjusts as the character of the video changes. A scene cut from a still interview to a high-motion action sequence shifts the cost statistics, and the threshold follows. This adaptivity is precisely what separates the approach from earlier fast algorithms that relied on precomputed parameters, spatial features, or offline-trained classifiers. Those methods can perform well on the content they were tuned for but degrade when the encoder meets something new. An online model, by contrast, learns from the very stream it is compressing, which makes it robust across sequences with very different motion characteristics.

The new scheme sits within a rich research tradition. Since HEVC was finalized, researchers have attacked its complexity from many angles: fast coding unit size decision algorithms using texture and spatial features, data mining approaches that train decision trees on encoding statistics, Bayesian adaptive methods that model the probability of partition outcomes, and, more recently, machine learning techniques including convolutional and LSTM neural networks that predict partition structures directly. The reference list of the paper spans more than a decade of this effort, from early intra-coding speedups through recent work on fast partitioning for HEVC-to-VVC transcoding and deep learning frameworks for coding tree unit partitioning. What distinguishes the present contribution is its focus on the inter-prediction PU mode decision stage and its deliberately lightweight, self-updating statistical formulation, which avoids the training pipelines and memory footprints that neural approaches require.

The experimental results reported by the authors show that the proposed method outperforms previous fast schemes, with negligible degradation in coding efficiency and video quality compared with the HEVC test model, known as HM, the reference software that defines the standard’s performance baseline. In the video compression community, negligible loss against HM is the gold standard for a fast algorithm: it means the encoder saves enormous computation while producing bitstreams whose size and decoded quality are essentially unchanged. Notably, the advantage held up across test sequences with different motion characteristics, which is exactly the scenario where fixed-threshold and offline-trained methods tend to falter. The authors attribute this robustness to the online nature of the threshold, which keeps the skip decisions aligned with the actual cost statistics of the content at hand.

The practical implications reach well beyond academic benchmarks. Every hour of video encoded for a streaming service consumes real energy and real money in data center CPU time, and HEVC encoding is among the most expensive operations those facilities perform. A scheme that eliminates a large share of the mode decision workload translates directly into higher throughput per server, lower latency for live encoding pipelines, and reduced power consumption, which matters both for cost and for the growing scrutiny of the streaming industry’s carbon footprint. On the other side of the pipe, the same principles ease the burden on devices that transcode or re-encode video locally, from broadcast head-ends to mobile handsets capturing and uploading footage. Because the method preserves rate-distortion performance, these gains come without the visible quality penalties that aggressive shortcut encoders often introduce.

The work also offers a lesson about where compression research is heading. As the successor standard H.266/VVC pushes complexity even higher with its larger coding tree units and richer partitioning toolbox, the demand for intelligent early-decision mechanisms only intensifies, and several of the works cited in the paper already apply similar ideas to VVC. The Pyongyang team’s online threshold model demonstrates that careful statistical modeling of the encoder’s own behavior, updated in real time, can rival far heavier machinery. It is a reminder that in video coding, as in many computational fields, the fastest path forward is often not to compute harder but to decide sooner which computations are not worth doing at all. For the billions of hours of video flowing across networks every year, that difference between computing harder and deciding sooner adds up to a very large number indeed.

Subject of Research: Fast prediction unit mode decision in HEVC video encoding using an adaptive online threshold model

Article Title: A research of online-threshold model for 2 N square PU- early detection in HEVC

Article References: Mun, K. S., Huang, J. G., & Jong, H. M. (2026). A research of online-threshold model for 2 N square PU- early detection in HEVC. Multimedia Tools and Applications, 85(9), Article 734. https://doi.org/10.1007/s11042-026-21906-2

Image Credits: AI Generated

DOI: 10.1007/s11042-026-21906-2

Keywords: HEVC, H.265, video compression, prediction unit, rate-distortion optimization, online threshold model, encoding complexity, skip mode, merge mode, AMVP, fast algorithms, Multimedia Tools and Applications