Smarter Storage for Smart Grids: Upgraded CRUSH Algorithm Tames the Flood of Power-Surveillance Video

Modern power grids are quietly drowning in video. Every substation, transmission line, and distribution room now feeds a constant stream of surveillance footage into storage systems that were never designed for this volume. A single substation can generate roughly 500 gigabytes of video per day, and critical sites must retain footage for years rather than months. As smart grids expand under China’s dual-carbon strategy, researchers have turned to an unlikely hero: a storage-placement algorithm borrowed from cloud computing, rebuilt from the ground up for the peculiar rhythms of power-grid video.

A team at Nanjing Suyi Industrial Co., Ltd., writing in Frontiers in Communications and Networks, has unveiled a distributed storage optimization method that pairs an improved version of the CRUSH algorithm with a dynamic metadata scheduling system. Their experiments show the approach cuts data read-write latency by roughly a third, slashes metadata query response times, and keeps storage nodes balanced even when the network is slammed by fault-triggered traffic bursts. The work addresses a bottleneck that has grown acute as video shifts from passive monitoring to active support for intelligent inspection, fault diagnosis, emergency response, and visualized dispatching.

CRUSH, which stands for Controlled Replication Under Scalable Hashing, is the placement engine at the heart of the Ceph distributed storage system. It maps data objects to storage nodes through a scalable hash mechanism, minimizing data migration when devices are added or removed. But the original algorithm has blind spots that matter enormously in a power-grid context. It assigns static weights to nodes, ignoring their real-time load, so a flood of high-definition video can pile onto a handful of servers while others sit idle. It also treats all data equally, unable to prioritize fault footage that operators need instantly over archived patrol video that can wait.

The Chinese team’s fix operates on three fronts. First, their improved CRUSH algorithm monitors each storage node’s disk I/O, network bandwidth, and CPU utilization in real time, folding these into a dynamic weight alongside a hardware performance baseline. Crucially, the weights are smoothed and applied only when changes exceed a dead-zone threshold, with migration rates capped to prevent the system from endlessly shuffling data in response to minor fluctuations. The result is what the authors call a quasi-dynamic placement method: responsive to genuine shifts in node health, but immune to the oscillations that plague naive real-time remapping.

Second, the algorithm classifies video shards into three priority tiers. High-priority footage, such as substation fault monitoring and emergency line patrols, is mapped only to lightly loaded nodes with bandwidth above 100 megabits per second, and receives an extra replica for reliability. Medium-priority urban substation video gets standard treatment, while low-priority archived rural patrol video can drop to a single copy, saving storage. A geographic hash function, keyed to power-grid area codes rather than raw latitude and longitude, keeps data close to the substations and maintenance regions that generate it, cutting cross-regional transmission delays.

Third, metadata, the index information that tells the system where every video shard lives, gets its own overhaul. The researchers split metadata between edge nodes at substations and power supply bureaus, and cloud nodes at provincial dispatching centers. High- and medium-priority video metadata lives at the edge, close to the terminals that need it; low-priority metadata and the global index sit in the cloud. An LRU-K cache strategy, tuned with K equal to 3, balances responsiveness to sudden fault-video bursts against stable recognition of genuinely hot metadata, achieving a 91.2 percent cache hit rate in testing.

The experimental setup was deliberately realistic. The team simulated a 15-node cluster divided into three fault domains, each node with 8 terabytes of capacity, processing H.265 video at 1920 by 1080 resolution across three scenarios: substation equipment inspection, transmission line fault monitoring, and dispatching-center surveillance. They compared their method against consistent hashing, traditional CRUSH, a load-aware CRUSH variant, and an edge-cache metadata scheduler, keeping all shared parameters identical for fairness.

The numbers tell a striking story. Average read-write latency fell to 30.1 milliseconds, down from 40.2 for traditional CRUSH and 56.4 for consistent hashing. The standard deviation of node storage utilization, a measure of load imbalance, dropped to 5.7 percent, less than half the traditional algorithm’s 12.8 percent. Metadata query response time fell to 13.5 milliseconds, and storage throughput climbed to 426.5 megabytes per second, a 31.2 percent improvement over the baseline. Network bandwidth occupancy dropped from 36.9 to 28.3 percent, all while maintaining a 99.8 percent data recovery success rate after simulated node failures.

Perhaps most impressive is how the system behaves under stress. When the researchers cranked write request rates to three times normal load, simulating the simultaneous upload of fault video from multiple substations, read-write latency rose only to 47.5 milliseconds and recovery success stayed above 99.5 percent. Under node-failure tests ranging from one to four failed nodes, recovery time grew linearly rather than exponentially, reaching 98.5 seconds at the worst case compared with more than 220 seconds for consistent hashing. The migration-rate constraint proved essential here, preventing dynamic weight adjustments from triggering excessive rebalancing during exactly the moments when the grid can least afford it.

The researchers also accounted for the cost of all this intelligence. Metadata servers consumed between 18.6 and 57.9 percent CPU utilization as request intensity scaled from 0.3 to full load, with memory growing roughly linearly with cache size, a modest price for the latency gains. The team suggests future work could integrate pre-trained models to further adapt the system to complex power scenarios. For now, the method offers grid operators a practical blueprint for edge-cloud collaborative video platforms, one that treats a fault-event video stream not as just another file, but as the urgent, geographically anchored, reliability-critical data it truly is.

Subject of Research: Distributed storage optimization for power system video using an improved CRUSH algorithm and dynamic metadata scheduling

Article Title: Optimization method for distributed storage of power system video based on improved CRUSH algorithm and dynamic metadata scheduling

Article References: Wang, W., Ji, Y., Zhang, C., Zhang, X., Qiao, M., & Zhou, X. (2026). Optimization method for distributed storage of power system video based on improved CRUSH algorithm and dynamic metadata scheduling. Frontiers in Communications and Networks, 7, Article 1914399. https://doi.org/10.3389/frcmn.2026.1914399

Image Credits: AI Generated

DOI: 10.3389/frcmn.2026.1914399

Keywords: distributed storage, CRUSH algorithm, metadata scheduling, smart grid, power system video, Ceph, edge computing, load balancing, data replication, fault tolerance, video surveillance, latency optimization