Federated Transformer Framework Slashes Energy Prediction Communication Costs by 300x

Buildings are among the world’s most consequential consumers of energy, and predicting how much power they will use tomorrow, next week, or next winter has become one of the quiet workhorses of the global sustainability effort. Accurate forecasts let facility managers tune heating, ventilation, and air-conditioning systems, help grid operators balance load, and support the broader decarbonization agenda championed by organizations such as the International Energy Agency. Yet the machine learning models best suited to this task have run into a stubborn wall: the very data needed to train them is scattered across hundreds of buildings, owned by different organizations, and increasingly locked behind privacy regulations such as the General Data Protection Regulation. A new study published in the journal Machine Learning offers a way through that wall, and it does so without asking anyone to hand over their raw data.

Researchers Jessica Al Achy and Abdallah Makhoul of the CNRS institut FEMTO-ST at Université Marie et Louis Pasteur in France, together with Hassan Harb of the American University of the Middle East in Kuwait, have introduced a framework called SWIFT-KD, short for Sliding Window Intelligent Federated Transformer Learning with Knowledge Distillation. The system is designed to train a powerful transformer-based forecasting model across large fleets of buildings while keeping every building’s energy records on its own premises. It also tackles a problem that has quietly hampered federated learning in the real world: the sheer cost of moving enormous neural network updates between resource-constrained edge devices and a central coordination server.

The core architectural idea behind SWIFT-KD is hierarchical. Transformers, the family of models that revolutionized natural language processing and have since swept through time-series forecasting, excel at capturing long-range dependencies, but their attention mechanisms grow computationally expensive as input sequences lengthen. For a building whose hourly energy consumption spans months, feeding the entire history into a monolithic transformer is often impractical on the modest hardware installed at the edge. The researchers instead decompose long energy sequences into overlapping segments using a sliding window scheme, processing these segments with a hierarchical arrangement of transformer layers. Lower layers capture fine-grained local patterns, such as the daily rhythm of occupancy and equipment use, while higher layers aggregate segment-level representations into long-term dependencies, such as seasonal drifts in heating demand. The overlapping windows ensure that no temporal boundary severs a genuine pattern, preserving continuity across the decomposed sequence while keeping the per-device computation manageable.

The second innovation addresses communication, the bottleneck that federated learning pioneer McMahan and colleagues identified as far back as their foundational 2017 work on federated averaging. In conventional federated learning, each participating device trains locally and then transmits its full set of model weights, which can number in the millions of parameters, to a server for aggregation. On smart meters and building controllers with limited bandwidth and intermittent connectivity, this exchange becomes prohibitive when repeated over many training rounds. SWIFT-KD replaces weight transmission with federated knowledge distillation. Rather than shipping its parameters, each building sends soft predictions, the model’s output distributions on shared or public proxy data, which act as a compressed distillation of what the local model has learned. The server aggregates these distilled signals into a global model, and the global model is then distilled back to the edges. According to the study, this substitution achieves a 300-fold reduction in communication volume, a figure that transforms the feasibility of large-scale federated deployment on real building hardware.

To test the framework, the team turned to the ASHRAE Great Energy Predictor III dataset, a widely used public benchmark hosted on Kaggle that contains hourly meter readings from more than 1,000 buildings of diverse types and climates. The evaluation focused on 100 heterogeneous buildings, deliberately chosen to stress the statistical challenge that federated learning researchers call non-IID data. Energy consumption patterns differ wildly between an office tower in one climate zone and a warehouse in another, and models trained under federated averaging frequently struggle when local data distributions diverge this sharply. Heterogeneity of this kind is precisely the condition that most often degrades federated systems in practice, making it a demanding proving ground for any new approach.

The results were striking. SWIFT-KD achieved a coefficient of determination, or R-squared, of 0.9708, a root mean squared error of 92.24 kWh, and a mean absolute error of 44.58 kWh. For context, an R-squared approaching unity indicates that the model explains nearly all of the variance in building energy consumption. More remarkable is the comparison against alternatives: the federated framework outperformed standard federated averaging by 21 percent and exceeded even centralized training, in which all data would be pooled in one place, by 13.3 percent. That last figure deserves emphasis, because the conventional wisdom has long held that federated methods necessarily pay an accuracy tax for the privilege of preserving privacy. Here, the distributed approach did not merely match the centralized baseline; it beat it, suggesting that the hierarchical structure and distillation process may itself act as a useful regularizer when data is heterogeneous.

Equally important for practical deployment is the framework’s communication efficiency over the training lifecycle. SWIFT-KD reached its peak predictive performance within just two communication rounds and maintained stable accuracy throughout the remaining rounds. Convergence this rapid compounds the benefits of the 300-fold compression per round: the total network traffic required to train a production-quality model collapses to a small fraction of what weight-based federated averaging would demand. For building operators weighing whether edge intelligence is worth the operational complexity, this combination of fast convergence and lightweight exchanges substantially lowers the barrier. The framework’s design also sidesteps the need for specialized aggregation infrastructure, since distilled predictions are far smaller and easier to combine than full model snapshots.

The broader significance extends beyond the building sector. The study sits at the intersection of three active research currents: the adoption of transformer architectures for time-series forecasting, the maturation of federated learning as a privacy-preserving training paradigm, and the growing use of knowledge distillation not just to shrink models for deployment but to compress the learning process itself. Surveys of federated distillation have catalogued long-standing challenges, including how to generate or select proxy data for distillation and how to prevent the distilled signal from leaking information about local datasets. By demonstrating that distilled federated learning can decisively outperform both weight-based federated learning and centralized training on a realistic, heterogeneous benchmark, the authors provide evidence that these challenges are tractable at scale.

The implications for smart buildings and the energy transition are immediate. City-scale energy management, demand-response programs, and grid decarbonization all depend on forecasts that respect both accuracy and privacy. A framework that lets hundreds of buildings collaboratively learn a shared forecasting model, each contributing knowledge without disclosing consumption records that could reveal occupancy patterns or business activities, aligns directly with regulatory requirements and public expectations. The researchers have made their source code, hyperparameter configurations, and exact data partitions publicly available on GitHub, and the ASHRAE dataset itself is open, which should allow other teams to verify, extend, and adapt the approach. Whether SWIFT-KD or its descendants become the standard for privacy-preserving energy analytics in commercial buildings, the study makes a compelling case that the trade-off between data privacy and model performance, long treated as inevitable, can be engineered away. In a field where a single percentage point of forecasting accuracy can translate into meaningful energy savings across a real estate portfolio, a method that simultaneously improves accuracy, slashes communication costs by two orders of magnitude, and eliminates the need to centralize sensitive data is likely to draw sustained attention from both researchers and the industry it aims to serve.

Subject of Research: Privacy-preserving federated transformer learning with knowledge distillation for building energy prediction

Article Title: SWIFT-KD: Sliding Window Intelligent Federated Transformer Learning with Knowledge Distillation for Building Energy Prediction

Article References: Al Achy, J., Harb, H., & Makhoul, A. (2026). SWIFT-KD: Sliding Window Intelligent Federated Transformer Learning with Knowledge Distillation for Building Energy Prediction. Machine Learning, 115(9), Article 216. https://doi.org/10.1007/s10994-026-07153-4

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07153-4

Keywords: federated learning, knowledge distillation, transformer models, energy forecasting, smart buildings, deep learning, privacy-preserving machine learning, time series prediction, edge computing, ASHRAE dataset, communication efficiency, sustainability