Research report: CLeaR: An adaptive continual learning framework for regression tasks

Neural networks have a notorious weakness: when they are retrained on new data, they tend to erase what they already knew. Researchers at the University of Kassel have now built a framework designed to tackle this problem in a domain where it has been largely ignored—regression tasks, the kind of continuous numerical prediction that underpins weather-driven energy forecasting. The framework, called CLeaR for Continual Learning for Regression Tasks, is presented in a research article published on 16 July 2021 in the open-access journal AI Perspectives, and its authors report that it can keep acquiring knowledge from a changing data stream while improving its prediction accuracy.

The phenomenon the framework addresses is known as catastrophic forgetting, or catastrophic interference. It was first observed in the late 1980s by McCloskey and Cohen and independently by Ratcliff, who found that well-learned knowledge in connectionist models can be wiped out when the models learn new tasks successively. The root cause lies in how neural networks store information: weights connect every neuron in one layer to all neurons in the next, and a single weight can be important for more than one task. When a network adjusts that shared weight to learn something new, the old knowledge it encoded can be destroyed. Overcoming this forgetting problem, the authors note, is considered a major challenge in machine learning and a crucial step toward implementing something resembling real intelligence.

Continual learning algorithms have flourished in classification, where new labelled samples arrive frequently, and have proven successful in applications ranging from handwriting recognition and image classification to sequentially learning to play games in reinforcement learning settings. Common strategies fall into three broad groups: regularization approaches, rehearsal approaches that replay stored or generated samples, and dynamic architectures that reshape the network itself. Yet according to the authors, no prior research had addressed catastrophic forgetting in regression tasks, where the network must predict continuous values rather than assign discrete labels. This gap matters because regression models in real applications face constantly shifting conditions. In renewable energy forecasting, for example, weather data is non-stationary—its probability distribution changes over time, from summer to winter—and unexpected events such as extreme weather, ageing generators, or climate shifts can alter the mapping between inputs and outputs.

CLeaR combines forecasting neural networks with two kinds of memory buffers that store newly collected data from a non-stationary data stream. At each point in time, the framework calculates the error between its prediction and the corresponding true value, and compares that error to a dynamically adjustable threshold. Data whose error exceeds the threshold is labeled a novelty and stored in a finite novelty buffer; data the model already handles well is stored as familiarity in an infinite familiarity buffer. When the novelty buffer fills up, updating is triggered: the network is retrained on the novelty data using continual learning, then tested on the familiarity buffer to measure how much old knowledge was retained. The threshold is then re-estimated based on the updated network’s performance on both buffers, the buffers are emptied, and the cycle repeats.

In the specific implementation tested by the authors, the Models block contains an autoencoder—a network that compresses inputs through a bottleneck and reconstructs them—plus a fully-connected neural network that serves as the predictor. The autoencoder’s reconstruction error detects changes in the input distribution, while the predictor’s error detects changes in the relationship between inputs and outputs. For updating, the framework adopts Online-Elastic Weight Consolidation, an algorithm that penalizes changes to weights that are important for previously learned tasks. Unlike its predecessor EWC, which must store Fisher information for every past task, Online-EWC compresses the posterior of all previous tasks into a single Gaussian approximation, requiring far less storage—a practical advantage given that privacy laws might prohibit long-term storage of historical data. The authors emphasize that these components are customizable: the predictor could be replaced by an LSTM, and the framework’s parts can be flexibly selected for a specific application scenario.

The team evaluated CLeaR in two sets of experiments, measuring fitting error on training data, prediction error on test data, and a forgetting ratio that quantifies how much old knowledge is lost. The first experiment used an artificial seven-dimensional time series generated with daily and yearly periodic fluctuations, mimicking non-stationary features such as temperature that peak at noon and fall at night. The authors ran a grid search over 56 pairs of two framework hyperparameters—the novelty buffer size and the threshold factor—repeating each configuration 20 times with different network initializations, for a total of 1120 results.

The results of that first experiment were strikingly lopsided. On fitting error, 1089 of the 1120 CLeaR models—97.2 percent—performed better than the best baseline autoencoder trained conventionally, indicating that the framework accumulates knowledge continually and effectively, and that this ability is relatively robust to hyperparameter choices. Prediction error told a more nuanced story: 360 of the 1120 models, or 32.1 percent, outperformed the best baseline model, while 455 models, or 40.8 percent, performed worse than the worst baseline. The winners shared telltale settings—253 of the 360 had a threshold factor of at least 0.95, and most of those used small novelty buffers of 1000 or fewer samples, which triggered more frequent updates. On average, the outperforming models updated 10.2 times during the update phase versus 6.87 times for the underperformers. The authors conclude that timely learning of high-entropy data—unusual, information-rich samples—can effectively improve prediction accuracy for non-stationary data, to the point where CLeaR could, in their words, predict non-stationary data more accurately than a network trained with sufficient historical samples. The forgetting side carried a cost, however: 174 models, 15.5 percent, showed a forgetting ratio above 0.1, and 172 of those had small novelty buffers, a trade-off the authors attribute to the decay of Fisher information across repeated updates and flag as a key focus for further research.

The second experiment tested the framework in a real-world application: predicting hourly wind power generation at ten European wind farms over two years of measurements. The inputs were seven meteorological features—wind speeds at 10 and 100 meters, zonal and meridional wind direction at 100 meters, air pressure, air temperature, and humidity—drawn from 24-hour-ahead forecasts produced with the European Centre for Medium-Range Weather Forecasts model. The data was split into a warm-up phase of 1000 samples for pre-training, an update phase of 10000 samples simulating an infinite data stream, and an evaluation phase of 1000 previously unseen samples. Three CLeaR instances competed against a baseline network trained on all 11000 samples with dropout regularization: one that never updated, one updated by plain fine-tuning with early stopping, and one—the full CLeaR approach—updated with Online-EWC.

Instance C, the Online-EWC version, clearly came out ahead. Compared to the baseline, it reduced fitting error both in reconstructing weather inputs and in predicting power output, and among the three CLeaR instances it delivered the best power prediction results. Instance B, which relied on fine-tuning, showed a slightly lower average fitting error on the autoencoder task but a markedly higher one on power prediction, and its high forgetting ratio pointed to catastrophic forgetting: fine-tuning simply adapted the model to new situations while eroding old knowledge. An error analysis over 12000 samples from one wind farm showed that periodic changes in the data produced fluctuating error curves, and that the CLeaR instance’s errors were the least volatile—evidence, the authors argue, that updating is significant for predicting non-stationary data streams, especially when the model is pre-trained on only a limited dataset.

The authors are candid about the framework’s limitations and future directions. Exceptions such as wind turbines shutting down in storms must still be reported and processed manually, a challenge they place in the domain of active learning. They relied on mean squared error to detect novelties, which requires true values to be available, and they suggest estimating the uncertainty of new predictions—perhaps via Monte Carlo dropout—as an alternative for unsupervised settings. Hyperparameters were found by grid search, and they propose transferring relevant knowledge from similar tasks instead. They also suggest that Online-EWC could be replaced or combined with other continual learning algorithms. Their long-term vision is a modular tool, likened to LEGO toys, whose components can be added, removed, replaced, or expanded so that researchers and users can adapt the framework to their own application scenarios—from smart-grid power prediction to any setting where a model must keep learning from a world that will not stand still.

Subject of Research: A continual learning framework for regression tasks that addresses catastrophic forgetting in non-stationary data streams such as wind power forecasting

Article Title: CLeaR: An adaptive continual learning framework for regression tasks

Article References: He, Y., & Sick, B. (2021). CLeaR: An adaptive continual learning framework for regression tasks. AI Perspectives, 3(1), Article 2. https://doi.org/10.1186/s42467-021-00009-8

Image Credits: AI Generated

DOI: 10.1186/s42467-021-00009-8

Keywords: continual learning, catastrophic forgetting, neural networks, regression, machine learning, wind power forecasting, Online-EWC, non-stationary data streams, autoencoder, renewable energy, AI Perspectives, University of Kassel