Federated learning hides failing clients behind strong average accuracy

Federated learning promises a way to train machine learning models across many devices or institutions without moving anyone’s private data. Each client trains locally, and a server combines the updates into a shared model. Yet a new empirical study warns that the standard way of judging these systems—looking at overall accuracy—can hide serious failures affecting individual participants, and that one of the most popular fixes for the problem can make things worse exactly when the data is most uneven.

The study, published in Discover Artificial Intelligence, was carried out by Jyotiprakash Panda and Om Gajanan Patil. Rather than proposing a new algorithm, the authors set out to diagnose when and why existing federated learning methods break down at the level of individual clients. Their central message is that average accuracy alone is not enough to describe how a federated system behaves, and that worst-client accuracy deserves to be treated as a first-class evaluation metric.

The motivation comes from a familiar real-world pattern. In federated deployments, client data are rarely identical: datasets differ in size, in label distribution, and in how the data were generated. Non-IID distributions are the norm rather than the exception. The authors illustrate the stakes with two examples. In a federated keyboard prediction system serving users of different languages, a global model could reach 90 percent average accuracy while a minority-language user drops to 30 percent—a qualitatively broken product that aggregate metrics would report as a success. In a federated medical imaging system spread across hospitals, a hospital whose patient demographics are underrepresented in the training coalition could receive systematically worse diagnostic support, with no visibility from global accuracy logs.

To study this systematically, the researchers ran controlled experiments on three standard image datasets: MNIST, CIFAR-10, and CIFAR-100. They simulated statistical heterogeneity using Dirichlet-based label skew, a partitioning method in which the proportion of each class held by each client is sampled from a Dirichlet distribution, following a methodology validated against real-world federated deployments. Smaller values of the concentration parameter α produce stronger non-IID behavior. The experiments spanned a fixed grid of α values from 10.0, which is close to IID, down to 0.1, which represents extreme non-IID conditions. Each dataset was divided into ten logical clients, and every client participated synchronously in every communication round, so that statistical heterogeneity could be isolated from system-level confounders such as client dropout or partial participation.

The models were deliberately lightweight. MNIST experiments used a multilayer perceptron with a single hidden layer of 128 units, while CIFAR-10 and CIFAR-100 used a shallow convolutional neural network with two convolutional layers. Training used stochastic gradient descent with a fixed learning rate, and each client trained the received global model for only a single local epoch per communication round. This limited local training was intentional: it ensures that apparent performance differences are driven mainly by data heterogeneity rather than by optimization depth. All experiments ran on CPU-only computation on a single local system, with no specialized accelerators or distributed infrastructure.

The baseline results reveal what the authors call a quiet failure scenario. On MNIST, mean client accuracy remained largely stable as heterogeneity increased, fluctuating only between 0.683 and 0.711—a spread of less than 4 percent—across the α grid. Worst-client accuracy, however, declined dramatically as α fell. In other words, the aggregate picture stayed reassuring while the weakest clients deteriorated sharply. The authors argue this apparent adaptability is an artifact of aggregate evaluation rather than evidence of robust client-level performance, and that depending solely on aggregate accuracy can conceal serious errors affecting individual participants.

The statistical analysis strengthens the point. All results were averaged over five independent random seeds, reported as mean plus or minus standard deviation, with 95 percent confidence intervals computed for worst-client accuracy. Under near-IID conditions, overlapping confidence intervals showed no statistically significant difference between baselines. Under strong heterogeneity at α = 0.5 on CIFAR-10, all methods exhibited increased variance: the standard deviation for FedAvg rose to 0.030 and for q-FedAvg to 0.028. Notably, q-FedAvg—a fairness-oriented method that reweights client updates based on local loss values—recorded a lower worst-client accuracy than plain FedAvg under these conditions, 0.401 versus 0.438. The authors report that this failure mode was not described in the original q-FedAvg work, and they interpret it as evidence that high heterogeneity destabilizes aggregation-based methods generally, with loss-based re-weighting amplifying the effect modestly.

The study then tested two mitigation strategies. The first was server-side client re-weighting, which modifies the Federated Averaging aggregation rule so that client updates are weighted in inverse proportion to local dataset size, giving clients with fewer samples more influence. This helped in near-IID and mildly heterogeneous settings at α = 10.0 and α = 1.0, improving worst-client accuracy without hurting mean performance. But as heterogeneity rose to α = 0.5 and α = 0.1, re-weighting destabilized training: average accuracy dropped significantly and worst-client accuracy plummeted, with some clients behaving almost randomly. The authors attribute this to a fundamental flaw in server-side weighting—amplifying the contribution of clients with highly skewed local data injects noisy, misaligned updates that reduce the consistency of the global model instead of improving fairness.

The second strategy, local data balancing, works on the client side instead. Before training, each client rebalances its own class distribution through simple random duplication: minority classes are upsampled by sampling with replacement until all classes reach the count of the majority class. No synthetic samples are generated, no inter-client data is exchanged, and the aggregation rule, learning process, and communication protocol remain untouched. This lightweight preprocessing outperformed both the baseline and re-weighting across all heterogeneity levels. At α = 1.0, worst-client accuracy rose from 0.474 to 0.756 while mean accuracy climbed to 0.808. At α = 0.5, worst-client accuracy stayed above 0.74 while re-weighting collapsed. Even at extreme heterogeneity of α = 0.1, local balancing lifted worst-client accuracy from near-random performance to roughly 0.50, avoiding catastrophic failure without fully eliminating the loss. Combining re-weighting with local balancing did not help: at α = 1.0 the combined worst-client accuracy of 0.638 fell below local balancing alone at 0.668, showing that re-weighting’s destabilizing effect persists even when paired with a more stable intervention.

The findings generalised beyond the simpler datasets. On CIFAR-100, a 100-class problem with substantially higher inter-class complexity, the same CNN architecture and Dirichlet partitioning were used with the output layer extended to 100 classes. At α = 0.1, baseline worst-client accuracy dropped 34 percent relative to mean accuracy, while local balancing consistently recovered worst-client performance across all heterogeneity regimes. The authors conclude that the core pattern—aggregate metrics masking worst-client degradation under non-IID conditions—is not an artifact of dataset simplicity. They also note the limits of the work: the experiments use shallow architectures, so whether the divergence persists under deeper networks such as ResNets or Vision Transformers remains open, and the simulation excludes client dropout, communication delays, and adversarial clients. Still, the practical guidance is clear: federated systems should be evaluated with client-aware metrics, and improving the quality of local updates appears more reliable than adjusting aggregate weights after biased updates have already been produced.

Subject of Research: Empirical evaluation of federated learning under data heterogeneity, worst-client failure modes, and mitigation strategies

Article Title: Evaluating federated learning under data heterogeneity with analysis of failure modes and practical mitigations

Article References: Panda, J., & Patil, O. G. (2026). Evaluating federated learning under data heterogeneity with analysis of failure modes and practical mitigations. Discover Artificial Intelligence, 6(1), Article 1706. https://doi.org/10.1007/s44163-026-02339-0

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02339-0

Keywords: federated learning, data heterogeneity, non-IID data, worst-client accuracy, FedAvg, FedProx, q-FedAvg, client re-weighting, local data balancing, Dirichlet label skew, CIFAR-10, CIFAR-100