Guiding Federated Learning with Low Rank Metrics

Abstract

Federated learning (FL) is a distributed machine learning method that protects the privacy of local data, and thus is used in scenarios like medical applications. However, FL algorithms are known to demonstrate deteriorated performance when the local data of each client is not independently and identically distributed (Non- IID). In this work, we explore the possibility of guiding the aggregation process in FL using low-rank based layer-wise metrics, and propose an aggregation framework that allows the use of any layer-wise metric to guide the aggregation process. specifically, a layer-wise metric is used as the weight for the weighted averaging process in the aggregation, and the metric is optionally smoothed by exponential averaging.
In this work, we propose two low-rank layer-wise metrics. However, experiments do not show signifi cant improvement of the proposed methods over the baseline method (FedAvg), thus this paper mainly serves as a record of our exploration and the experimental results.