Averaging is probably not the optimum way of aggregating parameters in federated learning

Peng Xiao, Samuel Cheng, Vladimir Stankovic, Dejan Vukobratovic

Research output: Contribution to journalArticle

1 Citation (Scopus)
3 Downloads (Pure)

Abstract

Federated learning is a decentralized topology of deep learning, that trains a shared model through data distributed among each client (like mobile phones, wearable devices), in order to ensure data privacy by avoiding raw data exposed in data center (server). After each client computes a new model parameter by stochastic gradient descent (SGD) based on their own local data, these locally-computed parameters will be aggregated to generate an updated global model. Many current state-of-the-art studies aggregate different client-computed parameters by averaging them}, but {none theoretically explains} why averaging parameters is a good approach. In this paper, we treat each client computed parameter as a random vector because of the stochastic properties of SGD, and estimate mutual information between two client computed parameters at different training phases using two methods in two learning tasks. The results confirm the correlation between different clients and show an increasing trend of mutual information with training iteration. However, when we further compute the distance between client computed parameters, we find that parameters are getting more correlated while not getting closer. This phenomenon suggests that averaging parameters may not be the optimum way of aggregating trained parameters.
Original languageEnglish
Article number314
Number of pages11
JournalEntropy
Volume22
Issue number3
DOIs
Publication statusPublished - 11 Mar 2020

Keywords

  • federated learning
  • decentralized learning
  • averaging
  • mutual information
  • correlation

Fingerprint Dive into the research topics of 'Averaging is probably not the optimum way of aggregating parameters in federated learning'. Together they form a unique fingerprint.

  • Projects

    Cite this