Communication-Efficient Design for Quantized Decentralized Federated Learning

Chen, Li; Liu, Wei; Chen, Yunfei; Wang, Weidong

Abstract

Decentralized federated learning (DFL) is a variant of federated learning, where edge nodes only communicate with their one-hop neighbors to learn the optimal model. However, as information exchange is restricted in a range of one-hop in DFL, inefficient information exchange leads to more communication rounds to reach the targeted training loss. This greatly reduces the communication efficiency. In this paper, we propose a new non-uniform quantization of model parameters to improve DFL convergence. Specifically, we apply the Lloyd-Max algorithm to DFL (LM-DFL) first to minimize the quantization distortion by adjusting the quantization levels adaptively. Convergence guarantee of LM-DFL is established without convex loss assumption. Based on LM-DFL, we then propose a new doubly-adaptive DFL, which jointly considers the ascending number of quantization levels to reduce the amount of communicated information in the training and adapts the quantization levels for non-uniform gradient distributions. Experiment results based on MNIST and CIFAR-10 datasets illustrate the superiority of LM-DFL with the optimal quantized distortion and show that doubly-adaptive DFL can greatly improve communication efficiency.

Citation

Chen, L., Liu, W., Chen, Y., & Wang, W. (2024). Communication-Efficient Design for Quantized Decentralized Federated Learning. IEEE Transactions on Signal Processing, 72, 1175-1188. https://doi.org/10.1109/TSP.2024.3363887

• Journal Article Type: Article
• Acceptance Date: Feb 3, 2024
• Online Publication Date: Feb 8, 2024
• Publication Date: Feb 8, 2024
• Deposit Date: Feb 6, 2024
• Publicly Available Date: Feb 15, 2024
• Journal: IEEE Transactions on Signal Processing
• Print ISSN: 1053-587X
• Electronic ISSN: 1941-0476
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• Volume: 72
• Pages: 1175-1188
• DOI: https://doi.org/10.1109/TSP.2024.3363887
• Public URL: https://durham-repository.worktribe.com/output/2226923