文献类型：期刊文献

英文题名：Ternary Compression for Communication-Efficient Federated Learning

作者：Xu, Jinjin[1,2];Du, Wenli[1,2];Jin, Yaochu[3,4];He, Wangli[1,2];Cheng, Ran[5]

机构：[1]East China Univ Sci & Technol, Minist Educ, Key Lab Adv Control & Optimizat Chem Proc, Shanghai 200237, Peoples R China;[2]Tongji Univ, Shanghai Inst Intelligent Sci & Technol, Shanghai 200092, Peoples R China;[3]Univ Surrey, Dept Comp Sci, Guildford GU2 7XH, Surrey, England;[4]Tongji Univ, Shanghai Inst Intelligent Sci & Technol, Shanghai 200092, Peoples R China;[5]Southern Univ Sci & Technol, Dept Comp Sci & Engn, Guangdong Prov Key Lab Brain Inspired Intelligen, Shenzhen 518055, Peoples R China

年份：2022

卷号：33

期号：3

起止页码：1162

外文期刊名：IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

收录：;EI(收录号：20205209677109);WOS:【SCI-EXPANDED(收录号:WOS:000766269100025)】；

基金：This work was supported in part by the National Natural Science Foundation of China under Basic Science Center Program 61988101, in part by the National Natural Science Fund for Distinguished Young Scholars under Grant 61725301, in part by the International (Regional) Cooperation and Exchange under Project 1720106008, in part by the National Natural Science Foundation of China under Major Program 61590923, and in part by the China Scholarship Council under Grant 201906745025.

语种：英文

外文关键词：Communication efficiency; deep learning; federated learning; ternary coding

摘要：Learning over massive data stored in different locations is essential in many real-world applications. However, sharing data is full of challenges due to the increasing demands of privacy and security with the growing use of smart mobile devices and Internet of thing (IoT) devices. Federated learning provides a potential solution to privacy-preserving and secure machine learning, by means of jointly training a global model without uploading data distributed on multiple devices to a central server. However, most existing work on federated learning adopts machine learning models with full-precision weights, and almost all these models contain a large number of redundant parameters that do not need to be transmitted to the server, consuming an excessive amount of communication costs. To address this issue, we propose a federated trained ternary quantization (FTTQ) algorithm, which optimizes the quantized networks on the clients through a self-learning quantization factor. Theoretical proofs of the convergence of quantization factors, unbiasedness of FTTQ, as well as a reduced weight divergence are given. On the basis of FTTQ, we propose a ternary federated averaging protocol (T-FedAvg) to reduce the upstream and downstream communication of federated learning systems. Empirical experiments are conducted to train widely used deep learning models on publicly available data sets, and our results demonstrate that the proposed T-FedAvg is effective in reducing communication costs and can even achieve slightly better performance on non-IID data in contrast to the canonical federated learning algorithms.