文献类型：期刊文献

英文题名：Embracing Multiheterogeneity and Privacy Security Simultaneously: A Dynamic Privacy-Aware Federated Reinforcement Learning Approach

作者：Jin, Chenying[1];Feng, Xiang[1];Yu, Huiqun[1]

机构：[1]East China Univ Sci & Technol, Dept Comp Sci & Engn, Shanghai 200237, Peoples R China

年份：2025

卷号：36

期号：5

起止页码：8772

外文期刊名：IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS

收录：;EI(收录号：20251918374183);WOS:【SCI-EXPANDED(收录号:WOS:001279038300001)】；

基金：This work was supported in part by the National Natural Science Foundation of China under Grant 62276097 and Grant 62372174, in part by the Key Program of National Natural Science Foundation of China under Grant 62136003, in part by the National Key Research and Development Program of China under Grant 2020YFB1711700, in part by the Special Fund for Information Development of Shanghai Economic and Information Commission under Grant XX-XXFZ-02-20-2463, and in part by the Scientific Research Program of Shanghai Science and Technology Commission under Grant 21002411000.

语种：英文

外文关键词：Privacy; Sensitivity; Data privacy; Convergence; Training; Servers; Learning systems; Differential privacy (DP); dynamic privacy allocation and adjustment; federated reinforcement learning (FRL); multiheterogeneity

摘要：With growing demand for privacy-preserving reinforcement learning (RL) applications, federated RL (FRL) hase merged as a potential solution. However, existing FRL methods struggle with multiple sources of heterogeneity, while lacking robust privacy guarantees. In this study, we propose DPA-FedRL,the dynamic privacy-aware FRL framework, to simultaneouslymitigate both issues. First, we innovatively put forward the concept of "multiheterogeneity" and embed the environmental heterogeneity into agents' state representations. Next, to ensure privacy during model aggregation, we incorporate a differentiallyprivate mechanism in form of Gaussian noise and modify its global sensitivity, tailored to suit FRL's unique characteristics.Encouragingly, our approach dynamically allocates privacy bud-get based on heterogeneity levels, which strikes a balance between privacy and utility. From the theoretical perspective, we give rigorous convergence, privacy, and sensitivity guarantees for our proposed method. Through extensive experiments on diverse datasets, we demonstrate that DPA-FedRL surpasses state-of-the-art approaches (PPO-DP-SGD, PAvg, and QAvg) in some highly heterogeneous environments. Notably, our novel privacy attack simulations enable quantitative privacy assessment, validating that DPA-FedRL offers over 1.359xstronger protection than baselines