文献类型：期刊文献

中文题名：基于自适应动量更新策略的Adams算法

英文题名：Adams algorithm based on adaptive momentum update strategy

作者：李满园[1];罗飞[1];顾春华[1];罗勇军[1];丁炜超[1]

机构：[1]华东理工大学信息科学与工程学院,上海200237

年份：2023

卷号：45

期号：2

起止页码：112

中文期刊名：上海理工大学学报

外文期刊名：Journal of University of Shanghai For Science and Technology

收录：Scopus;北大核心:【北大核心2020】；CSCD:【CSCD_E2023_2024】；

基金：国家自然科学基金资助项目(62276097);上海市自然科学基金资助项目(22ZR1416500);上海市青年科技英才扬帆计划(20YF1410900);上海市“科技创新行动计划”(21002411000)。

语种：中文

中文关键词：优化算法;自适应动量更新策略;一阶动量;二阶动量

外文关键词：optimization algorithm;adaptive momentum update strategy;first-order momentum;second-order momentum

摘要：Adam算法是目前最常用的优化算法之一,但其面临学习率震荡导致模型不收敛问题,其改进算法AMSGrad也存在梯度递减导致的二阶动量失效问题。针对上述问题,提出了基于自适应动量更新策略的Adams算法。首先,通过为一阶动量和二阶动量引入自适应更新参数,并在最后的参数更新期间采用较小的一阶动量更新参数,构建了一种自适应的动量更新策略。其次,基于该更新策略,提出了一种能够快速收敛的Adams算法。最后,通过理论分析证明了Adams算法的收敛性。基于文本分类和图像分类的对比实验表明,相比于Adam和AMSGrad算法,Adams收敛速度更快、训练结果更好,且具有优秀的泛化能力;消融实验证明了Adams算法自适应动量更新策略的有效性。
Adam is one of the most commonly used optimization algorithms at present,but it is faced with the problem that the model does not converge due to the vibration of learning rate,and its improved algorithm AMSGrad also has the problem of invalid second-order momentum caused by gradient decline.To solve the above problems,Adams optimization algorithms based on adaptive momentum update strategy was proposed.Firstly,an adaptive momentum update strategy was constructed by introducing adaptive update parameters for the first order momentum and the second order momentum,and a smaller first order momentum update parameters was used during the final parameter update phase.Secondly,based on this update strategy,a fast convergence algorithm named Adams was proposed.Finally,the convergence of Adams algorithm was analyzed theoretically.The comparative experiments based on text classification and image classification show that Adams has faster convergence speed,better training results,and more excellent generalization ability than Adam and AMSGrad.The ablation experiment proves the effectiveness of Adams adaptive momentum updating strategy.