文献类型：期刊文献

英文题名：MNGNAS: Distilling Adaptive Combination of Multiple Searched Networks for One-Shot Neural Architecture Search

作者：Chen, Zhihua[1];Qiu, Guhao[1];Li, Ping[2,3];Zhu, Lei[4,5];Yang, Xiaokang[6];Sheng, Bin[7]

机构：[1]East China Univ Sci & Technol, Dept Comp Sci & Engn, Shanghai 200237, Peoples R China;[2]Hong Kong Polytech Univ, Dept Comp, Hong Kong, Peoples R China;[3]Hong Kong Polytech Univ, Sch Design, Hong Kong, Peoples R China;[4]Hong Kong Univ Sci & Technol Guangzhou, ROAS Thrust, Guangzhou 511400, Peoples R China;[5]Hong Kong Univ Sci & Technol, Dept Elect & Comp Engn, Hong Kong, Peoples R China;[6]Shanghai Jiao Tong Univ, MoE Key Lab Artificial Intelligence, Dept Automat, Sch Elect Informat & Elect Engn,AI Inst, Shanghai 200240, Peoples R China;[7]Shanghai Jiao Tong Univ, Sch Elect Informat & Elect Engn, Dept Comp Sci & Engn, Shanghai 200240, Peoples R China

年份：2023

卷号：45

期号：11

起止页码：13489

外文期刊名：IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

收录：;EI(收录号：20232914403664);WOS:【SCI-EXPANDED(收录号:WOS:001085050900040)】；

基金：This work was supported in part by the National Natural Science Foundation of China under Grants 62272164, 62272298, and 62077037, in part by the National Key Research and Development Program of China under Grant 2022YFC2407000, in part by the Interdisciplinary Program of Shanghai Jiao Tong University under Grants YG2023LC11 and YG2022ZD007, in part by the College-level Project Fund of Shanghai Jiao Tong University Affiliated Sixth People's Hospital under Grant ynlc201909, in part by the Medical-industrial Cross-fund of Shanghai Jiao Tong University under Grant YG2022QN089, in part by the Science and Technology on Space Intelligent Control Laboratory under Grant HTKJ2022KL502010, and in part by The Hong Kong Polytechnic University under Grants P0042740, P0030419, P0043906, and P0044520.

语种：英文

外文关键词：Computer architecture; Training; Neural networks; Search problems; Heuristic algorithms; Computational modeling; Knowledge engineering; Image recognition; knowledge distillation; multiple searched networks; neural architecture search

摘要：Recently neural architecture (NAS) search has attracted great interest in academia and industry. It remains a challenging problem due to the huge search space and computational costs. Recent studies in NAS mainly focused on the usage of weight sharing to train a SuperNet once. However, the corresponding branch of each subnetwork is not guaranteed to be fully trained. It may not only incur huge computation costs but also affect the architecture ranking in the retraining procedure. We propose a multi-teacher-guided NAS, which proposes to use the adaptive ensemble and perturbation-aware knowledge distillation algorithm in the one-shot-based NAS algorithm. The optimization method aiming to find the optimal descent directions is used to obtain adaptive coefficients for the feature maps of the combined teacher model. Besides, we propose a specific knowledge distillation process for optimal architectures and perturbed ones in each searching process to learn better feature maps for later distillation procedures. Comprehensive experiments verify our approach is flexible and effective. We show improvement in precision and search efficiency in the standard recognition dataset. We also show improvement in correlation between the accuracy of the search algorithm and true accuracy by NAS benchmark datasets.