文献类型：期刊文献

英文题名：Doctor Sun: a bilingual multimodal large language model for biomedical AI

作者：Xue, Dong[1];Shao, Ziyao[1];Duan, Zhaoyang[1];Liu, Fangzhou[2];Li, Bing[3];Zhang, Zhongheng[3,4]

机构：[1]East China Univ Sci & Technol, Key Lab Smart Mfg Energy Chem Proc, Minist Educ, Shanghai, Peoples R China;[2]Harbin Inst Technol, Res Inst Intelligent Control & Syst, Harbin, Peoples R China;[3]Sir Run Run Shaw Hosp, Hangzhou, Zhejiang, Peoples R China;[4]Shaoxing Univ, Shaoxing, Peoples R China

年份：2026

卷号：12

起止页码：1

外文期刊名：PEERJ COMPUTER SCIENCE

收录：;EI(收录号：20261920669853);WOS:【SCI-EXPANDED(收录号:WOS:001778786100001)】；

基金：This work was funded by the National Natural Science Foundation of China (Grant No. 62173147, 62373123, 82472243 and 82272180), the China National Key Research and Development Program (No. 2023YFC3603104), the Huadong Medicine Joint Funds of the Zhejiang Provincial Natural Science Foundation of China (No. LHDMD24H150001), the China National Key Research and Development Program (No. 2022YFC2504500), a collaborative scientific project co-established by the Science and Technology Department of the National Administration of Traditional Chinese Medicine and the Zhejiang Provincial Administration of Traditional Chinese Medicine (No. GZY-ZJ-KJ-24082), the General Health Science and Technology Program of Zhejiang Province (No. 2024KY1099), the Project of Zhejiang University Longquan Innovation Center (No. ZJDXLQCXZCJBGS2024016), and Wu Jieping Medical Foundation Special Research Grant (320.6750.2024-23-07). There was no additional external funding received for this study. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

语种：英文

外文关键词：Multimodal machine learning; Large language model; Medical diagnosis

摘要：While biomedical artificial intelligence (AI) shows great clinical potential, existing systems relying on general-domain large language models (LLMs) often lack sufficient medical data to grasp complex healthcare concepts. Furthermore, recent Large Language and Vision Assistant (LLaVA)-based medical multimodal large language models (MLLMs) still struggle to effectively capture the intricate alignments between medical images and text. Therefore, we introduce Doctor Sun, an MLLM specialized in medicine, developed to encode, integrate, and interpret diverse biomedical data modalities such as text and images. In particular, Doctor Sun integrates a pre-trained vision encoder with a medical LLM and conducts two-stage training on various medical datasets, focusing on feature alignment and instruction tuning. Moreover, we release SunMed-VL, a wide-range bilingual medical multimodal dataset, along with all associated models, code, and resources, to freely support the advancement of biomedical multimodal research.