文献类型：期刊文献

中文题名：基于多模态模式迁移的知识图谱实体配图

英文题名：Entity Image Collection Based on Multi-Modality Pattern Transfer

作者：蒋雪瑶[1];力维辰[1];刘井平[2];李直旭[1];肖仰华[1]

机构：[1]复旦大学软件学院,上海200433;[2]华东理工大学信息科学与工程学院,上海200237

年份：2022

卷号：48

期号：8

起止页码：70

中文期刊名：计算机工程

外文期刊名：Computer Engineering

收录：CSTPCD;;北大核心:【北大核心2020】；CSCD:【CSCD_E2021_2022】；

基金：上海市科技创新行动计划(19511120400)。

语种：中文

中文关键词：多模态知识图谱;符号接地;模式迁移;链接预测;实体配图

外文关键词：multi-modality knowledge graph;symbol grounding;pattern transfer;link prediction;entity image collection

摘要：构建多模态知识图谱的核心在于为知识图谱中的实体匹配正确合适的图像。现有的实体配图方法主要将百科图谱以及图像搜索引擎作为实体候选图像的来源,但对图像数据元的应用方式比较简单,不能准确把握图像数据来源的特点,且可扩展性较差。提出一种基于多模态模式迁移的知识图谱实体配图方法,从不同类别的头部实体中抽取对应的语义模板及视觉模式迁移到同类非头部实体的图像获取过程中,其中语义模板用于构建搜索引擎检索关键词,视觉模式用于对检索结果去噪,最终为WikiData中25类共1.278×10^(5)个实体收集1.8×10^(6)幅图像。实验结果表明,与IMGpedia、VisualSem、Richpedia和MMKG这4种多模态知识图谱相比,利用该方法构建所得的知识图谱中实体对应的图像在准确性和多样性上更具优势,在下游任务链接预测中,通过引入该方法收集到的图像可使模型的预测链接准确性得到显著提升,在Hits@10的指标上取得59.74%的准确率,较对比方法提高12.7个百分点以上。
The core of constructing multi-modality knowledge graph is to ensure the correct and appropriate images match the entities in the knowledge graph.Existing entity image collection methods mainly use encyclopedias and image search engines as the source of images to serve as entity candidates;however,their application of image data elements is relatively simple in that they cannot accurately grasp the characteristics of image data sources,and their scalability is poor.Here,an entity image collection method based on multi-modality pattern transfer is proposed.The method extracts the corresponding semantic template from different types of head entities and transfers the visual mode to the image acquisition process of similar non-head entities.Semantic templates are used to build search engine search keywords,and visual modes are used to denoise the search results.Ultimately,the method collects 1.8×10^(6)images for 1.278×10^(5)entities in 25 categories of WikiData.The experimental results show that,compared with IMGpedia,VisualSem,Richpedia,and MMKG,the images corresponding to entities in the multi-modality knowledge graph constructed by the proposed method are more accurate with greater diversity.The accuracy of the link prediction in downstream task can be significantly improved by introducing the images collected by this method.In Hits@10,the accuracy of the index is 59.74%,which is at least 12.7 percentage points higher than that of the methods used for comparison.