文献类型：期刊文献

中文题名：HDFS中海量小文件合并与预取优化方法的研究

英文题名：Research on Optimization Method of Merging and Prefetching for Massive Small Files in HDFS

作者：郑通[1];郭卫斌[1];范贵生[1]

机构：[1]华东理工大学信息科学与工程学院,上海200237

年份：2017

卷号：44

期号：B11

起止页码：516

中文期刊名：计算机科学

外文期刊名：Computer Science

收录：CSTPCD;;北大核心:【北大核心2014】；CSCD:【CSCD_E2017_2018】；

基金：国家自然科学基金资助项目(61672227)资助

语种：中文

中文关键词：HDFS;海量文件;合并;映射;LRU;预取机制

外文关键词：HDFS;Massive files;Merging;Mapping;LRU;Prefetch mechanism

摘要：HDFS在存储海量文件时具有明显的优势,但在存储小文件占绝大多数的海量文件时,HDFS单个NameNode的存储架构会导致其性能严重降低。为此,提出一种基于合并思想的方案,即将小文件合并为大文件,同时建立小文件到合并文件的映射关系,并将其存于HBase中。为了提高读取速度,建立了基于LRU的预取机制。实验表明,该方法能明显提高HDFS在处理海量文件时的整体性能。
HDFS has a significant advantage on storing the massive files,however,its storage architecture which has only one NameNode will result in the decrease of performance when HDFS is used to store massive files which is mainly composed by small files.A solution based on the idea of that small files were merged into large files was proposed.Meanwhile,the mapping relationship from small files to merging files was established and stored into HBase.Finally,we provided a LRU based prefetching mechanism to improve the reading speed.The experiments show that the proposed method can improve the overall performance of HDFS with the large amounts of small files.