文献类型：会议论文

英文题名：A Novel Approach for Efficient Accessing of Small Files in HDFS: TLB-MapFile

作者：Meng Bing[1];Guo Wei-bin[1];Fan Gui-sheng[1];Qian Neng-wu[1]

机构：[1]East China Univ Sci & Technol, Sch Informat Sci & Engn, Shanghai, Peoples R China

会议论文集：17th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD)

会议日期：MAY 30-JUN 01, 2016

会议地点：Shanghai, PEOPLES R CHINA

语种：英文

外文关键词：HDFS; TLB; Rapid retrieval; the storage of small files; prefetching

摘要：Hadoop distributed file system (HDFS) was originally designed for streaming access large files, but the access and storage efficiency is low for the mass small files. This paper presents an access optimization approach for HDFS small file based on MapFile: TLB-MapFile. TLB-MapFile merges massive small files into large files by MapFile mechanism to reduce NameNode memory consumption and add fast table structure (TLB) in DataNode, and to improve retrieval efficiency of small files. First, according to MapFile mechanism, small files are merged into large files and stored in HDFS. Second, the access frequency and the ordered queue of small files (per unit time) can be obtained through accessing system audit logs in HDFS, and the mapping information between block and small files are stored in the TLB table with regularly being updated. TLB-MapFile improves access efficiency of small files through the prefetching of priori strategies based on TLB table. Experiment results show that this method can effectively reduce NameNode memory consumption and improve the reading speed of small files.