首页    期刊浏览 2025年07月12日 星期六
登录注册

文章基本信息

  • 标题:An Efficient Replicated System for the Metadata of HDFS
  • 本地全文:下载
  • 作者:Zhanye Wang ; Tao Xu ; Dongsheng Wang
  • 期刊名称:International Journal of Grid and Distributed Computing
  • 印刷版ISSN:2005-4262
  • 出版年度:2016
  • 卷号:9
  • 期号:5
  • 页码:175-190
  • DOI:10.14257/ijgdc.2016.9.5.16
  • 出版社:SERSC
  • 摘要:Hadoop HDFS is an open source project from Apache Software Foundation for scalable, distributed computing and data storage. HDFS has become a critical component in today's cloud computing environment and a wide range of applications built on top of it. However, the initial design of HDFS has introduced a single-point-of-failure, since HDFS contains only one active namenode, if this namenode experiences software or hardware failures, the whole HDFS cluster is unusable, this is a reason why people are reluctant to deploy HDFS for an application whose requirement is high availability. In this paper, we present a solution to enable the high availability for HDFS's namenode through efficient metadata replication. Our solution has 3 major advantages than existing ones: We utilize multiple active namenodes, instead of one, to build a cluster to serve requests of metadata simultaneously; We implement a pub/sub system to handle the metadata replication process across these active namonodes efficiently; We also propose a novel replication algorithm to deal with the network delay when the namonodes are deployed in different areas. Based on the solution we build a prototype called NCluster and integrate it with HDFS. We evaluate NCluster to exhibit its feasibility and effectiveness. The experimental results show that our solution performs well with low replication cost, good throughput and scalability.
  • 关键词:HDFS; namenode; metadata; availability; replication; NCluster
国家哲学社会科学文献中心版权所有