Improving the Performance of Resource Allocation in Multitenancy MapReduce Using YARN

Abstract Multitenancy MapReduce has earned the importance in the massive bigdata technology. In a multitenant MapReduce environment, multiple tenants with different demands can share a variety of Hadoop computing resources (e.g., network, processor, memory, storage and data) within a single Hadoop system, while each tenant remains logically isolated. This useful MapReduce Multitenancy concept offers highly efficient, and cost-effective systems without wasting Hadoop computing resources to enterprises requiring similar environments for data processing and management. In this paper, we are proposing an improved resource allocation approach supporting multitenancy features for Apache Hadoop, a large scale distributed system commonly used for processing big data using YARN. We initially implement the Hadoop framework focusing on “yet another resource negotiator (YARN)”, which is mainly used for managing Hadoop resources, map reduce application runtime, and Hadoop user access controls in the latest version of Hadoop. We then identify the problems of YARN for supporting multitenancy and then derive the solution framework to solve these problems. Based on these requirements, we design the details of multitenant Hadoop. We also present the industrial multitenant Hadoop implementation that results to validate the bigdata access control and to evaluate the performance enhancement of multitenant Hadoop. The proposed multitenant Hadoop framework work is optimized for geographically distributed data centers considering the locations of data and users. Keywords: Access control, big data, cloud, Hadoop, multitenancy, resource management, yet another resource negotiator (YARN), geographically distributed data centers.