DynMR: A Dynamic Slot Allocation Framework for MapReduce Clusters in Big Data Management using DHSA and SEPB

MapReduce is one among the famous processing model for huge scale information (Big Data) processing in distributed computing. Since there may be a possibility of slot based MapReduce framework (eg. Hadoop MRv1) displaying some poor execution as a result of its unoptimized resource allocation. To venture on this, this paper finds and further streamlines the data distribution and resource allocation from the following three key perspectives. To begin with, because of the pre-configuration of the map slots and reduce slots which are not replaceable slots can be extremely under used. Since map slots may be completely used while reduce slots are empty and the other way around, considering the slot based model we set forth an option strategy called Dynamic Hadoop Slot Allocation. It unwinds the slot allocation parameters to permit slots to be reallocated to map or reduce task assignments relying upon their needs. Second the speculative execution can handle the straggler issue which sufficiently fit to enhance the execution for a job however to determine the expense of cluster proficiency. In context of this we further show Speculative Execution Performance Balancing so as to adjust the execution exchange between a single job and a batch of jobs. Third, delay scheduling has indicated to enhance the information and data locality at the fair cost. On the other hand we propose a method called Slot Pre Scheduling that can enhance the data locality yet with no effect on cost. At last by melding all the strategies together we make an orderly slot allocation framework called DynMR (Dynamic Map Reduce) which can enhance the execution of MapReduce workloads significantly.