Survey on Resource Allocation in Phase-Level using MapReduce in Hadoop

MapReduce is programming tool for Hadoop cluster. While allocating resources, MapReduce has two levels Task-level and Phase-level. These levels should be used to check performance of each job. In existing system, the scheduling is focus on task level which tasks can have highly varying resource requirements during their lifetime and also its difficult to effectively utilize available resources to reduce job execution time. To address this limitation, this project proposes a PRISM (Phase and Resource Information -aware Scheduler MapReduce) which allocates a fine-grained resource at the phase-level to perform job scheduling. The job scheduling of prism is performed by the master node, which maintains a list of jobs in the system. Each node manager (slave node) periodically sends a heartbeat message to the scheduler. Upon receiving the status message from a node manager running on machine, the scheduler computes the utilization for set of candidate phases for the tasks using the jobs phase-level resource requirement. Then it select the phase with the highest utility for scheduling and update the resource utilization of the machine. This process is continued for until scheduled the phases of map and Reduce task is completed.