DETERMINATION OF RESOURCE USAGE CHARACTERISTICS FOR HADOOP MAP REDUCE TASKS

Hadoop is a common frame work used to process large amounts of data. It uses map reduce framework to divide the data and process it parallel on multiple nodes. Different jobs have different resource usages of CPU and IO and similarly different nodes have different loads. If resource usage of jobs and resource availability of nodes are considered in the decision of scheduling of multiple map and reduce tasks of different jobs, an optimized execution time can be obtained. It is more useful in could environment as map/reduce tasks execute on virtual machines in spite of physical machines. As parts of research conducted to build a dynamic scheduler for map reduce applications considering job and VM characteristics, this paper proposes a technique to study the job characteristic in terms of CPU and IO of usage.