Efficient Load Balancing With Distributed Hash Tables in Cloud

Distributed file systems are key building blocks for cloud computing applications based on the Map Reduce programming paradigm. In such file systems, nodes all at once serve computing and storage functions; a file is partitioned into a number of chunks allocated in distinct nodes so that Map Reduce tasks can be performed in parallel on the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be improved, replaced, and added in the system. This dependence is clearly incompetent in a large-scale, failure-prone environment on account of the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the attainment bottleneck and the single point of failure. Here, a fully distributed load rebalancing algorithm is presented to cope with the load imbalance problem. Our algorithm is compared against a centralized approach in a production system and a competing distributed solution presented in the literature. The simulation outputs indicate that proposal is comparable with the existing centralized approach and considerably outperforms the prior distributed algorithm in terms of load imbalance factor, migration cost, and algorithmic overhead.