Big Data : Real Challenge

In this paper, we review the background, stateof-the-art and management of big data. Big data is a large volume of structured and unstructured data which is too large to handle using traditional databases. It needs to be analyzed for determining various patterns to make better decisions. Its challenges include storage, analysis, search, transfer, visualization, querying and security of information. We give the general background of big data and related technologies, such as cloud computing, Internet of Things, Hadoop and Spark. After that we focus on Hadoop and Spark. Hadoop is an Open source Java based Framework used for processing large amount of data. It is built on simple programming model called MapReduce. Another framework discussed is Apache Spark, which is designed for faster computation. Spark is not a modified version of Hadoop and is not completely dependent on Hadoop because it has its own processing technique. The main feature of Spark is its in-memory cluster computing that increases the processing speed of an application. We discuss some applications of big data. This paper aims to provide a comprehensive overview, big-picture and the challenges faced by big data.