A SURVEY ON A NOVEL SPARK ON HADOOP YARN FRAMEWORK BASED IN-MEMORY PARALLEL PROCESSING FOR EFFECTIVE PERFORMANCE

Abstract A Novel spark is extends the popular MapReduce model to efficiently support more types of computations, including interactive queries and streaming. Nowadays speed is important in processing huge datasets, as it means the difference between exploring data interactively and waiting minutes or hours. One of the main features Spark offers for speed is the ability to run computations in-memory, but the system is also more efficient than MapReduce for complex applications running on disk. In this paper we are facilitate implementation and assure high performance of spark based algorithms in a complex cloud computing environment, for a parallel programming model is used. By incorporating RS data with Resilient Distributed Datasets (RDDs) of spark, all level parallel RS algorithms can be easily expressed with transformations and actions. And also to improve the performance Data-intensive multitasking algorithms and iteration-intensive algorithms were evaluated on Hadoop YARN framework. By supporting these workloads in the same engine, Spark makes it easy and inexpensive to combine different processing types, which is often necessary in production data analysis pipelines. In addition it reduces the management burden of maintaining separate tools. Keywords: Apache Spark, big data, Hadoop yet anotherresource negotiator (YARN), parallel processing, remote sensing (RS).