UNSTRUCTURED DATA ANALYSIS AND PROCESSING USING BIG DATA TOOL - HIVE AND MACHINE LEARNING ALGORITHM - LINEAR REGRESSION
Journal: JOURNAL OF COMPUTER ENGINEERING & TECHNOLOGY (JCET) (Vol.9, No. 2)Publication Date: 2018-04-25
Authors : NEHA MANGLA; PRIYA RATHOD;
Page : 61-73
Keywords : Big Data; HDFS; Hadoop; Hive; MapReduce; linear regression;
Abstract
Big data represents the information assets characterized by a high volume, velocity and variety to require specific technology and analytical methods for its transformation into value. The storage of large chunks of data is difficult as even terabytes and petabytes of traditional data warehousing solutions is insufficient and exorbitant [1][2]. It is viable to store and process these ransom amounts of data [13][14]15][16]17]18][19][20][21] on Hadoop; which is a low cost, reliable, scalable and fault tolerant Java-based programming framework that supports the processing of large data sets in a distributed computing environment. Hadoop implements MapReduce programing model for storing and processing large data sets with a parallel, distributed algorithm on commodity hardware. Nevertheless, the programming model expects the developers to write bespoke programs that are less flexible, time consuming, hard to code; maintain and reuse. This challenging task of writing complex MapReduce codes was rationalized by making use of HiveQL. Hive is the platform required to run HiveQL. Hive is built on top of Hadoop to query Big Data. Internally the Hive queries are converted into the corresponding MapReduce task [3][4]. In this paper, by making use of machine learning algorithm a movie rating prediction system is built based on MovieLens dataset
Other Latest Articles
- THE PRINCIPLE OF ADVANCED TEACHING AS AN IMPORTANT ASPECT OF PEDAGOGICAL MANAGEMENT
- THE USE EFFECTIVENESS OF STRUCTURAL-LOGIC SCHEMES IN THE PROCESS OF TEACHING SOCIAL AND HUMANITARIAN DISCIPLINES OF THE MILITARY INSTITUTION
- A REVIEW OF RESOURCE ALLOCATION TECHNIQUES IN CLOUD COMPUTING
- STUDY OF VARIOUS APPLICATIONS OF INTERNET OF THINGS (IOT)
- COMPARATIVE STUDY OF SUPERVISED LEARNING FOR STUDENT PERFORMANCE EVALUATION
Last modified: 2018-09-15 22:31:54