Real-time Streaming Data Analysis using Spark

Streaming data generates a lot of information in real-time. Various types of streaming data exist, and analyzing such streaming data in real-time is a very important issue. Streaming data such as IoT sensor data, SNS Twitter data, stock data could contain very sensitive information that need to be analyzed in real-time. Our approach to analyze streaming data is based on real-time twitter data. We used Spark streaming provided by Apache Spark API for processing streaming data. We analyzed hashtags of real time twitter streaming data to find some related information about specific interesting keywords. In this paper, we did two kinds of analysis. First, searching the Top 10 hashtags related to a keyword. Second, analyzing how many words used related with happiness represented as integer values.