PERFORMANCE COMPARISON BETWEEN HIVE AND SPARK IN STRUCTURED BIG DATA ANALYSIS

Abstract

With the explosion of IoT devices, the massive amount of data generated has rendered traditional data analysis platforms inadequate. Big data analytics platforms have emerged, enabling organizations to uncover hidden values within the data and make swift business decisions. Among these, Apache Hive and Apache Spark have risen as two modern data analysis platforms commonly used for analyzing structured big data. Several studies have compared these two platforms and found that in most cases, SparkSQL in Spark is faster than Hive on MapReduce. However, in some cases involving queries with lots of joins operations among large data tables, SparkSQL is slower than Hive on MapReduce. Recently, SparkSQL has undergone numerous optimization techniques that could lead to better query performance, such as upgrading the query processing scheduler optimizer. In this paper, we present the performance comparision between Hive on MapReduce and SparkSQL. To achieve this, we study the implementation of performance evaluation for big data queries using BigBench and the performance analysis tool PAT. We aggregate and analyze important performance parameters, including query execution time, disk usage, memory usage, CPU usage, and the amount of data read/written over the network. The experimental results show that Spark SQL outperforms Hive on MapReduce with all 20 query sets of BigBench.