Spark has become a necessary tool for big data computing in this
modern age because it is built around efficiency and uses the
combination of speed, ease of use and flexibility to provide
solutions to data needs across the industry. According to reading
material covered in this learning module, and the Apache Spark
website, Spark can run some data processing jobs up to 100 times
faster than Hadoop MapReduce.
Ease of use: is a main focus of the Spark creators and they achieve
this by offering more than 80 high-level, data processing operators
enabling developers, data scientists, and analysts to use and build
many useful data applications. The availability of the operators in
multiple different languages (e.g. Java, Python, R, etc) adds
additional advantages for software engineers, data scientists, and
data analysts by allowing them to choose their preferred language
to solve large-scale data processing problems with Spark.
Flexibility: Spark can solve multiple types of data processing
workloads, through one unified data processing stack, including
batch applications, interactive queries, and machine learning
algorithms; these tools and processes require multiple iterations,
and real-time streaming applications to extract insights very quickly.
However before Spark, each of these types of workloads used to
require a different solution and technology. Nowadays, Spark can
be used for all these data processing needs, which significantly
reduces cost.
Lastly, Spark has proven to be a viable and highly efficient solution
to the inefficiencies observed in how Hadoop MapReduce h handles
interactive and iterative data processing.
