Spark has become a necessary tool for big data computing in this
modern age because it is built around efficiency and uses the
combination of speed, ease of use and flexibility to provide solutions
to data needs across the industry. According to reading material
covered in this learning module, and the Apache Spark website, Spark
can run some data processing jobs up to 100 times faster than
Hadoop MapReduce.
Ease of use: is a main focus of the Spark creators and they achieve
this by offering more than 80 high-level, data processing operators
enabling developers, data scientists, and analysts to use and build
many useful data applications. The availability of the operators in
multiple different languages (e.g. Java, Python, R, etc) adds additional
advantages for software engineers, data scientists, and data analysts
by allowing them to choose their preferred language to solve large-
scale data processing problems with Spark.
Flexibility: Spark can solve multiple types of data processing
workloads, through one unified data processing stack, including batch
applications, interactive queries, and machine learning algorithms;
these tools and processes require multiple iterations, and real-time
streaming applications to extract insights very quickly. However
before Spark, each of these types of workloads used to require a
different solution and technology. Nowadays, Spark can be used for
all these data processing needs, which significantly reduces cost.
Lastly, Spark has proven to be a viable and highly efficient solution to
the inefficiencies observed in how Hadoop MapReduce f handles
interactive and iterative data processing.
