Simply put, Spark, and other tools like it, are required for big data
computing because of one thing, speed. Moving from using standard,
fine-tuned data to using Big Data means having to take a vastly larger
number of potentially unstructured datapoints and consolidating them
down to a tangible solution. For an older system like MapReduce, this
increase would throttle the system to an unusable state. With Spark,
however, the ability to process data in a multi-threaded manner,
coupled with the fact that it can use cache memory (like RAM), and not
just disk-based memory (like a hard drive), means that Spark can
process data 100 times faster than MapReduce. This increase is
processing speed is required when dealing with the large and complex
world of Big Data. Also, by being able to utilize cached and electronic
memory, Spark can operate well in the cloud, which is where the
majority of Big Data operations are heading towards. This cluster
computing can increase the productivity of any business by processing
more data at a faster rate, allowing for more time to analyze the data
and conclusions that come from it.
Two alternatives to Apache Spark, which happen to come from major
tech companies, are Google BigQuery and Amazon Redshift. Both of
these tools are used to process large amounts of unstructured data,
but they have their advantages and disadvantages. I think that Spark is
the safer tool to use at the moment. When compared to Redshift,
Spark is able to perform real-time processing, whereas Redshift has to
process in batches, meaning there will be some delay in getting new
data processed. So Spark would be better suited to a situation where
you need updates constantly and frequently. Redshift, given it's
architecture would perform processing faster when there is a bulk
amount of data waiting to be processes, and real-time turnaround is
not an issue. When comparing Spark to BigQuery, BigQuery tends to
be faster whether the datasets are large or small, but the language
used can be complex or requires a learning curve to become proficient.
Spark is by no means slow, so I think that if getting your Big Data
processing system up and running is the most important, than Spark is
probably the better option. If you have the time and resources to learn
how to code in it, BigQuery may be the better option. Like most things
in life, there is no one tool that is superior in every way.
An example of an RDD transformation with a narrow dependancy
would be the union function, which takes one or more RDDs and
combines them into a single child RDD. This transformation, and all
other transformations with a narrow dependency, is a one-to-one
transformation, meaning each partition of the original RDDs will only
contribute to one partition of the child RDD. GroupByKey is an
example of an RDD transformation with a wide dependancy. This
transformation is a one-to-many transformation, like all
transformations with a wide dependency, which means that each
partition of the original RDDs can contribute to any or all of the
partitions in the child RDD.
