Many engineers and top IT companies use Spark for big data
computing because it has the following advantages: 1. Spark is fast.
Its speed can reach several times or even dozens of times that of
MapReduce. Spark is suitable for real-time data processing. After
Spark loads the data into the memory, iterative calculations can
directly use the intermediate results in the memory for operations,
avoiding the need to frequently read data from the disk. 2.
Versatility. Spark provides components such as Spark RDD, Spark
SQL, Spark Streaming, Spark MLlib, and Spark GraphX, which can be
applied to offline batch processing, interactive query, stream
computing, machine learning, graph computing, etc. 3. Spark
supports programming languages such as Scala, Java, Python, and R.
4. Spark can run on Hadoop or on cloud environments such as
Amazon EC2, and can access various data sources such as HDFS,
Cassandra, HBase, Hive, etc. ([1]) The combination of Hadoop for
storage and Spark for big data computing is very popular in the field
of data.
Narrow dependencies mean that each partition of the parent RDD is
used by only one partition of the child RDD. Wide dependencies
mean that each partition of a parent RDD may be used by multiple
child RDD partitions. Since wide dependencies cause large overhead
among partitions and the transmission efficiency is low, we should
reduce the use of wide dependencies. ([2]) The union function is a
narrow dependency operation, and the groupByKey function is a
wide dependency operation. The union combines two result sets
into one and then filters out duplicate records. The groupByKey
receives key-value pairs (K, V) as input, groups the values based on
the key, and generates a dataset of (K, Iterable) pairs as output. The
groupByKey moves all the key-value pairs in the partition.
Reference:
1. Ian Pointer. What is Apache Spark? The big data platform that
crushed Hadoop.
https://www.infoworld.com/article/3236869/what-is-apache-spark-
the-big-data-platform-that-crushed-hadoop.html
2. Dave Canton. Wide and Narrow dependencies in Apache Spark.
https://medium.com/@dvcanton/wide-and-narrow-dependencies-in-
apache-spark-21acf2faf031
