week6

Lab# 9 – FALL 2019

Lab#9-Apache Spark

Narender Reddy Kudumula

University of Cumberlands

Data Science & Big Data Analysis (ITS-836)

Prof. Dr. Gasan Elkhodari

11/17/2019

The Data

1. Exercise Directory: $DEV1/exercises/spark-application

2. Data files (HDFS): /loudacre/weblogs

3. Application: CountJPG.py ( see code below)

The Task

4. In this Exercise you will write your own Spark application instead of using the interactive Spark Shell application.

5. Write a simple program that counts the number of JPG requests in a web log file. The name of the file should be passed into the program as an argument.This is the same task you did earlier in the “Use RDDs to Transform a Dataset” exercise. The logic is the same, but this time you will need to set up the Spark Context object yourself.

6. Depending on which programming language you are using, follow the appropriate set of instructions below to write a Spark program.

7. Before running your program, be sure to exit from the Spark Shell.

The Code

import sys

from pyspark import SparkContext

from pyspark import SparkConf

if __name__ == "__main__":

if len(sys.argv) < 2:

print >>sys.stderr, "Usage: CountJPGs<file>"

exit(-1)

sc = SparkContext()

logfile = sys.argv[1]

count = sc.textFile(logfile).filter(lambda line: '.jpg' in line).count()

print "Number of JPG requests: ", count

sc.stop()

4