week6

Lab #7

Narender Reddy Kudumula

University of Cumberlands

Data Science & Big Data Analysis (ITS-836)

Prof. Dr. Gasan Elkhodari

11/03/2019

Lab#07

Apache web server logs are generally stored in files on the local machines running the server. In this exercise, you will simulate an Apache server by placing provided web log files into a local spool directory, and then use Flume to collect the data. Both the local and HDFS directories must exist before using the spooling directory source

1. Create a directory in HDFS called /loudacre/weblogs to hold the data files Flume ingests, e.g.

$ hdfs dfs -mkdir /loudacre/weblogs

2. Create a local directory for web server log output

sudo mkdir -p /flume/weblogs_spooldir

3. Give all users the permissions to write to the /flume/weblogs_spooldir

$ sudo chmod a+w -R /flume

4. Configure Flume

In $DEV1/exercises/flume , create a Flume configuration file with the characteristics listed in the attached file in the week’s content.

5. Run the Agent

$ flume-ng agent --conf /etc/flume-ng/conf \

--conf-file solution/spooldir.conf \

--name agent1 -Dflume.root.logger=INFO,console

6. Wait a few moments for the Flume agent to start up. You will see a message like: Component type: SOURCE, name: webserver-log-source started

7. Open a separate terminal window and change to the exercise directory. Run the script to place the web log files in the HDFS: /flume/weblogs_spooldir directory:

$ cd $DEV1/exercises/flume

./copy-move-weblogs.sh /flume/weblogs_spooldir

This script will create a temporary copy of the web log files and move them to the spooldir directory.

8. Return to the terminal that is running the Flume agent and watch the logging output. The output will give information about the files Flume is putting into HDFS.

9. Once the Flume agent has finished, enter CTRL+C to terminate the process.

10. Using the hdfs command line or Hue File Browser, list the files in HDFS that were added by the Flume agent.

Note that the files that were imported are tagged with a Unix timestamp corresponding to the time the file was imported, e.g. FlumeData.1427214989392