Security Operations

Where to collect logs

4

Network components

e.g. DHCP, DNS, Proxy, IPS/IDS, Firewall, VPN, NetFlow

Application services

e.g. Web/email server, DNS proxy server

Endpoints

e.g. windows, linux

Endpoints - windows

5

Windows Event Viewer:

Application logs,

System logs,

Security logs.

Critical: events that require your immediate attention, and generally mean that an application or system component failed or stopped responding.

Error: events that show problems exist, but which are not necessarily critical to the welfare of the application or of system components.

Warning: events that indicate potential problems, but which do not mean problems are sure to happen.

Information: events that are simply sending bits of information about their operation.

Verbose: shows detailed information about events.

Endpoints - linux

6

Linux has the following types of logs:

Application Logs

Event Logs

Service Logs

System Logs

Logs manipulation

7

Indexing and log normalisation

8

To be able to correlate log events

Normalisation is an effort to change event log formats from different vendors and network components so they are as universal as possible within your network.

Log indexing is a method of log management where logs are arranged as keys based on some attributes. 

Normalize Timestamps in a Log File

9

Wed, 28 Jun 2017 13:27:18 GMT = 1498656439

Human Readable Unix Epoch

Demo: Converting Epoch to Human Readable Timestamps with AWK

Consider the applicationX_in_epoch.log file below:

2|Z|1219071600|AF|0

3|N|1219158000|AF|89

4|N|1220799600|AS|12

1|Z|1220886000|AS|67

5|N|1220972400|EU|23

6|R|1221058800|OC|89

2|Z|Mon 18 Aug 2008 11:00:00 AM EDT|AF|0

3|N|Tue 19 Aug 2008 11:00:00 AM EDT|AF|89

4|N|Sun 07 Sep 2008 11:00:00 AM EDT|AS|12

1|Z|Mon 08 Sep 2008 11:00:00 AM EDT|AS|67

5|N|Tue 09 Sep 2008 11:00:00 AM EDT|EU|23

6|R|Wed 10 Sep 2008 11:00:00 AM EDT|OC|89

awk 'BEGIN {FS=OFS="|"}{$3=strftime("%c",$3)} {print}' applicationX_in_epoch.log > applicationX_in_human.log

Timestamps are used in log entries to specify when the recorded event took place. While it is best practice to record timestamps in UTC, the format of the timestamp varies from log source to log source. There are two common timestamp formats, known as Unix Epoch and Human Readable.

Unix Epoch timestamps record time by measuring the number of seconds that have passed since January 1st 1970.

Human Readable timestamps record time by representing separate values for year, month, day, hour, minute, and second.

The Human Readable Wed, 28 Jun 2017 13:27:18 GMT timestamp is the same as 1498656439 in Unix Epoch.

The command above is an AWK script. It may seem complicated. The main structure of the AWK script above is as follows:

awk – This invokes the AWK interpreter.

‘BEGIN – This defines the beginning of the script.

{} – This defines actions to be taken in each line of the input text file. An AWK script can have several actions.

FS = OFS = “|” – This defines the field separator (i.e., delimiter) as the bar (|) symbol. Different text files may use different delimiting characters to separate fields. This operator allows the user to define what character is used as the field separator in the current text file.

$3 – This refers to the value in the third column of the current line. In the applicationX_in_epoch.log, the third column contains the timestamp in epoch to be converted.

strftime - This is an AWK internal function designed to work with time. The %c and $3 in between parenthesis are the parameters passed to strftime.

applicationX_in_epoch.log – This is the input text file to be loaded and used. Because you are already in the lab.support.files directory, you do not need to add path information, /home/analyst/lab.support.files/applicationX_in_epoch.log.

Exercise

10

Make: Part 1 Normalize Timestamps in an Apache Log File

10 minutes

Download week 2b inclass log exercises

Query logs

11

SIEM interface (Regular Expressions)

SQL

Script/command line

Grep

Output redirection

Pipe

Regular Expression

12

A regular expression (regex) is a pattern of symbols that describes data to be matched in a query or other operation.

Regular expression engines: C, java, Perl, Python, Ruby, Unix, Apache, MySQL, etc

Tools: grep, Atom, notepad++ , RegexBuddy, RegexMagic,

Online https://regexr.com, https://regex101.com/,

Tutorial

13

https://regexone.com/