Lab 01 | Processes, File Systems and Buffer Overflows

IS 3033 – Operating Systems Security

Mila Paul, PhD

Lab 1

Linux Processes, File Systems and Buffer Overflows

Instructions:

1. Complete Part 1 and screenshots

2. Complete Part 2 and screenshots

3. Answer the Reflection Questions 4. Turn it all in on one document (no need to turn in the instructions and

questions).

Reflection Questions (provide answers in bold red to distinguish them from the

questions or just turn in answers in full sentences)

Answer these questions when you have completed Part 1 and Part 2.

1. In a multitasking operating system, why is it essential to have a mechanism like ps to list running processes?

2. Can you think of real-world scenarios where identifying running processes would be

crucial for system management or security?

3. Why is it valuable for system administrators to know the PID of running processes?

4. In terms of system performance optimization, how can understanding CPU and

memory usage help in resource management?

5. Explain how processes transition between states and the role of the scheduler.

6. Explain the importance of CPU time sharing among processes.

7. Discuss the importance of understanding processes and scheduling for

cybersecurity?

8. What is a buffer overflow and, concisely, how does it happen?

9. What is the –m32 in this command gcc -o vulnerable.c -m32 vulnerable?

10. What does this python command ./vulnerable $(python -c 'print("A" * 80)') do?

11. What is ASLR and how does it prevent a buffer overflow?

12. What can you use GDB for in Linux?

13. Why is GDB used when a program crashes or gets a segmentation fault?

Play with Processes on your Ubuntu VM: Lab 01 Part 1

Turn in screenshots to show your work

Use docx or pdf

Turn in only ONE DOCUMENT with all of your screenshots and questions.

1. Set Up Ubuntu VM: (This part is your Lab 00 no need to resubmit)

Download and install Hypervisor (VirtualBox or VMware)

Download an Ubuntu Desktop ISO image from the official website.

Create a new virtual machine and install Ubuntu using the ISO image (CD/DVD) as the installation source.

2. Open Terminal and List Running Processes Launch the Terminal on your Ubuntu system

Run the following command to list all running processes ps aux

Screenshot

3. Understand Key Information displayed by the ps command

PID (Process ID): Unique identifier for each process.

%CPU: CPU usage as a percentage.

%MEM: Memory usage as a percentage.

VSZ: Virtual memory size in kilobytes.

RSS: Resident Set Size (actual memory used) in kilobytes.

TTY: Terminal associated with the process (e.g., pts/0).

STAT: Process status (e.g., R for running, S for sleeping).

START: Start time of the process.

TIME: Total time the process has been running.

COMMAND: Command or program associated with the process.

4. Demonstrate different ps command options to customize the output

Use ps -ef to display a full listing with additional details.

Use ps -e --sort=-%cpu to sort processes by CPU usage in descending order.

Use ps -u username to show processes for a specific user (replace "username" with an actual username).

Screenshot

5. Understand Process States and Process Management Understand Process States (Running, Waiting, etc.) and their meanings.

Manage processes using commands like kill, killall, and pkill.

Demonstrate how to use these commands on their processes

Screenshot

6. Process Prioritization and Process Scheduling Review the concept of process priority and scheduling policies (e.g., Round Robin, Priority Scheduling) in Linux.

Show how to change process priority using nice and renice commands

Use the top command to monitor and interact with the scheduler in real-time

Screenshot

7. Process-Related Files and Directories Explore the /proc filesystem and its role in providing information about running processes.

Navigate and extract useful information from /proc

Screenshot

8. Create New Processes Create new processes using the & operator, nohup, and bg commands.

Screenshot

Create a Buffer Overflow on your Ubuntu VM: Lab 01

Part 2 Turn in screenshots to show your work

Use docx or pdf

Turn in only one document

1. Set Up Ubuntu VM: (This part is your Lab 00) Download and install Hypervisor (VirtualBox or VMware)

Download an Ubuntu Desktop ISO image from the official website.

Create a new virtual machine and install Ubuntu using the ISO image (CD/DVD) as the installation source.

2. Install Development Tools Open a terminal in your Ubuntu VM.

Install build-essential and gdb for development and debugging tools.

Install multilib so you can compile your program as 32-bit

You will run it as 32-bit to ensure the vulnerability works

sudo apt update

sudo apt install build-essential gdb

sudo apt-get install gcc-multilib

3. Create a Vulnerable C Program Create a simple C program with a buffer overflow vulnerability.

Type gpedit in the terminal window

Paste the code below in the text file that pops up. Check it twice.

Save it as vulnerable.c

#include <stdio.h>

#include <string.h>

void vulnerable_function(char *input) {

char buffer[64];

strcpy(buffer, input);

printf("Buffer: %s\n", buffer);

}

int main(int argc, char **argv) {

if (argc != 2) {

printf("Usage: %s <input>\n", argv[0]);

return 1;

}

vulnerable_function(argv[1]);

return 0;

}

4. Compile the Vulnerable Program

gcc -o vulnerable.c -m32 vulnerable

5. Disable ASLR -Disable ASLR which is a Microsoft feature that prevents buffer overflows by randomly moving data to different places in

memory. This makes it more unpredictable for the malware writers to know where to send the malicious code.

sudo sysctl -w kernel.randomize_va_space=0

6. Test the Vulnerable Program Run the program with a long input to trigger the buffer overflow.

./vulnerable $(python -c 'print("A" * 80)')

You should observe a segmentation fault due to the buffer overflow (This means that if your program crashes, you

did it right!).

In the next step, we will begin inspecting our code by stepping through it line by line with GDB

GDB will help us see exactly where the program crashed in the buffer overflow!

7. Set Up GDB for Debugging Open the program in GDB for debugging.

gdb -q ./vulnerable

8. Debug the Vulnerable Program Set a breakpoint at the vulnerable function and run the program

Break vulnerable_function

run $(python -c 'print("A" * 80)')

You can now examine the stack and registers to understand how the buffer overflow occurred (These stacks and

registers will be the parts of your virtual memory with memory addresses!)

9. Cleanup After completing the lab, reset ASLR to its default value (if you don't do this, your VM will be vulnerable to

common attacks!)

sudo sysctl -w kernel.randomize_va_space=2