Computer Architecture Reflective Journal Report

*

Systems Architecture,
Seventh Edition

Chapter 12

Secondary Storage Management

*

Systems Architecture, Seventh Edition

Chapter Objectives

• In this chapter, you will learn to:
• Describe the components and functions of a file management system
• Compare the logical and physical organization of files and directories
• Explain how secondary storage locations are allocated to files and describe the data structures used to record those allocations

*

Systems Architecture, Seventh Edition

Chapter Objectives (continued)

• Describe file manipulation operations
• List access controls that can be applied to files and directories
• Describe file migration, backup, and recovery methods
• Explain methods for ensuring fault tolerance
• Compare storage consolidation methods, such as storage area networks, network-attached storage, and cloud-based storage services

*

Systems Architecture, Seventh Edition

FIGURE 12.1 Topics covered in this chapter

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

File Management Overview

• Users and programs don’t interact directly with secondary storage blocks and devices
• They interact with files – groups of storage blocks that hold programs and data
• File management functions such as creation, copying, reading, and writing are implemented within the service layer (a.k.a. file control layer)
• Application programs interact directly with file management service layer routines
• Users interact with the command layer which, in turn, uses service layer routines to perform file management tasks

FIGURE 12.2 File management system layers

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

Logical and Physical Storage Views

• Secondary storage devices hold bits, but access those bits in larger units generically called blocks (for example, one sector of a magnetic hard disk)
• Blocks are to small for users and programs to directly interact with
• Users and programs need larger and more useful logical abstractions, including:
• File – a group of blocks that holds one related set of data or instructions
• Folder (directory) – a container for a related set of files
• Volume – a container for a hierarchically organized set of folders and files
• For example, C: on Windows

FIGURE 12.3 Logical and physical secondary storage views for a typical laptop computer

*

Systems Architecture, Seventh Edition

File Content and Type

• Most FMSs directly support only a few file types:
• Executable programs
• Operating system commands
• Textual or unformatted binary data
• Other file types are indirectly supported by file association: a defined relationship between file types and the programs or OS utilities that manipulate them

*

Systems Architecture, Seventh Edition

Volumes, Files, and Folders in Windows Explorer

*

Systems Architecture, Seventh Edition

Volumes

• A single disk can be divided into multiple volumes or a volume can span multiple disks
• A volume is an independent FMS management unit for purposes of:
• Driver letter (Windows) or mount point (UNIX/Linux) assignment
• Error checking, backup/recovery, defragmentation, and recycling
• Root directory access controls
• Quotas, versioning, journaling, and other optional FMS features

*

Systems Architecture, Seventh Edition

File Content and Organization

• Files can contain many different kinds of data:
• Machine or OS instructions (for example, a Windows .exe file or Linux shell script)
• Textual data (for example, as created by Notepad)
• Image data (for example, as capture by a camera)
• Formatted documents (for example, a Web page)
• Variations in file content lead to variations in internal file organization, for example:
• Files containing machine language programs are organized to simplify loading the instructions into memory
• Files containing web pages are organized to simplify transmission across a network and Web browser display
• Image files are organized to simplify display and editing

*

Systems Architecture, Seventh Edition

File Type and Association

• Operating systems generally support two file-related:
• File type – A limited set of file types that the OS keeps track of and uses to enable or restrict certain functions, for example:
• Types – executable instructions, OS commands, other
• Enable/restrict:
• Executable files can be loaded into memory and executed by the CPU but can’t be printed or read/written by programs
• OS command files (scripts) can be executed by an OS utility and can be printed and read/written by programs
• Other files can’t be executed but can be printed and read/written by programs
• File type association – Specific file types are associated with specific programs that perform specific actions on those files, for example:
• Ordinary text file – Notepad used to print/edit
• JPEG image file – Adobe Illustrator used to print/edit
• HTML file – Edited by Word and displayed/printed by Internet Explorer
• Windows provides an extensible set of file types
• Filename extension determines file type, for example
• Document.txt is a text file associated with Notepad
• Document.doc is a document file associated with Microsoft Word
• Double-clicking a file opens it within the program associated with that file’s type

*

Systems Architecture, Seventh Edition

File Type and Association - Exercise

• Perform the following tasks on computer running Windows:
• From the desktop, click Start and select Control Panel
• Click Programs
• Click Default Programs
• Click Associate a file type or protocol with a specific program
• View the list of file types and their default program associations
• Select .txt and click Change Program
• View the list of programs that could be assigned to .txt files (click Cancel when you’re finished)
• Close all of the open control panel windows
• Open a folder containing some of your own files, preferably of several different types – note the displayed file names
• Click Organize in the upper left menu bar
• Click Folder and search options
• Click the View tab
• Uncheck the box labeled Hide extensions for know file types
• Click OK
• Reexamine the displayed file names and note that all files with the same extension have the same icon displayed to the left of the filename.

*

Systems Architecture, Seventh Edition

FIGURE 12.4 Registered Windows file types and associated programs

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

FIGURE 12.5 Context menus and commands for a Windows file type

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

Folder Content and Structure

• Contain information about files and other folders, typically:
• Name
• File type
• Location
• Size
• Ownership
• Access controls
• Time stamps

*

Systems Architecture, Seventh Edition

FIGURE 12.7 Windows folder listing with additional details

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

Hierarchical Folder Structure

• In most OSs, folders can contain other folders, but no folder can be contained within more than one folder
• Result is a hierarchical (or tree) folder structure as shown in the left frame of the next slide
• Terminology (refer to next slide for examples)
• Current (working) directory – folder currently displayed or the folder location within which a program looks for files
• For example, Chapter08 in the figure
• Home folder – default current folder for a user or program
• For example, C:\Users\Burd\My Documents for the user Burd under Windows
• Complete path or fully qualified reference – filename plus all folder names upward through the volume root
• For example, T:\Systems Architecture\6e\Chapters\Chapter08\Solutions_08_Au.doc
• Relative path – file name plus all folder names upward to the current (working) folder
• For example, if the current folder is 6e then .\Chapters\Chapter08\Solutions_08_Au.doc is a relative path to Solutions_08_Au.doc where . is a shorthand for the current folder

*

Systems Architecture, Seventh Edition

Current (working) directory

FIGURE 12.8 A hierarchical directory structure

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

Graph Folder Structure

• Graph folder structures allow a folder to be contained within multiple other folders
• That introduces the possibilities of loops, which complicate operations such as error checking and enumerating volume contents
• Many OSs take an “in-between” approach based on links or shortcuts (see dotted line in figure)

FIGURE 12.9 A graph folder structure

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

Storage Allocation Overview

• The user doesn’t “see” hard disk drives, SSDs, and other secondary storage devices
• The user does see volumes, folders, and files managed by the operating system
• Storage allocation is the processes of managing or linking the relationship(s) between:
• Storage objects the user sees (a logical view), and
• Storage devices/locations in hardware (a physical view)

*

Systems Architecture, Seventh Edition

Allocation Units

• Recall from chapter 5 that most disk drives (and devices that simulate them) exchange data move data back and forth to memory in 512 or 4096 byte chunks called sectors
• An operating system manages chunks of secondary storage called allocation units, which may be the same size as sectors or a multiple thereof
• Allocation units are assigned (or allocated) to logical storage objects such as volumes, directories, and files
• Allocation unit size is determined when a volume is initialized – size determines:
• How efficiently storage space is allocated and managed
• The size of data structures (tables) that track allocation units

*

Systems Architecture, Seventh Edition

Allocation Units (continued)

• Allocation unit size trade-offs
• Efficient use of secondary storage space for files
• Size of storage allocation data structures
• Efficiency of storage allocation procedures
• Smaller units: more efficient use of storage space
• Larger units: allow for smaller storage allocation data structures

*

Systems Architecture, Seventh Edition

Storage Allocation Tables

*

Systems Architecture, Seventh Edition

Storage Allocation Tables - Continued

• Small allocation unit size (and large storage allocation tables) wastes less storage space when there are many small files (smaller than the allocation unit size)
• For example, storing a 1 byte file wastes:
• 511 bytes if allocation unit size is 512 bytes
• 4095 bytes if allocation unit size is 4096 bytes
• Larger allocation unit size (and smaller storage allocation tables) speeds tasks that need to search or update the table
• For example, calculating total free space or copying a large file
• The proper trade-off among the two factors depends on many factors including average file size, volume size, disk I/O speed, and available RAM for buffering and caching
• Allocation unit size can be chosen by the system administrator or automatically determined by the OS when a volume is created

*

Systems Architecture, Seventh Edition

FIGURE 12.8 Storage blocks allocated to three files

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

Free allocation units are assigned to a hidden system file called SysFree.

TABLE 12.1 Directory content for the files in Figure 12.8

*

Systems Architecture, Seventh Edition

FIGURE 12.9 A storage allocation table matching the storage allocations in Figure 12.8

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

Blocking

• A logical record is the unit of data read from or written to a file by an application program (e.g., all of the data about one customer)
• A physical record is the unit of data read from or written to a storage device in a single operation
• Comparing the relative size of logical and physical records, there are three possible relationships:
• Logical record size = physical record size - unblocked
• Logical record size < physical record size
• Logical record size > physical record size

*

Systems Architecture, Seventh Edition

Blocking - Continued

If logical and physical records are of different sizes then the term blocking factor describes the number of logical records per physical record

Burd, Systems Architecture, 7th edition, Figure 12.12 Copyright © 2016 Cengage

*

Systems Architecture, Seventh Edition

Blocking and Buffering

• The FMS or OS uses buffers to support data transfer between secondary storage and an application program
• Each buffer is the same size as a physical record – 512 or 4096 bytes for most disk drives
• Multiple buffers can be used to improve performance
• A physical read operation moves one physical record from a secondary storage device to a buffer
• A logical read operation moves one logical record from one or more buffers to the application program

FIGURE 12.11 Input from secondary storage to an application program using a buffer

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

File Manipulation

• Exact set of service layer functions varies among FMSs, but typically includes create, copy, move, delete, read, and write
• Application programs interact directly with FMS through OS service layer
• Users interact indirectly with FMS through command layer

*

Systems Architecture, Seventh Edition

File Open and Close Operations

• The OS service layer contains utility functions to open and close files:
• File open operation

Locate the file in the directory structure and read its directory entry

Search an internal table of open files to see whether the file is already open

Ensure that the process has enough privileges to access the file

Allocate one or more buffers

Update an internal table of open files

• File close operation

Flush the program’s file I/O buffers to secondary storage

Deallocate buffer memory

Update the file’s directory entry time stamps

Update the open file table

*

Systems Architecture, Seventh Edition

File Delete and Undelete Operations

• There are multiple ways for an OS/FMS to perform a file delete operation – the most common are:
• Move the file’s directory entry to a directory of deleted files (e.g., a recycle bin)
• A file undelete operation simply moved the directory information back to its original location
• Mark the file as deleted in its directory entry and mark its allocation units as free in the storage allocation table
• File undelete is possible if neither the directory entry nor the storage allocation table have been overwritten with new data
• Overwrite the data content of the file’s allocation units with zeros, mark the allocation units as free in the storage allocation table, and overwrite the file’s directory entry with zeros
• File undelete isn’t possible via the OS/FMS, though some advanced forensic techniques can examine individual disk bits and determine earlier values of those bits

*

Systems Architecture, Seventh Edition

Access controls

Each file or folder includes an access control list (ACL) that specifies users or groups and what they’re allowed to do (sometimes described by the word privileges)

The ACL is checked against user ID when a file is opened and manipulated

FIGURE 12.14 Windows access control list content

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

File Migration (Version Control)

• Versioning - As file contents are updated creating new versions, older versions are archived
• Older versions “migrate” from:
• Fast local storage
• Slower local storage
• Remote storage such as cloud-based backups
• Older versions can still be “seen” in folder listings and are copies back if needed
• Balances storage cost of each file version with anticipated user demand for that version

*

Systems Architecture, Seventh Edition

File Backup

• Protects against data loss (file content, folder content, and storage allocation tables)
• Storing backup copies:
• Different storage device within local computer
• Removable storage device attached to local computer
• A local network-attached backup device or server
• A remote network-attached backup server

*

Systems Architecture, Seventh Edition

Archive Attributes and Timestamps

• Each file/directory entry includes data used by backup utilities to determine if or when a file was backed up.
• Windows uses a Boolean archive attribute which is set if file has been modified since last backup.

*

Systems Architecture, Seventh Edition

Backup Types

• Full backup
• Copies all selected files and directories
• Clears archive attributes
• Incremental backup
• Copies only files and directories created or changed since the last incremental backup (files with the archive attribute set)
• Clears archive attributes
• Differential backup
• Copies only files and directories created or changed since the last normal or incremental backup (files with the archive attribute set)
• Doesn’t clear archive attributes

*

Systems Architecture, Seventh Edition

Transaction Logging

• Automatically records all changes to file content and attributes in a separate storage area
• Also writes them to the file’s I/O buffer
• Provides high degree of protection against data loss due to program or hardware failure
• Imposes a performance penalty
• Used only when costs of data loss are high

*

Systems Architecture, Seventh Edition

File Recovery

• Automated and manual components
• Can search backup logs for copies of lost or damaged files
• Can perform consistency checking and repair procedures for crashed system or physically damaged storage device
• All storage locations appear in the storage allocation table and other data structures.
• All files have correct folder entries.
• All storage locations of a file can be accessed through the storage allocation table.
• All storage locations can be read and/or written.

*

Systems Architecture, Seventh Edition

Fault Tolerance

• What is fault tolerance?
• The term fault tolerance describes software, hardware, and operating procedure characteristics that ensure:
• Minimal resource/service unavailability due to faults
• Minimal data loss due to faults
• Common fault causes:
• Power interruption
• Hardware failure (e.g., disk or network crash)
• Operating system failure (BSOD - blue screen of death)
• Service failure (e.g., web service crash, denial of service attack)

*

Systems Architecture, Seventh Edition

Fault Prevention and Mitigation

• Most faults can’t be completely prevented, but their probability of occurrence can be reduced and their negative consequences can be minimized
• Common fault prevention strategies:
• Adequate security
• Power conditioning
• Reliability testing and configuration
• General fault mitigation strategies
• Hardware redundancy (e.g., RAID, backup servers)
• Service redundancy (e.g., load balancing)
• Resource redundancy (e.g., file replication and periodic backup to redundant and/or remote storage devices)

*

Systems Architecture, Seventh Edition

Mirroring

• All disk write operations are made concurrently to two different storage devices
• Provides high degree of protection against data loss with no performance penalty if implemented in hardware
• Disadvantages
• Cost of redundant disk drives
• Higher cost of disk controllers that implement mirroring

*

Systems Architecture, Seventh Edition

RAID Overview

• RAID stands for redundant array of inexpensive disks
• RAID (most levels) achieves fault tolerance through various types and levels of redundant storage
• Higher redundancy usually implies greater fault tolerance
• RAID (most levels) achieves performance improvements through parallelism
• There are multiple kinds of RAID (a.k.a. RAID levels)
• Each level is a different combination or parallelism, redundancy amount, and redundancy type
• “Standard” RAID levels include 0-5
• RAID 2, 3, and 4 are antiques
• RAID 10 is widely used but there is no “standard”

*

Systems Architecture, Seventh Edition

RAID 0 (Striped Volumes)

• Striped Volume (RAID-0)
• Volumes on multiple disks are combined into one volume
• Data is systematically distributed across all disks
• For example, a 1 MB file is spread equally across four disks in rotating 64 KB blocks resulting in 256 KB stored on each disk
• There is some associated CPU overhead.
• Pros - data access is faster IF all disks can be read/written in parallel (disk controller must support parallelism)
• Cons - if any disk fails the entire volume is trashed
• Alternatives:
• One big VERY fast disk
• RAID-5 or higher
• Striping by itself is primarily used to increase read/write performance (e.g., a gaming desktop)
• Higher RAID levels combine striping with redundancy

*

Systems Architecture, Seventh Edition

FIGURE 12.16 Data striping across four disks

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

RAID 1 (Mirrored Volumes)

• Mirrored Volumes (RAID-1)
• Every volume has a duplicate on a different disk
• All writes go to both disks in parallel
• Reads come from either disk
• Pros:
• High fault tolerance
• Cons:
• Double cost or half capacity
• Write performance is always slower but not by much if using a smart disk controller
• As disks have become cheaper, mirroring (alone or in combination with redundancy) has become much more common

*

Systems Architecture, Seventh Edition

RAID-5

• RAID-5 Volume
• Volume spans multiple disks (at least three)
• Data content is striped
• Parity information is added within file in such a way that parity data alternates across disks per file/directory
• If any drive fails, data can be recovered from the remaining drives
• Lost parity information isn’t a problem (it’s redundant)
• Lost data can be recomputed from remaining blocks
• For example, assume 9 disks, 1 byte, even parity
• 1 1 0 0 1 1 0 0 + parity bit (0)
• A “lost” bit is zero if there are an even number of remaining 1 bits, one otherwise

*

Systems Architecture, Seventh Edition

RAID-5 ― Continued

• Pros:
• High fault tolerance - access continues in the event of any single disk failure
• High read performance - same as striped volume (parity data is ignored when reading)
• Cons:
• Slower write performance than striped volume due to computation and storage of parity information
• Storage capacity is less than striped - one disk worth of data is redundant parity information

*

Systems Architecture, Seventh Edition

RAID-5 Failure Recovery

• If one disk dies the system continues to function but without protection against additional failures
• To restore failure protection:
• Install new disk
• Format new disk
• Recreate lost data on new disk by doing “parity math”
• Recovery operation is slow because it’s both
• CPU intensive
• I/O intensive
• System throughput for “real” work typically declines – by how much?

*

Systems Architecture, Seventh Edition

FIGURE 12.18 16 KB stored in a four-disk RAID 10 array

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

Storage Consolidation

• Until at least the 1990s:
• File systems were always integral parts of an operating system
• Each operating system controlled one set of hardware, including secondary storage devices
• The term direct-attached storage (DAS) describes any architecture in which software “talks” to secondary storage devices that are directly connected to the computer system that executes the software

*

Systems Architecture, Seventh Edition

Direct-Attached Storage Limitations

• DAS made a lot of sense in the mainframe era
• Most organizations owned one mainframe
• All software ran on the mainframe
• All data was stored on directly-attached disks
• By the late 1980s it started to be a limitation:
• Organizations use multiple computer systems
• Could a file or database stored on disks attached to one computer system be accessed by software running on another computer system?
• Could two computer systems share a pool of disks or a common file system or database stored on that pool of disks?
• After multiple evolutionary steps – three main approaches were developed to “answer” the questions above:
• Storage-area network
• Network-attached storage
• Cloud-based storage

*

Systems Architecture, Seventh Edition

Storage Area Network

• A storage area network (SAN) consists of:
• A “back-end” network connecting a family of servers
• One server is a storage server and accepts I/O requests from the other servers via the network
• A pool of secondary storage devices attached to a storage server and managed as one large set of allocation units (disk sectors)

FIGURE 12.19 A server cluster with a storage area network

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

Network-Attached Storage

• Network-attached storage (NAS) uses:
• A pool of secondary storage devices attached to a storage
• A storage server manages the storage pool as one or more file systems and responds to read/write requests from other servers
• An ordinary WAN or LAN connects all servers

FIGURE 12.20 Network-attached storage

Courtesy of Course Technology/Cengage Learning

*

Systems Architecture, Seventh Edition

SAN and NAS Compared

• Access types
• SAN: Client servers perform direct disk I/O at the sector level via the storage server
• NAS: Client servers transfer service layer file I/O requests to the storage server for processing
• Performance
• SAN: Supports high-performance disk I/O for a small number of servers in close geographic proximity
• NAS: Supports lower-performance file I/O but can potentially support a larger number of servers over a wide geographic area
• Cost
• Relatively high – High-capacity back-end switching network
• Relatively low – storage server is an “appliance” and existing network connections are used

*

Systems Architecture, Seventh Edition

Cloud-Based Storage

• DAS, SAN, and NAS are holdovers from an era when files were assumed to be stored and accessed from a single device
• Data stored on a server and users accessed via an application
• Data stored on a personal computing device
• In the modern world, users have access to many computing devices and they want to access their files from any of them
• Examples of modern tools that enable user-centric storage:
• GoogleDrive
• DropBox
• Microsoft OneDrive

*

Systems Architecture, Seventh Edition

Cloud-Based Storage - Continued

• Common features of cloud-based storage services:
• Each user has a unique account and user ID
• A user can associated their cloud storage user ID with one or more computing devices (e.g., their smartphone, tablet, and laptop)
• User files stored on any associated device are automatically synchronized to and from the cloud and then to and from all other associated devices
• A Web-based application provides a way for users to access files from devices that haven’t been associated with their user ID
• The underlying technology is similar to caching
• A server or server group is the main storage device
• A volume or folder points to cloud-based storage
• As files area access they’re cached on local storage of each computing device from which the user accesses files
• Changes to cached copies are replicated back to the server(s)
• When a user access a cached file copy on any device, it’s compared to the server copy and a refresh is performed if needed

*

Systems Architecture, Seventh Edition

Summary

• File management systems
• Directory content and structure
• Storage allocation
• File manipulation
• Access controls
• File migration, backup, and recovery
• Storage consolidation

*