HARDWARE PROJECT 1

Chapter 13: Selected Storage Systems and Interfaces

Overview

Storing data is one thing; retrieving data is everything.

—Fred Moore, President

Horison Information Strategies, 2003

Not many of us know what to do with 1,000 20-terabyte drives—yet, that is what we have to design for in the next five to ten years.

—Jim Gray, 2005

13.1 Introduction

The world's craving for data and passion for information seem to have no upper boundary. If it is at all possible to somehow capture the digital essence of an activity, it seems we are utterly compelled to do so. It's as though the single byte that we fail to lock in our archives will be the only one that's important ten years hence.

Consider a trip to the grocery store, as an example. If you drive there, your automobile may be recording status information from its internal computer systems, while its global positioning system continually relays your location to a satellite. Once inside the store, your picture may be taken several times and recorded in a digital security system. Some of the items you place in your basket might contain embedded radio frequency identification (RFID) tags. Sensors throughout the store constantly record the location of the packages containing the tags while you move from aisle to aisle. As you check out, the purchase of each item is recorded in a computer, whereupon financial and inventory records are updated. When you hand over your "frequent buyer" or "customer loyalty" card, your purchases become associated with your personal information. Paying for your purchases with a credit or debit card generates several more transactions in various computers in the financial processing chain. Days later, some of this data may be extracted by data warehousing, data mining, or decision support systems, creating even more rows in a database table somewhere. Thus, the unremarkable activity of buying one's groceries in the twenty-first century could produce multiple megabytes of data that may persist for years. Whether mere humans can make sense of it all—and what happens if they do—are questions we won't try to address here. We will instead describe the hardware structures that help computer systems deal with this so-called information explosion.

Historically, electronic records in an enterprise were stored on fairly homogeneous, centralized disk and tape storage systems that were directly connected to a large host system. The entire collection of disk drives, tape drives, and the main CPU were under the control of a single operating system (or multiple images of the same operating system). Over the past 20 years, centralized configurations have been replaced or supplemented by myriad smaller servers that offer specialized services including email, e-commerce, end user reporting, and general applications. System management challenges grow proportionately with the number and diversity of server platforms and applications. Not the least of these challenges is enterprise storage management.

A number of storage architectures have recently been put forth to help get things under control. The main purpose of this chapter is to provide you with an overview of various important I/O and storage implementations, with particular attention given to enterprise storage implementations. You will see how these implementations are becoming systems in their own right, having architecture models that are distinct from the host systems to which they attach. We begin by discussing SCSI, one of the most important and enduring I/O interfaces.

13.2 SCSI Architecture

The Small Computer System Interface , or SCSI (pronounced "scuzzy"), was invented in 1981 by a then-premiere disk drive manufacturer, Shugart Associates, and NCR Corporation, formerly also a strong player in the small computer market. This interface was originally called SASI for Shugart Associates Standard Interface. It was so well designed that it became an ANSI standard in 1986. The ANSI committees called the new interface SCSI, thinking it better to refer to the interface in more general terms.

The original standard SCSI interface (which we now call SCSI-1) defined a set of commands, a transport protocol, and the physical connections required to link an unprecedented number of drives (seven) to a CPU at an unprecedented speed of 5 megabytes per second (MBps). The groundbreaking idea was to push intelligence into the interface to make it more or less self-managing. This freed the CPU to work on computational tasks instead of I/O tasks. In the early 1980s, most small computer systems were running at clock rates between 2 and 8.44MHz; this made the throughput of the SCSI bus seem nothing short of dazzling.

Today, SCSI is in its third generation, aptly called SCSI-3. SCSI-3 is more than an interface standard; it is an architecture, officially called the SCSI Architecture Model-3 (SAM-3). This architecture includes the "classic" parallel SCSI interface as well as three serial interfaces and one hybrid interface. We have more to say about SAM in Section 13.2.2.

Ironically, SCSI is no longer the dominant interface for small systems. It has long been supplanted in personal systems by simpler, cheaper disks. However, as of this writing, SCSI is employed in 80% of enterprise-class storage systems. Because of its dominance in this area, it is well worth understanding how it works.

13.2.1 "Classic" Parallel SCSI

Suppose someone says to you, "We just installed a new BackOffice server with three huge SCSI drives," or "My system is screaming since I upgraded to SCSI." The speaker is probably referring to a SCSI-2 or a traditional parallel disk drive system. In the 1980s, these statements would have been quite the techno-brag because of the intractability of connecting and configuring the first generation of SCSI devices. Today, not only are transfer rates a couple of orders of magnitude higher, but intelligence has been built into SCSI devices so as to virtually eliminate the vexations endured by early SCSI adopters.

Parallel SCSI disk drives support a variety of speeds ranging from 10MBps (for downward compatibility with early SCSI-2) to as much as 320MBps for Wide, Fast, and Ultra implementations of the latest SCSI devices. One of the many beauties of SCSI is that a single SCSI bus can support this range of device speeds with no need for recabling or drive replacement. (However, no one will give you any performance guarantees.) Some representative SCSI capabilities are shown in Table 13.1.

Table 13.1: A Summary of Various SCSI Capabilities

Open table as spreadsheet

SCSI Designation	Cable Pin Count	Theoretical Maximum Transfer Rate (MBps)	Maximum Number of Devices
SCSI-1	50	5	8
Fast SCSI	50	10	8
Fast and Wide	2 × 68	40	32
Ultra SCSI	2 × 68 or 50 and 68	320	16

Much of the flexibility and robustness of the SCSI parallel architecture can be attributed to the fact that SCSI devices can communicate among themselves. SCSI devices are daisy-chained (the input of one drive cabled from the output of another) along one bus. The CPU communicates only with its SCSI host adapter, issuing I/O commands when required. The CPU subsequently goes about its business while the adapter takes care of managing the input or output operation. Figure 13.1 shows this organization for a SCSI-2 system.

Figure 13.1: A SCSI-2 Configuration

"Fast" parallel SCSI cables have 50 conductors. Eight of these are used for data, 11 for various types of control. The remaining conductors are required for the electrical interface. The device selection (SEL) signal is placed on the data bus at the beginning of a transfer or command. Because there are only eight data lines, a maximum of seven devices (in addition to the host adapter) can be supported. "Fast and Wide" SCSI cables have 16-bit data buses, allowing twice as many devices to be supported at (presumably) twice the transfer rate. Some Fast and Wide SCSI systems use two 68-conductor cables, which can support twice the transfer rate and double the number of devices that can be supported by systems using only one 68-conductor cable. Table 13.2 shows the pinouts for a 50-conductor SCSI cable.

Table 13.2: SCSI D-Type Connector Pinouts

Open table as spreadsheet

Signal	D-Pin Number
Ground	1 →12
Termination power	13
12V or 5V power	14
12V or 5V (logic)	15
Ground	17 →25
Data bit 0 → Data bit 7	26 → 33
Parity bit	34
Ground	35
Motor power	36
12V or 5V power	37
Ground	39, 40
n Attention	41
Synchronization	42
n BuSY	43
n ACKnowledge	44
n reset	45
n MeSsaGe	46
n SELect	47
n C/D	48
n REQuest	49
nI/O	50

Parallel SCSI devices communicate with each other and the host adapter using an asynchronous protocol running in eight phases. Strict timings are defined for each phase. That is, if a phase has not completed within a certain number of milliseconds (depending on the speed of the bus), it is considered an error and the protocol restarts from the beginning of the current phase. The device that is sending the data is called the initiator and the destination device is called the target device. The eight phases of the SCSI protocol are described below. Figure 13.2 illustrates these phases in a state diagram.

Figure 13.2: State Diagram of Parallel SCSI Phases (Dotted Lines Show Error Conditions)

· Bus Free: Interrogate the "bus busy" (BSY) signaling line to see whether the bus is in use prior to entering the next phase; or lower the BSY signal after data transfer is complete.

· Arbitration: The initiator bids for control of the bus by placing its device ID on the bus and raising the busy signal. If two devices do this simultaneously, the one with the highest device ID wins control of the bus. The host must always have the highest device ID. The loser waits for another "Bus Free" state.

· Selection: The address of the target device is placed on the data bus, the "selection" (SEL) signal is raised, and the BSY signal is lowered. When the target device sees its own device ID on the bus with SEL raised and BSY and I/O lowered, it raises the BSY signal and stores the ID of the initiator for later use. The initiator knows that the target is ready when it sees the BSY line asserted and responds by lowering the SEL signal.

· Command: Once the target detects that the initiator has negated the SEL signal, it indicates that it is ready for a command by asserting the "ready for command" signal on the "command/data" (C/D) line, and requests the command itself by raising the REQ signal. After the initiator senses that the C/D and REQ signals are raised, it places the first command on the data bus and asserts the ACK signal. The target device will respond to the command thus sent and then raise the ACK signal to acknowledge that the command has been received. Subsequent bytes of the command, if any, are exchanged using ACK signals until all command bytes have been transferred.
At this point, the initiator and target could free the bus so that other devices can use it while the disk is being positioned under the read/write head. This allows greater concurrency, but creates more overhead, as control of the bus would have to be renegotiated before the data could be transferred to the initiator.

· Data: After the target has received the entire command, it places the bus in "data" mode by lowering the C/D signal. Depending on whether the transfer is an output from the source to the target (say, a disk write) or an input from the source to the target (such as a disk read), the "input/output" line is negated or asserted, respectively. Bytes are then placed on the bus and transferred using the same "REQ/ACK" handshake that is used during the command phase.

· Status: Once all the data has been transferred, the target places the bus back into command mode by raising the C/D signal. It then asserts the REQ signal and waits for an acknowledgment from the initiator, which tells it that the initiator is free and ready to accept a command.

· Message: When the target senses that the initiator is ready, it places the "command complete" code on the data lines and asserts the "message" line, MSG. When the initiator observes the "command complete" message, it lowers all signals on the bus, thus returning the bus to the "bus free" state.

· Reselection: In the event that a transfer was interrupted (such as when the bus is released while waiting for a disk or tape to service a request), control of the bus is renegotiated through an arbitration phase as described above. The initiator determines that it has been reselected when it sees the SEL and I/O lines asserted with the exclusive OR of its own and the ID of the target on the data lines. The protocol then resumes at the Data phase.

Synchronous SCSI data transfers work much the same way as the asynchronous method just described. The primary difference between the two is that no hand-shaking is required between the transmission of each data byte. Instead, a minimum transfer period is negotiated between the initiator and the target. Data is exchanged for the duration of the negotiated period. A REQ/ACK handshake will then take place before the next block of data will be sent.

It is easy to see why timing is so critical to the effectiveness of SCSI. Upper limits for waiting times prevent the interface from hanging when there is a device error. If this were not the case, the removal of a floppy disk from its drive might prevent access to a fixed disk because the bus could be marked busy "forever" (or at least until the system is restarted). Signal attenuation over long cable runs can cause timeouts, making the entire system slow and unreliable. Serial interfaces are much more tolerant of timing variability.

13.2.2 The SCSI Architecture Model-3

SCSI has evolved from a monolithic system consisting of a protocol, signals, and connectors into a layered interface specification, separating physical connections from transport protocols and interface commands. The new specification, called the SCSI Architecture Model-3 (SAM-3), defines these layers and how they interact with a command-level host architecture called the SCSI Primary Commands (SPC) to perform serial and parallel I/O for virtually any type of device that can be connected to a computer system. Layers communicate with adjacent layers using protocol service requests, indications, responses, and confirmations. Loosely coupled protocol stacks such as these allow the greatest flexibility in choices of interface hardware, software, and media. Technical improvements in one layer should have no effect on the operation of the other layers. The flexibility of the SAM has opened a new world of speed and adaptability for disk storage systems.

Figure 13.3 shows how the components of the SAM fit together. Although the architecture retains downward compatibility with SCSI parallel protocols and interfaces, the largest and fastest computer systems are now using serial methods. The SAM-3 serial protocols are Serial Storage Architecture ( SSA ), Serial Bus (also known as IEEE 1394 or FireWire ), Serial Attached SCSI , iSCSI, and Fibre Channel (FC). Because of the speeds of the SCSI buses and the diversity of systems that SCSI can interconnect, the "small" in "Small Computer System Interface" has become a misnomer, with variants of SCSI being used in everything from the smallest personal computer to the largest mainframe systems.

Open table as spreadsheet

SCSI Primary Commands
SCSI Parallel Interface (SPI-2, SPI-3, SPI-4, SPI-5)	SCSI RDMA Protocol (SRP, SRP-2)	Seria Alttached SCSI	iSCSI	Fibre Channel Protocol (FCP, FCP-2, FCP-3)	SSA SCSI-3 Protocol (SSA-S3P)	Serial Bus Protocol-2 (SBP-2)	Transport Protocols
					SSA-TL2
(Also known as Ultra2, Ultra3, Ultra320, and Ultra640)		(SAS, SAS 1.1, SAS 2.4)
	InfiniBand (™)		Local Area Network and Internet	Fibre Channel (FC-PH)	SSA-PH1 or SSA-PH2	IEEE 1394 (PHY)	Physical Interconnections

Figure 13.3: The SCSI Architecture Model-3

Each of the SCSI serial protocols has its own protocol stack, which conforms to the defined SCSI primary command at the top and clearly defined transport protocols and physical interface systems at the bottom. Serial protocols send data in packets (or frames). These packets consist of a group of bytes containing identifying information (the packet header), a group of data bytes (called the packet payload), and some sort of trailer delimiting the end of the packet. Error-detection coding is also included in the packet trailer in many of the SAM protocols.

We will examine a few of the more interesting SAM serial protocols in the sections that follow.

IEEE 1394

The interface system now known as IEEE 1394 had its beginnings at the Apple Computer Company when it saw a need to create a faster and more reliable bus than was provided by the parallel SCSI systems that were dominant in the late 1980s. This interface, which Apple called FireWire , today provides bus speeds of 480MBps, with greater speeds expected in the near future.

IEEE 1394 is more than a storage interface; it is a peer-to-peer storage network . Devices are equipped with intelligence that allows them to communicate with each other as well as with the host controller. This communication includes negotiation of transfer speeds and control of the bus. These functions are spread throughout the IEEE 1394 protocol layers, as shown in Figure 13.4.

Figure 13.4: The IEEE 1394 Protocol Stack

Not only does IEEE 1394 provide faster data transfer than early parallel SCSI, but it does so using a much thinner cable, with only six conductors—four for data and control, two for power. The smaller cable is cheaper and much easier to manage than 50-conductor SCSI-2 cables. Furthermore, IEEE 1394 cables can be extended about 15 feet (4.5 meters) between devices. As many as 63 devices can be daisy-chained on one bus. The IEEE 1394 connector is modular, similar in style to Game Boy connectors.

The entire system is self-configuring, which permits easy hot-plugging (plug and play) of a multitude of devices while the system is running. Hot-plugging, however, does not come without a price. The polling required to keep track of devices connected to the interface places overhead on the system, which ultimately limits its throughput. Furthermore, if a connection is busy processing a stream of isochronous data, it may not immediately acknowledge a device being plugged in during the transfer.

Devices can be plugged into extra ports on other devices, creating a tree structure as shown in Figure 13.5. For data I/O purposes, this tree structure is of limited use. Because of its support of isochronous data transfer, IEEE 1394 has gained wide acceptance in consumer electronics. It is also poised to overtake the IEEE 488 General Purpose Interface Bus for laboratory data acquisition applications as well. Because of its preoccupation with real-time data handling, it is not likely that IEEE 1394 will endeavor to replace SCSI as a high-capacity data storage interface.

Figure 13.5: An IEEE 1394 Tree Configuration, Laden with Consumer Electronics

Serial Storage Architecture

Serial Storage Architecture ( SSA ) was the first storage interface to break away from parallel connections. Although it has been superseded by other technologies, SSA was the turning point for industry thinking about storage interfaces. In the early 1990s, IBM was among the many computer manufacturers seeking a fast and reliable alternative to parallel SCSI for use in mainframe disk storage systems. IBM's engineers decided on a serial bus that would offer both compactness and low attenuation for long cable runs. It was required to provide increased throughput and downward compatibility with SCSI-2 protocols. By the end of 1992, SSA was sufficiently refined to warrant IBM proposing it as a standard to ANSI. This standard was approved in late 1996.

SSA's design supports multiple disk drives and multiple hosts in a loop configuration, as shown in Figure 13.6. A four-conductor cable consisting of two twisted pairs of copper wire (or four strands of fiber-optic cable) allows signals to be transmitted in opposite directions in the loop. Because of this redundancy, one drive or host adapter can fail and the rest of the disks will remain accessible.

Figure 13.6: A Serial Storage Architecture (SSA) Configuration

The dual loop topology of the SSA architecture also allows the base throughput to be doubled from 40MBps to 80MBps. If all nodes are functioning normally, devices can communicate with one another in full-duplex mode (data goes in both directions in the loop at the same time).

SSA devices can manage some of their own I/O. For example, in Figure 13.6, host adapter A can be reading disk 0 while host adapter B is writing to disk 3, disk 1 is sending data to a tape unit, and disk 2 is sending data to a printer, with no throughput degradation attributable to the bus itself. IBM calls this idea spatial reuse because no parts of the system have to wait for the bus if there is a clear path between the source and the target.

Because of its elegance, speed, and reliability, SSA was poised to become the dominant interconnection method for large computer systems … until Fibre Channel came along.

Fibre Channel

In 1991, engineers at the CERN ( Conseil Européen pour la Recherche Nucléaire ) (or European Organization for Nuclear Research) laboratory in Geneva, Switzerland, set out to devise a system for transporting Internet communications over fiber-optic media. They called this system Fibre Channel , using the European spelling of fiber. The following year, Hewlett-Packard, IBM, and Sun Microsystems formed a consortium to adapt Fibre Channel to disk interface systems. This group grew to become the Fibre Channel Association (FCA), which is working with ANSI to produce a refined and robust model for high-speed interfaces to storage devices. Although originally chartered to define fiber-optic interfaces, Fibre Channel protocols can be used over twisted pair and coaxial copper media as well. Fibre Channel storage systems can have any of three topologies: switched, point-to-point, or loop. The loop topology, called Fibre Channel Arbitrated Loop ( FC-AL ), is the most widely used—and least costly—of the three Fibre Channel topologies. The Fibre Channel topologies are shown in Figure 13.7.

Figure 13.7: Fibre Channel Topologies

FC-AL provides 100MBps packet transmission in one direction, with a theoretical maximum of 127 devices in the loop; 60 is considered the practical limit, however.

Notice that Figure 13.7 shows two versions of FC-AL, one with (c) and one without (b) a simple switching device called a hub . FC-AL hubs are equipped with port bypass switches that engage whenever one of the FC-AL disks fails. Without some type of port-bypassing ability, the entire loop will fail should only one disk become unusable. (Compare this with SSA.) Thus, adding a hub to the configuration introduces failover protection. Because the hub itself can become a single point of failure (although they don't often fail), redundant hubs are provided for installations requiring high system availability.

Switched Fibre Channel storage systems provide much more bandwidth than FC-AL with no practical limit to the number of devices connected to the interface (up to 224). Each drop between the switch and a node can support a 100MBps connection. Therefore, two disks can be transferring data between each other at 100MBps while the CPU is transferring data to another disk at 100MBps, and so forth. As you might expect, switched Fibre Channel configurations are more costly than loop configurations because of the more sophisticated switching components, which must be redundant to ensure continuous operation.

Fibre Channel is something of an amalgamation of data networks and storage interfaces. It has a protocol stack that fits both the SAM and the internationally accepted network protocol stacks. This protocol stack is shown in Figure 13.8. Because of the higher-level protocol mappings, a Fibre Channel storage configuration does not necessarily require a direct connection to a CPU: The Fibre Channel protocol packets can be encapsulated within a network transmission packet or passed directly as a SCSI command. Layer FC-4 handles the details.

Figure 13.8: The Fibre Channel Protocol Stack

The FC-2 layer produces the protocol packet (or frame) that contains the data or command coming from the upper levels or responses and data coming from the lower levels. This packet, shown in Figure 13.9, has a fixed size of 2,148 bytes, 36 of which are delimiting, routing, and error-control bytes.

Figure 13.9: The Fibre Channel Protocol Packet

The FC-AL loop initializes itself when it is powered up. At that time, participating devices announce themselves, negotiate device (or port) numbers, and select a master device. Data transmissions take place through packet exchanges.

FC-AL is a point-to-point protocol, in some ways similar to SCSI. Only two nodes, the initiator and the responder, can use the bus at a time. When an initiator wants to use the bus, it places a special signal called ARB(x) on the bus. This means that device x wishes to arbitrate for control of the bus. If no other device has control of the bus, each node in the loop forwards the ARB(x) to its next upstream neighbor until the packet eventually gets back to the initiator. When the initiator sees its ARB(x) unchanged on the bus, it knows that it has won control.

If another device has control of the loop, the ARB(x) packet will be changed to an ARB(F0) before it gets back to the initiator. The initiator then tries again. If two devices attempt to get control of the bus at the same instant, the one with the highest node number wins and the other tries again later.

The initiator claims control of the bus by opening a connection with a responder. This is done by sending an OPN(yy) (for full-duplex) or OPN(yx) (for half-duplex) command. Upon receiving the OPN(??) command, the responder enters the "ready" state and notifies the initiator by sending the "receiver ready" (R_RDY) command to the initiator. Once the data transfer is complete, the initiator issues a "close" command (CLS) to relinquish control of the loop.

The specifics of the data transfer protocol depend on what class of service is being used in the loop or fabric. Some classes require that packets be acknowledged (for maximum accuracy) and some do not (for maximum speed).

At this writing, there are five classes of service defined for Fibre Channel data transfers. Not all of these classes of service have been implemented in real products. Furthermore, some classes of service can be intermixed if there is sufficient bandwidth available. Some implementations allow Class 2 and Class 3 frames to be transmitted when the loop or channel is not being used for Class 1 traffic. Table 13.3 summarizes the various classes of service presently defined for Fibre Channel.

Table 13.3: Fibre Channel Classes of Service

Open table as spreadsheet

Class	Description
1	Dedicated connection withac knowledgment of packets. Not supported by many vendors because of the complexity of connection management
2	Similar to Class 1 except it does not require dedicated connections. Packets may be delivered out of sequence when they are routed through different paths in the network. Class 2 is suitable for low-traffic, infrequent-burst installations.
3	Connectionless unacknowledged delivery. Packet delivery and sequencingare managed by upper-level protocols. In small networks with ample bandwidth, delivery is usually reliable. Well-suited for FC-AL because of temporary paths negotiated by the protocol.
4	Virtual circuits carved out of the full bandwidth of the network. For example, a 100MBps network could support one 75MBps and one 25MBps connection. Each of these virtual circuits would permit differentclasses of service. In 2002, no commercial Class 4 products had yet been brought to market.
5	Multicasting from one source with acknowledgment delivery to another source. Useful for video or audio broadcasting. To prevent flooding of the broadcasting node (as would happen using Class 3connections for broadcasting), a separate node would be placed on the network to manage the broadcast acknowledgments. As of 2002, no Class 6 implementations had been brought to market.

13.3 Internet SCSI

Along with its superb performance and expansion capabilities, Fibre Channel comes with major drawbacks: Its hardware components are costly, and the Fibre Channel protocol presents a formidable learning curve. These factors, along with the continuous improvements taking place in less expensive technologies, are providing fertile ground for the growth of several alternatives for high-performance enterprise storage. One of the most widely heralded is Internet SCSI (iSCSI), which capitalizes on well-understood Internet and LAN protocols to provide fast, reliable transport services for SCSI commands and data.

The general idea behind iSCSI is to replace the SCSI bus with the Internet, as shown in Figure 13.10. Although the concept is simple, the protocol overhead is substantial. When a host sends data to an iSCSI disk array, the SCSI data is encapsulated as an iSCSI payload, which in turn is a payload for TCP. The TCP packet is placed inside one or more IP packets, which itself is a payload in one or more gigabit Ethernet frames, as shown in Figure 13.11. The Ethernet frames are carried across the network (this trip could be measured in meters or kilometers) to an Ethernet interface on the disk array. The payloads are extracted as the packet works its way up the protocol stack, until it is written to a disk (at last). It's important to keep in mind that a data transfer may span many Ethernet frames because of the limitations of TCP, IP, and various hardware components along the way.

Figure 13.10: Replacing the SCSI Bus with the Internet a) Traditional Parallel SCSI b) The Protocol Stack of iSCSI

Figure 13.11: Internet SCSI Protocol Data Unit (PDU) Encapsulation

Unlike Fibre Channel, iSCSI has no distance limitations. Theoretically, using iSCSI, you could save a file to a disk drive that appears as if it is local storage to you, but the file could end up actually being stored thousands of miles away. This idea ignores the latency characteristic of long-distance file transfers—that a user can't help but notice. To provide tolerable performance, iSCSI requires the fastest possible network connections (at this writing, 10 gigabit Ethernet is recommended) and hardware-based TCP processors called TCP offload engines (TOEs).

The Internet presents additional challenges of security and transmission integrity that can be largely ignored in an isolated Fibre Channel installation. However, these issues loom large in iSCSI. iSCSI security measures include transmission encryption and firewalls. Transmission integrity is provided at both the outer protocol levels and inside the iSCSI payload itself, which is protected by a 32-bit CRC. Defective packets are retransmitted unless the TCP or IP session fails, in which case the connection is terminated and reestablished.

An organization does not necessarily have to choose between the exclusive use of iSCSI and Fibre Channel. The two technologies can be combined so that one complements the other. Fibre Channel is best suited for high-throughput applications, and iSCSI for larger pools of less frequently used data. A good use for iSCSI is to provide a cost-effective remote mirror site for a high-availability Fibre Channel installation.

13.4 Storage Area Networks

Fast network connectivity such as provided by Fibre Channel and 10 gigabit Ethernet has enabled construction of dedicated networks built specifically for storage access and management. These networks are called storage area networks (SANs). SANs logically extend local storage buses, making collections of storage devices accessible to all computer platforms—small, medium, and large. Storage devices can be collocated with the hosts or they can be miles away serving as "hot" backups for a primary processing site.

SANs offer leaner and faster access to large amounts of storage than can be provided by the network attached storage (NAS) model. In a typical NAS system, all file accesses must pass through a particular file server, incurring all of the protocol overhead and traffic congestion associated with the network. The disk access protocols (SCSI Architecture Model-3 commands) are embedded within the network packets, giving two layers of protocol overhead and two iterations of packet assembly/disassembly.

SANs, sometimes called "the networks behind the network," are isolated from ordinary network traffic. Fibre Channel storage networks (either switched or FC-AL) are potentially much faster than NAS systems because they have only one protocol stack to traverse. They therefore bypass traditional file servers, which can throttle network traffic. NAS and SAN configurations are compared in Figures 13.12 and 13.13.

Figure 13.12: Network Attached Storage

Figure 13.13: A Storage Area Network (SAN)

Because Fibre Channel SANs are independent of any particular network protocols (such as Ethernet) or proprietary host attachments, they are accessible through the SAM upper-level protocols by any platform that can be configured to recognize the SAN storage devices. Even in the most complex SANs, storage management is greatly simplified because all storage is on a single SAN (as opposed to sundry file servers and disk arrays). Data can be vaulted at remote sites through electronic transfer or backed up to tape without interfering with networkor host operations. Because of their speed, flexibility, and robustness, SANs are becoming the first choice for providing high-availability, multiterabyte storage to large user communities.

13.5 Other I/O Connections

A number of I/O architectures lie outside the realm of the SCSI-3 architecture model but can interface with it to some degree. The most popular of these is the AT Attachment (ATA) used in most low-end computers. Others, designed for computer architectures apart from the Intel paradigm, have found wide application on various platform types. We describe a few of the more popular I/O connections in the sections that follow.

13.5.1 Parallel Buses: XT to ATA

The first IBM PCs were supported by an 8-bit bus called the PC/XT bus. This bus was accepted by the IEEE and renamed the Industry Standard Architecture (ISA) bus. It originally operated at 2.38MBps, and it required two cycles to access a 16-bit memory address because of its narrow width. Because the XT ran at 4.77MHz, the XT bus offered adequate performance. With the introduction of the PC/AT ("AT" for Advanced Technology) with its faster 80286 processor, it was obvious that an 8-bit bus would no longer be useful. The immediate solution was to widen the bus to 16 data lines, increase its clock rate to 8MHz, and call it an "AT bus." It wasn't long, however, before the new AT bus became a serious system bottleneck as microprocessor speeds began exceeding 25MHz.

Several solutions to this problem have been marketed over the years. The most enduring of these is an incarnation of the AT bus—with several variations—known as AT Attachment (ATA), ATAPI , Fast ATA , and EIDE. The latter abbreviation stands for Enhanced Integrated Drive Electronics, so-called because much of the controlling function that would normally be placed in a disk drive interface card was moved into the control circuits of the disk drive itself. The ATA offers downward compatibility with 16-bit AT interface cards, while permitting 32-bit interfaces for disk drives and other devices. No external devices can be directly connected to an ATA bus. The number of internal devices is limited to four. Depending on whether programmed I/O or DMA I/O is used, the ATA bus can support 22MBps or 16.7MBps transfer rates with a theoretical maximum of 100MBps. Ultra ATA provides burst rate transfers of 133MBps. At these speeds, ATA provides one of the most favorable cost-performance ratios for small system buses in the market today.

13.5.2 Serial ATA and Serial Attached SCSI

Notwithstanding its satisfactory performance and low cost, ATA is starting to fade from the small system scene. As processor speeds increase, even Ultra ATA starts to become a bottleneck. Moreover, faster processors generate a great deal of heat, which must be moved away from the processor and other sensitive components. Anything that impedes airflow inside the main system housing is problematic, and the two-inch flat cabling of parallel ATA is certainly no help. With this in mind, the next-generation ATA interface was designed as a serial interface. The serial ATA , or SATA , interface supports much faster transfer rates than parallel attachments can reliably provide, and it requires only seven conductors (four for data, three for grounding) that fit nicely in a quarter-inch cable.

Besides having thinner cabling, the many attractive features of SATA include:

· Faster data transfer than parallel ATA: 300MBps versus 133MBps (burst rate); faster SATA speeds are expected in the near future

· Lower voltage: 500mV versus 3.0 or 5.0V

· Longer cables: lm versus 0.5m

· Software compatibility with parallel ATA—no changes to drivers, BIOS, or operating systems required

· Enhanced error checking: 32-bit CRC for all bits, as opposed to data—only CRC for parallel Ultra ATA

· Point-to-point configuration, as opposed to master-slave, enables various devices along the interface to pass data concurrently

Many of these improvements to ATA have also been carried over to a serial version of SCSI called serial attached SCSI , or SAS . The plugs and cabling of SAS are identical to SATA, and for systems that support both ATA and SCSI, the devices distinguish themselves to the host at power-up time. SAS drives connect through a backplane bus that moves data at rates up to 300MBps (with faster speeds planned). SAS is hugely scalable, with more than 16,000 devices theoretically possible within one domain. With all these advantages, it is clear that it is only a matter of time until SAS and SATA drives completely replace their parallel counterparts.

13.5.3 Peripheral Component Interconnect

By 1992, the AT bus had become the major inhibiting factor with regard to over-all small system performance. Fearing that the AT bus had reached the end of its useful life, Intel sponsored an industry group charged with devising a faster and more flexible I/O bus for small systems. The result of their efforts is the Peripheral Component Interconnect ( PCI ).

The PCI bus is an extension to the system data bus, supplanting any other I/O bus on the system. PCI runs as fast as 66MHz at the full width of a CPU word. Data throughput is therefore theoretically 264MBps for a 32-bit CPU (66MHz × (32 bits ÷ 8 bits/byte) = 264MBps). For a 64-bit bus running at 66MHz, the maximum transfer rate is 528MBps. Although PCI connects to the system bus, it can autonomously negotiate bus speeds and data transfers without CPU intervention. PCI is fast and flexible. Versions of PCI are used in small home computers as well as large, high-performance systems that support data acquisition and scientific research.

13.5.4 A Serial Interface: USB

The Universal Serial Bus (USB) isn't really a bus, but it is universal. USB is a serial peripheral interface that—in one form or another—is provided on practically every electronic consumer product that is rechargeable or stores data. The family of USB specifications is under the control of a consortium of equipment manufacturers called the USB Implementers Forum (USB-IF). There have been three major releases of USB starting with USB 1.0 in 1996 to the most current, USB 3.1, in 2013. Speeds have increased from the 12 Mbps offered by USB 1.0 to 10 Gbps for USB 3.1 in Superspeed+ mode. The 280 Mbps speed provided by the ubiquitous USB 2.0 is sufficient for most everyday file transfers. USB 3.1 is better suited for bulk transfers such as disk backups and isochronous transfers such as video streaming. USB 3.1 is backward compatible with all versions to USB 2.0.

USB requires an adapter card in the host called a root hub. The root hub connects to one or more external multiport hubs that can connect directly to a large variety of peripheral devices, including video cameras and telephones. Multiport hubs can be cascaded off one another up to five deep, supporting as many as 127 devices through a single root hub. Properly equipped devices can be daisy-chained and addressed by the host through their respective unique device IDs.

The goal of USB was to make attaching peripheral devices as easy as "plugging a telephone into a wall jack," and it has achieved this goal despite the proliferation of device types that has occurred in the past decade. The joys of USB's plug-and-play capabilities are lost on those who have never known the agony of resolving conflicting interrupt request vectors and rewiring cables that required a few pin swaps. USB achieves its plug-and-play feat through publication of device driver software and a host-device protocol that associates devices with their respective drivers.

Several steps must take place when a device is plugged into a host system:

1 When a device is plugged into a USB port, the electrical state of the port is changed. The host system detects this change and dispatches a reset packet back to the device.

2 The host requests the device's Device Descriptor information. This information includes the device type, device manufacturer's code (assigned by the USB-IF), the manufacturer's product ID, and the USB specification number (e.g., 1.0, 2.0, etc.)

3 As soon as the host is able to do so, it loads the device driver that corresponds to the product ID.

4 The host may request one or more Configuration Descriptors from the device. This step is necessary whenever there is more than one configuration available at the device.

5 Once everything is known about the characteristics of the device and the appropriate driver is loaded, the host dispatches an address assignment to the device. Both the host and the device are now ready to negotiate data transfers.

USB supports four different data transfer modes, each having its respective underlying protocol:

· Control transfers—Protocol exchanges between the host and the device, such as plug-and-play, and setups for other transfer types.

· Isochronous transfers—Time-sensitive data transfers such as music and video.

· Interrupt transfers—Bursty data movement such as the ones generated by mice and keyboards.

· Bulk transfers—Transfers between the host and bulk devices such as flash drives, cameras, and scanners.

USB cables require only four conductors: two for data transfer, one for power (+5V), and one for ground. USB 3.0 augments these four with six more: four dedicated for signaling, a signal ground, and one for managing USB on-the-go. USB On-the-Go (USB OTG), available since USB 1.1, allows a device to act as both a host and a slave device within the same connection. Tablet computers commonly utilize USB OTG, because they can be hosts to external devices like keyboards, or slave devices to desktop computers when files are being transferred.

Portable device manufacture were quick to exploit the 5V AT 5000mA power readily available at ubiquitous USB 2.0 ports, and they soon became charging stations for every type of portable device. (USB 3.0 supplies 900mA.) In response, the USB-IF published its Battery Charging Specification in 2009. The Forum observed, "USB has evolved from a data interface capable of supplying limited power to a primary provider of power with a data interface." This specification includes a plug-and-play feature to determine the optimal means by which to charge the attached device.

The main objection to USB 1.0 was its slow data transfer rate, so computer manufacturers were slow to adopt it for anything other than keyboards and mice. Since then, data transfer rates have been steadily climbing to 10Gbps for USB 3.1. The evolution of these data rates is shown in Table 13.4.

Table 13.4: Data Rates of Several USB Versions

Open table as spreadsheet

USB Version	Year	Maximum Speed
1.0	1996	12Mbit/s
2.0	2000	480Mbit/s
3.0	2008	5Gbit/s
3.1	2013	10Gbit/s

USB is arguably the most important and successful computer interface. No other type of connectivity has found its way into, such a wide variety of device types, from the smallest MP3 players, to smartphones, to file servers. It has achieved this penetration through its performance and ease of use. It is also a shining example of what can be achieved through standardization and cooperation within the equipment manufacturing industry.

13.6 Cloud Storage

Cloud storage builds on the Cloud computing idea mentioned in Chapters 1 and 9. Cloud storage provides a scalable data storage platform that is accessible via the Internet. Similar to Cloud computing, the idea behind Cloud storage is that one pays only for the storage one uses. Capacity is elastic: It can be allocated and deallocated on demand. The servers are configured in redundant clusters to provide failover protection and a scalable architecture.

Cloud storage capabilities vary considerably according to the ways in which it will be used by its customer. Consumer-grade storage provides a convenient platform that subscribers may access from anywhere in the world. Several well-known providers in this area include Amazon Cloud Drive, Apple iCloud, Drop-box, Microsoft SkyDrive, and CX, to name only a few. As of late 2013, prices are between $0.04 and $0.12 per gigabyte per month, depending on features. Most of these providers offer a small amount of storage space for free, only charging after certain thresholds are reached.

Enterprise-class Cloud storage bills itself as a platform suitable for storing an organization's most precious asset: its data. This data must be accessible when needed and must be protected from unauthorized access. The enterprise must be able to control who has access to its data at any given time. Unlike consumer-grade Cloud services, enterprise-grade Cloud storage must meet certain agreed-upon performance requirements. The providers' fees are based on the service parameters and the amount of data stored. For example, one major provider advertised its product with a tenfold price difference between the lowest- and highest-level performance specifications. Because of the wide price variations caused by these performance parameters, service-level agreements (SLAs) can be put in place to make sure the purchaser gets his money's worth. SLAs state specific monetary penalties that the Cloud storage provider incurs when the performance parameters are not met. These parameters typically include (among others) the following:

· Availability—Usually stated in terms of percentage of uptime based on 24-hour days in a service month. This category also includes disaster recovery considerations such as the number of disaster drills conducted in a year and the amount of time required to restore service to full capacity.

· Reliability—Concerns itself with the number of read and write errors during a service month.

· Responsiveness—Typically measured in average seconds per transaction. This metric might also be qualified as to peak- and nonpeak-period response times.

· Manageability—Determines to what extent the service consumer can control the configuration and allocation of the storage elements. How difficult is it to expand or contract the amount of storage utilized?

· Security—States the types of controls put in place by the Cloud provider and consumer. SLAs can include penalties for data breaches; however, they rarely cover the actual costs of the breach.

In comparing Cloud storage providers, total cost of ownership must be determined. Providers' fees might not be limited to simply costs per gigabyte of storage. Separate charges could be assessed for number of I/O operations per month, bandwidth consumed, and technical support services, as well as "transition fees" that are incurred just for moving data into the providers' Cloud infrastructure.

Nearly every major technology company has some sort of enterprise Cloud storage offering. As of late 2013, the leaders are Amazon's Simple Storage Service (S3), Google, HP, and Microsoft. Even at their highest levels, the prices charged by these companies are a fraction of the total cost of ownership of a data storage facility. Thus, the costs are quite tempting to CIOs under continued pres-sure to deliver more services for less money.

As with anything, there are downsides to moving to Cloud storage. The greatest of all is that storing one's critical data in the Cloud is fraught with risk. First, there's the risk of availability. Handing over control of one's data infrastructure to an outside company means entering into contracts that include SLAs that must be monitored and enforced. Then there's the risk of the outside company going insolvent. The greatest impediment of all concerns security: It is simply not possible to provide the same security in the Cloud as in a fortified, company-controlled data center. Barriers presented by various government and financial regulations, such as HIPAA and Sarbanes-Oxley, are formidable. Surely, companies will be using the Cloud for storage, but will likely be doing so in small ways for the foreseeable future.

Chapter Summary

This chapter has outlined some popular I/O architectures suitable for large and small systems. SCSI-2, ATA, SATA, EIDE, PCI, USB, and IEEE 1394 are suitable for small systems. Fibre Channel and some of the SAM-3 protocols were designed for large, high-capacity systems. The SCSI Architecture Model-3 has defined numerous high-speed interfaces. Aspects of SCSI Architecture Model-3 overlap into the area of data communications because computers and storage systems continue to become more interconnected. For ease of reference, we have provided a summary of the storage interconnections discussed in this chapter in Table 13.5.

Table 13.5: A Summary of Various I/O Interfaces

Open table as spreadsheet

Interface	Max. Cable Length Between Devices	Maximum Data Rate	Maximum Devices per Controller
ATA	0.9m	133MBps	4
FC-AL	Copper: 50m (165ft) Fiber: 10km (6mi)	25MBps 100MBps	127
IEEE 1394	4.5m (15ft)	480MBps	63
SCSI	12m	320MBps	16
Serial ATA	1m (3ft)	300MBps	15
Serial SCSI	6m (18ft)	300MBps	16,256 (with expanders)
SSA	Copper: 20m (66ft) Fiber: 680m (0.4mi)	40MBps	129
USB 3.1	5m (16.5ft)	10GBps	127

Fibre Channel is one of the fastest interface protocols, and it is the first choice for deployment in server farms. However, other protocols are on the horizon, including iSCSI and SATA. It is certain that the industry is replacing parallel interfaces with serial interfaces. This change is driven by the need for speed, the general compatibility of serial protocols with any number of physical interconnection methods, and—in the case of SATA—the need to control heat inside the CPU cabinet.

A new growth industry is emerging around the concepts of "managed storage" and "storage services," where third parties take care of short- and long-term disk storage management for client companies. One can expect that this area of outsourced services will continue to grow, bringing with it many new ideas, protocols, and architectures to include trustworthy Cloud storage.

Further Reading

Because SCSI has been around for such a long time, it is the topic of numerous books, including those by Schmidt (1999) and Field and Ridge (1999). Field and Ridge's SCSI book is noteworthy for its readability. An excellent introduction to SAN and NAS systems is written by Spalding (2003). Tate et al. (2005) contains a good introduction and detailed information about specific products that will improve your understanding of the technology. The books by Clark (1999) and Thornburgh (1999) are very good introductions to Fibre Channel SANs. No-hype introductions to Cloud storage can be found in the papers by Abadi (2009) and Buyya (2009). Goldner (2003) provides a concise discussion of iSCSI. The technical details of iSCSI can be found in Internet RFC 3720 (www.ietf.org). The SCSI Architecture Model-3 is nicely explained by Reidel and Goldner (2003). A great deal of storage and interface information can be found on the InterNational Committee on Information Technology Standards (INCITS) websites: INCITS T10 working group (www.t10.org) is the oversight group for SCSI, T11 (www t11.org) deals with Fibre Channel and HiPPI, and T13 (www.t13.org) concerns itself with ATA. Other sources of technical information include the SCSI Trade Association (www.scsita.org), the USB Implementers Forum (www.usb.org), the Storage Networking Industry Association (www.snia.org), and the Serial ATA International Organization (www.serialata.org). Good sites for storage news include www.byteandswitch.com, www.wwpi.com (Computer Technology Review), and www.storagemagazine.techtarget.com.

Axelson has published an entire series of detailed books on the subject of the USB interface. Her USB Complete (2009) is an ideal starting point in the series. It includes a clear and thorough description of USB architecture and protocols. Code samples are provided to aid the reader in interfacing with various types of USB devices. A wealth of information can also be found on the official USB website: www.usb.org.

Intended as a graduate text, Hill et al. (2013) in their Guide to Cloud Computing succinctly describe Cloud architectures, including many examples from the major providers. The chapter on data in the Cloud is outstanding in its discussion of the ways in which Cloud data architectures differ from traditional architectures. The authors' presentation of various trade-offs involved with Cloud storage is especially noteworthy. Practitioners may find Erl et al. (2013) useful in its focus on business issues including delivery models, governance, and economics of Cloud. In a similar work, Schultz (2011) describes Cloud storage in the context of the storage hierarchy as well as the computing services hierarchy. The IEEE maintains an educational Cloud computing portal at www.cloudcomputing.ieee.org.