asign 6

CHAPTER 16

VITAL RECORDS RECOVERY

Covering Your Assets

Every vital organization owes its birth and life

to an exciting and daring idea.

—James B. Conant

INTRODUCTION

What are your personal vital records? Are they your car title, your home’s deed, a marriage license, or even a divorce decree? Whatever they are, you spend a lot of time and effort to be sure they are safe because you know they may be difficult and time consuming to re-create later. The same holds true for your company’s business records. They need a well-thought-out emergency management program if they are to be there when you need them.

Throughout this chapter, the references to documents and records apply to information stored on any media, including paper, magnetic, optic, or microfilm. There may be a bit of difference in how they are stored, but the issues for their handling and management are essentially the same.

This chapter focuses more on mitigating actions than on recovery, as that will address most of the emergency situations that you will encounter. Also included are recovery actions for small, contained records damage. There is always a chance that an emergency will overpower the best defenses and a recovery action will be needed. Few facilities are staffed or equipped for a large-scale recovery. Your best plan is to prearrange for a professional recovery service to come onsite to assist. Such a service is also a resource for designing your records mitigation processes. Remember that in a wide-area disaster, an outside service may already be engaged, so be prepared to take the initial preservation actions on your own.

A professional storage facility can be a safe and secure place for storage of documents not needed to run the business day-to-day and can also be a good source of information. Companies that provide this service include:

Fireproof Records Center— www.fireproof.com

Iron Mountain— www.ironmountain.com

Archive America— www.archiveamerica.com

The whole point of storing your vital records is that they will be available when you need them. If they are not accessible, then why are you spending all this time and money to store them? Every company has a set of records that it must safeguard for future reference. These records might be contracts, customer lists, or personnel files. Vital records can encompass just about anything.

But typically, vital records refer to documents that your company must retain to comply with legal requirements. This could be accounting records to support tax reporting, hazardous waste disposal forms, or even quality verification records to defend against future product liability lawsuits. Other types of vital records might be engineering plans and drawings, product specifications, trade secrets, and computer database backup tapes.

Vital records can be stored on many different forms of media. They can be on paper, microfiche, CDs, data backup tapes, or removable disks. On which media are your company’s vital information stored? The answer is probably all of these. Where are they stored? All over your facility! They are squirreled away in closets, under desks, on CDs, and sometimes at employees’ homes. Even the vital records that are properly situated in routine storage—how well are they climate controlled for preservation? If they are stored off-site, who ensures they are well cared for? Will they be accessible and readable when you need them?

Whatever the media your vital records are stored on and wherever they are stored, you must have a plan for safeguarding them and recovering them in the event of a disaster. Each type of media requires its own recovery strategy. Each type of document has its own level of recovery urgency.

Our goal is to safeguard these documents, whatever media they are stored on. In an emergency, your best plan is to call in a records recovery company that you previously contracted with on an as-needed basis. In an emergency, you will not have the leisure time to shop around for the best service and the best price. Every hour counts.

Our plan reviews the different types of primary vital records storage media, action steps you can take to safeguard them, and steps you can take to recover them yourself. Even if you have a company lined up for an emergency, they may be busy with another customer when you call if this is a wide-area emergency (such as a flood or earthquake).

A company’s records retention plan should address destroying out-of-date documents, explain how to store records, and detail how to identify each container. If we uncover out-of-date or unmarked documents, we can use the records retention plan as authority to get the records custodians to clean up the storage areas. It is also a chance to educate people about the records retention and storage standards. This information will be very valuable as you work to build your plan.

If your company does not have a written records management and retention program, you may need to write one. It will make your recovery planning a bit easier.

VITAL RECORDS INVENTORY

By now you know that the first step is to make an assessment of what you are going to protect. This will tell you how big the issue is. In the case of vital records, you need to know at a minimum four essential things. Refer to Form 16-1 on the CD-ROM. The inventory, also known as a shelf list, can be combined with the risk assessment spreadsheets. Other information elements may also be useful, such as the document’s expiration date, but you want to keep this plan basic:

Records. What are these documents about? Are they customer records with credit card numbers, personnel files with legally protected information, or legal documents referring to lawsuits and court actions? The information content of a record will help determine its recovery priority.

Media. Is this information stored on paper, magnetic tape, CD, microfilm? This will tell you what its storage conditions should be to protect its readability.

Originating Department. This helps you track down someone who may know something about this document to properly prioritize it in the event of a recovery.

Location. Vital records turn up in the strangest places, so you need to know where they are all stored. Otherwise, you may lose those vital first few hours after an emergency and the records will be damaged or unrecoverable.

Locate Your Records

The location for storing your records is extremely important. If most of your vital records are stored far away at some distant corporate headquarters, then that saves us from a lot of mitigation and recovery actions. Mitigation and recovery will be the responsibility of people at headquarters. You can then focus your efforts on ensuring a safe delivery to them for storage.

The best place to begin your vital records inventory is with your company’s records retention plan. This will detail what types of records you are expected to keep and how long each type of record must be maintained. If you are lucky, then most of these records are stored in the same place, or in only a few places. The key thing is that you know where they are. Ask your records custodian for a copy of the records inventory or a listing of what types of documents are stored at which locations. Your next step is to visit these file rooms and see how much volume is involved. Vital records tend to be bulky collections, so expect to see a lot of boxes.

Companies that keep records in top shape should be admired. Most companies probably have outdated records lingering everywhere. The concern here is not housekeeping, but rather that excess records distract you from dealing with the truly vital records. Once the records custodian sees that you are looking at the records, he or she should wake up and purge the storage areas of outdated documents. Just like cleaning out the garage or seeing the dentist, some people won’t do what they should until they must.

It is time well spent to speak to the originators of these records. They can explain to you what is vital and why. Add to your documents inventory the retention period for each document type. Try to uncover any other vital records storage sites not listed, such as interim storage sites, various offices where vital records are stored for office use, and any other records storage. Most departments keep their own cache of records regardless of the need to safeguard or environmentally protect them. Now is not the time to fight that battle. Some of these are in boxes under a desk. Some are in the bottom of coat closets. Some are even stored in people’s homes, as if that would be any safer. As you work with each department, find where these records are stored and add them to your records inventory.

Make a list of the locations and the types of media stored in each. Knowing which type of media is stored there helps you to quickly form a containment and recovery plan based on whether the room suffered water damage, smoke damage from a fire in an adjacent room, deep-freezing due to loss of power in the winter, etc. Each type of media has its own preferred storage conditions to protect readability. Paper can tolerate freezing so long as it is not wet. Airborne particles and pollution can damage microfiche. Extreme heat or cold can damage magnetic media such as tapes, diskettes, and fixed disk cartridges.

So now with the records inventory list in hand of the types of media you need to protect, the quantities of material and the locations, we can begin to figure out how to protect these records in an emergency.

Prioritize Your Records

The biggest time saver you can do for your vital records disaster recovery plan is to classify your records according to how valuable they are to your business. This single action will help everyone involved to know which records are to be kept safest or to be recovered first. Skip this step, and valuable time may be wasted on low-value records.

Record priority can be determined by legal requirements. It may be based on the cost to reproduce the same information from other sources. It can be determined by who the originating department is. Select a system that suits your business. Be sure you understand the legal retention period for all document types. If in doubt, consult a lawyer. Do not guess!

Once priority has been established, tag every record or record container. Priority tags should follow a simple color code to speed the containment effort. Consider using 1-inch-square stickers with different colors to indicate priority, such as red for top priority, black for do-not-recover records that have copies at a backup site, etc. Post the color code explanation on the walls and be sure it is documented in your plan. Color coding is especially helpful when there is a fire and someone other than the company employees (fire/police) is performing the salvage operation.

Next, ensure that all your top-priority records are stored in the safest locations. This might be in a fireproof cabinet or in special moisture-proof containers. Typically, the safest place for your documents is on the middle shelf—midway between the floor (moisture, vermin, etc.) and the ceiling (sprinkler heads).

With all the records marked with their recovery priority, make up a floor plan for each storage site indicating the location of your top-priority records. Keep this plan posted in all copies of your recovery plan books. The facility security office should have this also for immediate action during a late-night disaster.

Another classification is by originating department. A visual identification tag is needed so you know which records belong to Human Resources, Finance, Legal, etc. Refer to your company’s records retention plan for your standard marking information block. As these documents are recovered, be sure the originating departments are involved in the effort. They may be reluctant to have others perusing their stored documents for reasons of legal or privacy issues.

You might mark or stamp the originating department’s name in the recovery priority colored tab.

RECORDS TRANSPORTATION

Just as important as your storage process is how your records are handled during transportation. Do you have an off-site storage facility? Most companies have off-site storage for backup computer media. Imagine how valuable this information would be to an outsider. They can’t hack through your network defenses but could they easily snatch your tapes while en route to or from the storage room?

Just because records are traveling to somewhere, don’t let your guard down about their security or environmental controls. Once outside the cozy confines of your storage room, they are susceptible to the ravages of all sorts of environmental dangers. Their security must be safeguarded just as well as when they are locked in your storage room. Their environmental “comfort” must also be protected. Improper handling can negate all the careful handling we have used so far. Prudent actions when shipping these records will ensure that they return to you as fresh as they were when they were sent out.

Security

Security during transit is not a lot different from security in your storage room. Keep the curious and the criminal away from your documents. Safeguard the documents from environmental threats so they will be readable upon arrival. Limit access to your records by starting with locked shipping containers. If the records are of very high value, employ a courier to personally carry these records to their destination.

Shipping of vital records should be by an overnight express delivery. This minimizes the amount of time the records are exposed to security and environmental threats. Shipments should be adequately insured to cover the expense of re-creating the material and should be in as inconspicuous a container as possible. Always require a signature from the receiving party. The shipper should provide a tracking number so the progress of the shipment can be monitored. This provides valuable clues when looking for materials missing in transit.

Magnetic Recordings

If a magnetic recording is being sent to someone to review, make a copy of it and send them the copy. Never ship the original. Other things to consider when shipping magnetic media include:

Open-reel magnetic tape should be wrapped in bubble wrap or shock-absorbing material and packed snugly into containers. This will reduce their movement within the container and reduce the likelihood of damage during transit. It also acts as an insulator against temperature swings.

Cassettes with a locking hub should have their hubs locked and be placed in a rigid container for shipping.

Temperatures in transit should not exceed 110 degrees Fahrenheit. The best time of the year to ship these materials is the spring and the fall.

Tapes and cassettes should be shipped in the same way that they are stored, on their edge. The weight of the media should be supported by the hub.

RECORDS RETENTION

Every company has its own records retention requirements. If your company does not have any, then either you are accumulating massive amounts of paper (creating a fire hazard, housekeeping issues, and storage charges), or you are throwing out documents that you should be retaining to meet legal requirements. Either situation is a problem.

The details of the many things that go into a records retention plan are beyond the scope of this book. An important issue is the elimination of obsolete documents. This means we will have fewer documents to protect and to recover. Some organizations like historical societies try to keep everything. That is the nature of the service they provide. But a business cannot afford to hang on to stored documents that have no value. It takes floor space to store them. It takes energy to maintain them within an acceptable environment. It takes people to move them around. Check with your lawyers, check with your accountants, and properly dispose of the excess. Some laws and regulations may require that the original documents be retained even if they are also recorded on other media.

As you implement your plan, work to identify those critical records that must be restored after a disaster. This is a very time-consuming and expensive process. But it begins with being able to quickly find these critical documents. The less clutter there is to wade through, the easier the plan will be to exercise.

The records retention plan will detail a standard way to identify documents. Know what each of the markings mean and where they are supposed to be placed. Be sure that all the high-priority records encountered are properly marked according to this standard.

When documents reach the end of their useful life, they are destroyed. When this happens, a record is made of the event so we know what happened to that document. A portion of a records retention plan will deal with how your company will record document destruction. In an emergency, documents not at the end of their useful life may be destroyed. A part of your recovery effort will be to make a list of what was destroyed. Be sure that your emergency procedure for reporting accidentally destroyed records agrees with your overall records destruction documentation policy.

MEDIA STORAGE

Most business records have a fixed useful life, usually less than 10 years. If your records collection includes documents of historical or artistic value that you must retain indefinitely, then the storage and recovery of those types of documents are beyond the scope of this book. Always consult a restoration professional for questions about storing or recovering these types of artifacts.

In general, your vital records can be reasonably stored at room temperature. This is true for records whose useful life is 10 years or less. The ideal situation is for your records storage facility to be a separate room or set of rooms, with its own air filtration, heating, air conditioning, and humidity controls. The storage environment of your records will be the major determinant of their useful life. The more controlled the environment, the longer they will be readable.

In general, your storage room should be between 62 and 68 degrees Fahrenheit with a relative humidity of 30 to 40 percent all year long. Temperature and humidity should not vary more than 10 percent from your established settings. If your room is cooler than this, allow magnetic media time to slowly warm to room temperature before use. Temperature and humidity will vary in different parts of the storage room so keep the air circulating with a strong air-handling unit.

As the seasons change, so will the weather outside. Be sure your environmental control equipment can compensate for the temperature and humidity variations of the change of seasons and those that occur between day and night.

To monitor the climate in your storage area, install a thermograph and hygrometer. They will chart the conditions inside of the room over time. Pay particular attention to how well the humidity and temperatures stay within tolerance between daylight and nighttime hours—and between winter and summer. If something occurs when no one is around, these charts will indicate when the problem began. You might discover someone is turning off the heat to your storage room over the holidays and weekends, not realizing the impact on stored magnetic media.

Many smaller companies will assign a trusted employee to take the backup tapes home as a remote storage solution. This is not a good idea, as there are security and accessibility issues and risks involved. While you would like to trust all your employees, corporate espionage and damage caused by disgruntled employees are not unheard of. You wouldn’t want to be in a position where you have to deal with performance issues with the employee who is storing your corporate data at his or her home. A private home is also not going to be as secure as a professionally managed storage facility. Accessibility could be a problem if a disaster occurs and the employee is not at home due to vacation or other reasons. Listed below are some of the different media types you may have in storage and their particular requirements:

Paper Document Storage. Paper is sensitive to humidity and to temperature. It slowly deteriorates. Rapid swings in either or both accelerate this process. If the documents are exposed to low humidity and high temperatures, they gradually dry out and become crumbly. If they are exposed to high humidity and warm temperatures, they become susceptible to mold.

To protect your most important records, store them on the middle to lower-middle shelves. This keeps them well off the floor in case of a flood or pipe leak and away from the sprinkler in case of a fire. It also keeps the records in the middle of the room’s temperature range (cooler near the floor, warmer near the ceiling).

Microfilm Storage. The first key to safeguarding your microfilm is to store it in an airtight container. This will protect it from dust, humidity, and impurities in the air. The container should be made from a noncorroding material, such as anodized aluminum or stainless steel. If your microfilm is on a roll, the reel should be made of the same material as the container. Always use lint-free gloves when touching the film.

Store your microfilm on shelving and storage racks made from noncorroding material. Special cabinets designed for protecting microfilm can be obtained from industrial equipment suppliers. If possible, do not store your microfilm in rooms containing pressboard or particleboard as these may give off fumes that will damage the film.

The storage room should have its own temperature and humidity controls. Humidity should be between 30 and 40 percent, and it should never change more than 5 percent within a given day. Large swings in temperature and humidity are damaging to microfilm. The temperature should stay around 65 degrees Fahrenheit and vary no more than 5 percent in a given day.

Fire is always a threat to a storage facility. If possible, install a gas fire suppression system. Water from sprinklers is very damaging to microfilm. If you choose to use fireproof cabinets, be sure they are rated for microfilm storage. A cabinet designed to keep paper from burning will not be able to keep damaging heat away from your microfilm. The cabinet must be able to hold the internal temperature below 150 degrees Fahrenheit.

Magnetic Media Storage. Magnetic storage media requires every bit as much care as printed documents. The useful life of a tape can be sharply reduced by improper storage or handling. Unlike paper, you cannot readily look at a tape and determine if it is still readable or not. Improper handling can result in a tape becoming unreadable.

Steps must be taken to ensure that no contaminant of any kind comes into contact with the recording media. You should never touch the magnetic surface at any time. Therefore, magnetic media, such as reel tapes, cassettes, floppy disks, etc., should only be stored and used in a very clean environment. In these rooms, smoking, eating, and drinking must be prohibited. Ideally, an air filtration system will be employed to screen out airborne contaminants.

Magnetic media are susceptible to variations in temperature and humidity. If storage and operating temperatures vary more than 15 degrees Fahrenheit, you must allow time for the media to adjust to the different conditions. Allow 4 hours for every 15 degrees Fahrenheit of temperature difference. Strong sunlight will also damage magnetic tape.

The components that make up magnetic tape will react with moisture and slowly begin a process of breaking down the chemical bonds. Carefully avoid water and moisture when tapes are exposed.

Before using a tape, inspect it for surface damage. Look for any debris on the tape (but never touch the media itself without lint-free gloves). If in doubt, clean the tape surface before use. Always return tapes promptly to their protective containers. Minimize how much tapes are handled. Ensure that any tapes being returned to service are first thoroughly bulk erased.

Magnetic tapes should never be dropped or treated roughly. When in storage, they should always be stored on end and never stored “flat” (with the reels parallel to the table). Magnetic tapes are susceptible to damage from magnetic fields. Never store tapes on or near machinery, on windowsills, or on top of electronic devices.

Magnetic media is worthless without the proper equipment required to read the media. As your storage media ages, periodically check to make sure you still have the proper equipment for reading the media. Work with your data processing hardware support team to make sure you are involved when new backup hardware is being considered. Software can also be an issue, as formats may change slightly as backup software and operating systems are upgraded. You may need to copy the old media to a new format as the technology changes.

RISK ASSESSMENT

This is a good place to conduct a risk assessment of the threats facing your vital records. Refer again to Form 16-1 (on the enclosed CD). Use the inventory list of documents to fill in the left side. Once all the documents are listed, identify the storage risks faced by these documents, based on the type of media and where they are stored.

If life were fair, you would be able to select your own storage facility. It would be a secure place without water pipes in the ceiling, with a reliable, steady environmental control system, and no external walls. The facility would be located in a place not susceptible to natural disasters, with a separate clean room for magnetic media storage. Unfortunately, vital records storage is usually on the low end of the floor space priorities, and you must compete for adequate floor space with everyone else. Vital records storage is an overhead cost and does not bring in any revenue. Therefore, it is treated like the coat closet, important to have but must fit in wherever it can.

A key part of the risk assessment is to identify the types of documents to retain and their priorities for restoration. With computers, many of these records can be duplicated from stored media—unless it was in the midst of the fire zone.

Storage risks include:

Water. This could be caused by a leaking roof, a burst water pipe, a sheared-off sprinkler head, a water leak on the floor above—any number of things. If this storage area is below ground, is it susceptible to flooding? Are the temperature and humidity automatically controlled? Never establish a records storage area in a room under a rooftop air conditioner, as they tend to collect water. Also, keep away from rooms with overhead water pipes, steam pipes, and exterior walls. Spot coolers used in storage rooms collect water and are a haven for mold and bacteria. Subfloor cooling in computer rooms also collects water.

Smoke. If there is a fire in this room, what is the potential for smoke particles to penetrate the packing crates?

Structural Problems. Does the roof leak? Is it strong enough to withstand a very heavy snowfall or an unusual downpour? Look at the ceiling. Is it discolored at any place as if moisture was collecting there or had previously leaked?

Fire. Is there adequate fire suppression? Does this include fire-suppressing gas or are you relying solely on sprinklers? Is the room clear of clutter? Is anything stored against an electrical appliance or receptacle? Are extension cords in constant use? Are the walls fireproof? Is a fire hazard on the other side of any of the walls?

Humidity. How much humidity is floating about in this room? Does it vary appreciably during the year? Is a functioning humidifier/dehumidifier in operation?

High Heat and Deep Cold. Excess heat or cold can be very damaging to stored documents—especially for magnetic media and microfiche. Is this room well insulated and climate controlled?

Wide Temperature Swings. Wide temperature swings can age your material and cause humidity variations. The best long-term storage environment for your materials is a stable one that varies in temperature no more than 5 degrees Fahrenheit.

Theft. If there is something of value in these documents, you must take steps to reduce the likelihood of theft. This can be personnel information, credit card numbers in your customer files, or any number of things. Securing the room with a lock and key is a good start.

Sabotage. Similar to theft, except in this case, they just want to destroy what you are safeguarding. Like theft, sabotage may be difficult to detect.

Insects and Rodents. No food or drinks should ever be allowed in the storage area. Look around for signs of animal or insect infestation.

Magnetic Fields. If you are storing magnetic tape or diskettes, your materials may be susceptible to damage from magnetic fields. Be sure there are none present in your storage areas and also be aware of what is going on in adjacent rooms. Over-the-weekend construction work may not be reported to you yet could damage your media. This hazard includes small magnets and magnetized tools in the storage room.

MITIGATION

Now you need to identify the mitigation actions to be taken to minimize the identified threats. Risks to your records depend on the types of media that they are stored on, how carefully they are stored, and how accessible they need to be.

Use the table in Form 16-1 (on the enclosed CD), where you have previously identified the threat to each document. Now identify the mitigation actions you will use to reduce the impact or likelihood of that threat.

Fire Control System

Smoke Alarms and Fire Detection System. Early warning is your best deterrent against losing records to a fire—and most likely is mandated by the local fire code. These alarms must not only alert anyone in the storage area of the danger, but also your facility’s security team so that the local fire department is dispatched immediately. These alarms are often used to trigger the gas fire suppression system. Special fire alarms are available that detect fires in their earliest stages. This permits quick intervention with a fire extinguisher before the sprinklers can kick in.

Sprinklers. These are necessary to contain fires and save the building. They also provide valuable time for people to escape an inferno. Most of the material stored in the records room will readily burn, so sprinklers are essential. A problem is that the tool you are using to save the room is damaging to what is stored there. Sprinklers are an inexpensive fire suppression tool and regulated by local fire code. Do not defeat the sprinklers’ action by lining your shelves with plastic. It will only allow the fire to grow larger before the plastic melts (and is likely a violation of local fire codes).

Gas Fire Suppression. Gas fire suppression is the best first line of defense against a fire. It can snuff out a fire before the sprinklers’ fusible link melts. Although the gas discharge may spew some particles in the air, the damage is far less than from sprinklers. A gas fire suppression system is expensive and requires a sealed room for best effect, so be sure the doors close automatically and snugly.

Fire Extinguishers. These come in several types based on the type of fire. In general, they should be of the “A” type, which is for combustible materials. Fire extinguishers must be inspected monthly. Employees should know where they are and how to use them.

Fireproof Containers. Use these for cash, checks, and vital records that cannot be replaced. After a fire, never open these containers until they are completely cooled, inside and out. If the inside is still hot, sudden exposure to fresh air may cause a flash fire. Documents protected from a fire by a fireproof container may be charred but readable.

Fire Drills. Drills should be conducted at least every 3 months, or more often if required by your local fire code.

Good Housekeeping. This minimizes the amount of rubbish in your storage areas. Rubbish accumulation is nothing more than fuel for a fire or food for vermin. Be sure it is removed daily. If possible, do not allow trash cans in the storage areas at all.

Electrical Equipment. Minimize electrical equipment in the storage areas. This is a potential source of fire and magnetic pollution.

Move all possible electrical equipment out of and away from your storage area. This reduces the possibility of a fire starting in this equipment. Also, some equipment, such as copiers, stir up paper dust.

Electrical outlets should not be overloaded as this could start a fire. Always use equipment with properly grounded plugs.

Extension cords are another potential fire source. Be sure to only use heavily insulated extension cords and never on a permanent basis.

Environmental Issues

Moisture Sensors and Alarms. These alarms alert you to the presence of moisture in your storage room. These are very useful if you have a raised floor or an area that is difficult to see such as a drain in the room. These alarms may alert you to water buildup or excess moisture due to temperature changes.

Humidifier/Dehumidifier. This device will help to keep your records storage area within the proper humidity range.

Temperature Control. Some records may be stored on media that is susceptible to temperature damage. Actually, all media are susceptible, but some, like magnetic media, have little tolerance for high or freezing temperatures. These extremes even work to degrade your paper records but not as severely. Steady temperatures will reduce the load on your humidifier/dehumidifier.

Magnetic Check. Wherever magnetic media is stored, be sure to run a periodic check of magnetic influences on the storage area. Magnetic influences are difficult to see but will degrade or damage the data stored on magnetic tapes, diskettes, cartridges, etc.

Other Issues

Secured Access. Eliminate the people problem by limiting who has access to the storage areas.

Off-Site Duplication of Key Records. If you have very critical records, one of your best solutions is to store copies of these records off-site. Then, if a crisis occurs, you will not need to labor through an expensive records recovery process. This would require, however, that you maintain the off-site storage facility to the same high standards as used in your primary records storage facility.

Pest Extermination. These creatures are not welcome in your storage area. Insects, rodents, and anything else that might want to dine on your documents must be vigorously kept away from your records.

Proper Storage

Identification tags must be attached to every container. In a crisis, these tags will be used to prioritize the records to be recovered. Records recovery is an expensive and time-consuming process. Proper identification allows everyone to focus on the most critical records first.

Any documents containing water-soluble ink should be stored on microfilm. If these documents get wet, they will probably not be recoverable.

Store your most critical records on the middle shelf. This keeps them off the floor in case of a water problem, and the records above will slow down the flow of water from a sprinkler or broken water pipe.

Store all vital records at least 4 inches above the floor. If shelving is not available, use clean pallets.

Do not store anything within 6 inches of the ceiling or lights.

Do not store anything within 18 inches of a sprinkler head. That would interfere with the sprinkler’s ability to put out a fire.

Do not store anything in contact with an electrical device or obstructing any of the air-handling ducts.

A quick note on alarms. We talked about moisture sensor alarms, excess temperature alarms, and fire alarms. Alarms are useless unless people know what they mean and what to do when they hear one. During your training exercises, let your staff hear each kind of alarm and explain what to do when it sounds. Repeat this step with every exercise! Ensure that alarms in the closed storage room can be detected and acted on during the weekends and evenings.

Security Mitigation Actions

Unfettered access can lead to theft problems, increased insect issues (people bringing in food), potential for sabotage, and, in some cases, just nosy people rifling through documents. Always secure your vital records storage areas. If they are climate controlled, then the less often the door is opened, the better.

Begin with controlled access to your vital records. This may be as simple as locking the door to the storage closet. Controlling access is important to prevent someone from these actions:

Reading Your Records. If these are trade secrets, such as customer lists, you may not want anyone to casually peruse them. Someone reading your records and copying the information can be difficult to detect. If documents are worth keeping, they are worth keeping in secured storage.

Stealing Records Is a Form of Employee Sabotage. If these records are required for regulatory compliance, a disgruntled employee could attempt to damage a company’s reputation or an executive’s job performance by removing records. This can go undetected for a long time. Sometimes companies victimized in this way find out as the documents are published online. An angry employee could also steal the records in an effort to damage your ability to prove company compliance with legal requirements or even to hide their own crimes.

Damaging Your Records Is a Variation of Employee Sabotage. Similar to stealing, someone trying to hide their own actions or trying to damage a company’s reputation could damage records. Often this is done on a wide scale rather than stealing a few select documents; the miscreant may opt for damage through arson or heavy water damage.

Rodents and Insects Mitigation

Your vital records may be very appetizing to insects and rodents. Basic housekeeping steps can minimize your exposure to these pests. Clutter, dirt, and dust should never be allowed to accumulate. Eating and drinking should never be allowed in your records storage area. Break rooms and cafeterias should be as far from the storage room as possible to reduce exposure to these pests.

These pests also like to be comfortable. They prefer high temperatures and high humidity. Keep your storage room at the optimal temperatures for storing your media; this provides a built-in defense against pests. Ensure that there are no “dead spots” in your air circulation that might create a safe haven for these creatures.

A key way to prevent these creatures from setting up housekeeping is to not invite them into the room in the first place. Doors, windows, and vents should be opened as little as possible. Seal cracks in the walls and ceiling promptly. Inspect incoming materials for signs of insects before admitting them to your storeroom. Remove packing material before entering the storeroom as that is a conduit for the spread of insects.

If an infestation is discovered, the quickest method is to bring in a professional exterminator. Rodents are easier to treat as they can be trapped. Poisons should be avoided as a contaminant to the room’s atmosphere.

Insect eradication is a tougher job. If possible, take an example to the exterminator so they can apply the proper solution. Isolate all documents around the infestation. The best solution is to carefully freeze infested paper documents and all the containers around them to kill the insects.

ACTION STEPS FOR YOUR PLAN

In an emergency, you will have an immediate need for damage containment supplies. These supplies should be purchased in advance and stored in a locked room far across the facility from the records storage area. The goal is that an emergency in the storage room will not also destroy your containment supplies. If possible, store the emergency materials on a cart for rapid deployment.

A list of the recommended supplies is found in Form 16-2 (on the enclosed CD). What you need for your site depends on your risk assessment (things likely to go wrong) and your inventory (what types of media you are protecting).

The materials needed for an emergency fall into several general categories:

General items are basic items needed to clean up a mess. Some of these materials age over time (such as flashlight batteries) and should be rotated at least annually (out of the closet and into general use, fresh batteries into the closet). Some of these items may be in regular use in departments across the facility from the storage area and a separate storage stockpile may not be necessary. Not listed here but useful will be a wide range of hand tools.

Portable equipment is the heavy tools you may need to address more severe problems. Smaller items, such as water vacuums and portable dehumidifiers, may be kept in your storage closet. Larger items, such as water pumps, may need to be obtained from the facility’s maintenance department. In addition, you must list the telephone numbers of ALL local companies that will rent trucks with freezer compartments in case you must freeze and/or ship documents for off-site recovery.

Individual equipment is the safety equipment for the recovery team. Be sure to inspect this annually and rotate out the older materials to the facility’s cleaning staff.

Drying and cleaning materials will be quickly consumed in a large emergency, so be sure you know who the local suppliers are.

Containment material: if the flood waters are rising, if the roof or wall is missing, if the fire is now out, these materials can slow the spread of damage or prevent additional damage from occurring.

Maintenance Activities

Now that you have your storage facility safeguards in place, ensure that you don’t let your guard down. Plan to make these activities a part of your normal routine:

DAILY ACTIONS

Trash emptied.

During off hours and weekends, ask your security guard to step into the room and see if it feels too hot or humid.

Check locks on windows and doors.

Look for ceiling leaks, especially after a major storm.

EVERY WEEK

Housekeeping inspection: ensure all trash is promptly removed.

Change the paper on the hygrothermograph’s plotter.

Check the corners of the room for warm, moist air circulation “dead spots.”

QUARTERLY ACTIONS

Pest control. Check sticky traps, doors, foundations, walls.

Test fire and humidity alarms.

Test water detection sensors.

Fire extinguishers inspection.

Magnetic check; also do this whenever neighboring rooms change, including the floor above and below.

Meet with local emergency officials.

Rotate supplies out of your emergency stock. Be sure that emergency recovery supplies are stored away from the vital records storage so they aren’t lost at the same time.

Be sure air filters on all equipment (such as air conditioning) are changed.

You may also want to consider hiring a records storage professional to perform an audit on your off-site storage location, the security procedures in place, and the retrieval process. Whether you perform the work yourself or use a dedicated storage company, this can help you to identify gaps in the company’s storage and retrieval process.

IMMEDIATE ACTIONS IN AN EMERGENCY

In an emergency, the first concern is the safety of your people. You must wait until the vital records areas are structurally safe to enter. Buildings are substantially weakened by fire, flood, and any major shock to their structure. In the event of a fire, you must check with the on-scene fire marshal in case the site needs to be sealed for a criminal investigation. This is where your predisaster liaison with local emergency services will pay off. An investigation may not start for days. Work with local officials to gain access to remove your undamaged records—but only do so with the permission of the proper authorities. Before entering, put on the hard hats stored with your emergency supplies. Ceilings are easily weakened in a structural emergency.

In the meantime, scramble around to line up emergency supplies to be ready to act once the go-ahead to enter is given. Contact your company security team and inform them where your records recovery operation will take place so they can assign a detail to keep the curious away from your documents. Immediately call sister companies requesting help from their records custodians. Begin setting up your damage mitigation area so recovery operations can begin as soon as the teams are ready.

When entering a damaged area begin your initial damage assessment. This is a quick walk-through to see which records are obviously damaged. Determine which vital records are damaged. Use the color coding on your containers to see the recovery priority of all damaged containers and their type of damage (heat, water, exposed to air, etc.).

Before opening any file cabinets, use your hand to feel their outside temperature. If they are still hot, allow them to cool thoroughly before opening. A fireproof cabinet prevents a fire by sealing the contents from an oxygen supply. If the contents are sufficiently hot, and you open the cabinet too soon, you will see your documents turn into a flash fire and quite possibly injure someone.

Based on your initial damage assessment, divide your helpers into teams. There is no set size on a team since each emergency is unique:

Damage Containment Team. These people focus on containing the damage. If there are now holes in the walls or ceiling, they should hang heavy-ply plastic to keep out further weather damage. If documents are strewn about on the floor that are too numerous or for whatever reason cannot be picked up, the damage containment team will locate and lay plywood to protect the documents from foot traffic.

Assessment Team. These folks will identify the records to be retrieved from the storage area based on their preestablished priority color code. They should take many photographs during all phases of the operations. Assessment pictures can be reviewed for understanding the amount of damage and may be useful to the insurance company. Pictures taken during the recovery can be used as source material for the after-action report.

Shuttle Team. These are the people who are carrying documents from the storage room to the recovery area or for transportation to the off-site storage location.

Triage Team. This team will log all documents as they are received from the damaged store room to begin tracking them through the recovery process. They will examine incoming documents and assign them to three categories: not damaged, damaged, or beyond recovery. Damaged documents will be categorized by the recovery technique to be used. They also ensure that the priority documents are addressed first. The triage team will monitor the flow of documents to the recovery team and may identify documents to send on for immediate freezing and later recovery.

The triage team will also identify those documents that are unlikely to be salvageable. They may be charred beyond recovery, or deteriorated due to water or physical damage. Note these on your recovery log, tag them and, if made of paper, freeze them for later evaluation.

Some teams may use a color code for documents to indicate their disposition. Take care not to confuse these with the color codes assigned to the documents in normal storage. Use whatever color system suits your situation, but a suggested one is this:

Green for undamaged documents: send these on to storage.

Red for priority documents: to be recovered first.

Yellow for lower-priority documents: to be frozen and reviewed for potential recovery later.

Black for documents beyond hope of recovery.

What If the Emergency Missed Me?

In many instances, your building may be damaged but your records are intact. In those cases, you must decide if they are safe where they are or if they must be evacuated to a safer place. Safety involves both physical security and environmental security. If the air conditioning system still works, plan to stay where you are. Work to return the storage area to its proper environment to inhibit the growth of mold.

Before the emergency, you had a secure building. There were secure walls, locks on the doors, guards at the front door, and other security measures. Once a major structural emergency is contained, there may be holes in the roof or walls, strangers wandering about, and less than adequate physical security for your records. In addition, power may not be functioning in the building until major repairs are completed. You must decide to stay or go.

If you stay, and if your temperature and humidity control equipment are not working, then it is just a matter of time until problems begin. Insects may begin to creep in, mold begins to grow, and your records begin to deteriorate. Still, it is a major effort to pack everything up and move out. Packing, transportation, reestablishing a controlled atmosphere at the new site, and then moving everything back later is a frighteningly difficult challenge. What to do?

The key to this question is how soon electrical service, air conditioning, heating, and humidity control can be restored to your storage areas. If your rooms are unharmed, turn off the air circulation immediately until the air has settled. This should prevent circulating smoke fumes throughout your storage areas. After the emergency has been contained, try to maintain the flow of clean, filtered air at the proper temperature and humidity levels to avoid a forced move of your records.

If you stay, ensure there are adequate air filtration, ventilation, and climate controls in your storage room. This may require the use of a large portable electrical generator and portable air-handling units. With wire runs all the way from the generator in the parking lot up to your storage area, and then with the expense and effort involved with portable air-handling units, you can quickly see what a major job this will be.

If service restoration is likely to be soon, then seal the storage area as tightly as possible and press for prompt temporary repairs to the storage area.

What If the Emergency Hit Me?

There are detailed recovery steps later in this chapter, but the issue here is that if your storage facility is unusable, you must relocate it to an off-site facility. This off-site facility must have security for your documents. It should have as much of the climate control capabilities as your old site as possible. On short notice this could be a problem, so if possible, contract with a storage company to be used on an as-needed basis. If practical, ship the documents to another company site. This will greatly simplify the security arrangements.

Once the disaster has passed and the document recovery process is underway, the new records storage room must be carefully prepared. Be sure that it is completely dry. All the old carpeting, shelving, furniture, and anything else that may harbor mold or fungus must be replaced. Walls, floors, and ceiling must be treated for mold and fungus before returning documents to this room. Be on the lookout for hidden water under tile or raised floors.

Allow fresh paint to dry for at least 2 weeks. This allows the solvents to dissipate and the airborne paint particles to settle.

When all the excitement is passed, sit down and write an after-action assessment. This is where you can recognize the people who helped through the crisis and critique how realistic your plan was. Include the photographs taken during the emergency. You should also review actual expenses incurred for future budgeting.

RECOVERY TECHNIQUES

There are many recovery processes that can be used. Most companies turn this over to a professional recovery service as they lack the expertise and equipment to do this in the face of a major emergency. Document recovery is a very delicate business that, if not properly done, will complete the destruction of your vital records. If you expect to recover your own documents, here are some of the steps to take. You should also study the finer details of document recovery from books dedicated solely to that subject. Time spent practicing before an emergency is an excellent idea.

Water Damage to Paper Records

Water is a threat to all your vital records. Just about any paper documents can be recovered from water damage (except those containing water-soluble ink, which should be microfilmed before storage) if promptly treated. Paper records begin deteriorating in as little as 3 hours. Within the first day, mold, fungus, and bacteria begin growing on paper. Recovery is basically to remove the documents from the water, and then remove the water from the documents. If the document is not to be immediately recovered, then it should be quick-frozen until it can be processed. Freezing can protect a paper document for up to 5 years.

Begin your paper recovery process by stabilizing the atmosphere in the work area to between 50 and 60 degrees Fahrenheit, with a humidity level between 25 and 35 percent. Temperatures and humidity in a room tend to vary based on how close you are to the heater or dehumidifier, so use fans to circulate the air and equalize the conditions. Remove from the room any wet things that are not the documents being treated, such as wet clothes, unneeded packing material, etc.

Review paper documents for damage. Water-soluble inks will not likely survive a good soaking. The wettest records are usually the ones that were on the lower shelves or directly under the fire sprinkler (so be sure not to store your most valuable records in either location!). Among your priority records, process the wettest ones first.

Remove all metal fasteners from the documents. This will prevent rust from forming on the fastener and then spilling over onto the document. Use plastic milk crates or similar containers to transport documents because they allow for some of the water to drain off. Never pack them more than three quarters full as the weight of the wet papers will further damage your documents. For the same reason, you should not stack books atop each other in these crates.

Wrap the documents in freezer paper before placing them in the crate, about 200 sheets at a time. Wrap books and set them in the crate with their spine toward the bottom. Always make a list of any documents you have found, their condition, and where you sent them. Mark the identity of the documents on the outside of the freezer paper.

AIR-DRYING PAPER RECORDS Air-drying is the easiest but most labor-intensive process for recovering paper documents. It is most suitable for small amounts of documents or lightly damp books. Drying documents in the open air requires a lot of space and time. After drying, the documents will never look the same and may be permanently stained by soot and water. Note these considerations:

Wet paper is easily torn. Handle every document very carefully.

Individual sheets of coated paper are very difficult to air-dry. Send them to a freeze-dry facility. If they are to be air-dried, carefully separate them immediately. Books printed on coated paper should never be air-dried. They should be frozen immediately and sent for professional recovery.

Books suffer the most from air-drying. Most will be distorted from the moisture and will require rebinding. Very wet books should always be freeze-dried. If you decide to air-dry books, interleave absorbent paper every few pages. Do not stress the spine. Place absorbent paper inside the front and back covers. Change the absorbent paper every several hours. Dampness will persist in the spine and the covers for quite some time, so you must check often for mold. Never return books to shelves until fully dry to reduce introduction of mold into your facility.

Air-dried documents, especially books, are susceptible to mold.

Mud can be brushed from dry documents. Trying to remove mud while the paper is still wet simply pushes the mud into the document fibers.

As you begin your recovery efforts, use the nylon fishing wire in your emergency supplies to string some drying lines. Take care where you place the wire as it is hard to see, especially in low light, because people may run into it. Separate the sheets of paper and hang them on this drying line.

In your drying room, keep temperatures lower than 70 degrees Fahrenheit and humidity below 50 percent to inhibit the growth of mold. Use fans to circulate the air to the dehumidifiers to accelerate drying. If your drying efforts are conducted outside, keep in mind that prolonged exposure to sunlight will accelerate the aging of paper.

An alternative to a drying line is to spread the documents out on tables covered with absorbent paper. Interleave sheets of paper with absorbent paper if they are very wet or in a book. Change this paper as needed, depending on how wet the documents are. Use your fans to keep the air circulating around the room to the dehumidifier.

Other recovery methods include photocopying damaged documents and discarding the original. This solution may depend on any legal requirements for maintaining the original document. Another is to use a low-heat clothing iron to gently heat the moisture from the paper.

Dried records always require more storage space when finished. Photocopy water-damaged documents if possible and keep the copy (assuming there is not a legal requirement to keep the original).

FREEZE-DRYING PAPER RECORDS Freezing is a way to stop the progress of damage to your damp paper-based documents. Those documents that cannot be recovered quickly or those which will be transported off-site for recovery should be frozen. If the quantity of documents is small, use dry ice to freeze them during transport. If the quantity is large, call in freezer trucks. Freeze documents to between 20 and −40 degrees Fahrenheit. Freeze as quickly as possible to prevent damage from the formation of ice crystals.

If a commercial recovery service is used, they will freeze your documents and possibly vacuum dry (freeze-dry) them. This process reduces stains and odors caused by smoke and also eliminates mold. Freeze-drying is a passive process and may take several weeks or more to complete. Freeze-drying is the best solution for recovering wet books. In the case of slightly damp books, this will kill any mold. In the case of very wet books, this will reduce the damage to the book in addition to killing any mold.

Wrap bundles of documents in freezer paper and place in interlocking milk cartons. Document bundles should be about 2 inches thick. The milk cartons allow for air circulation and moisture drainage. Be sure to label the bundles so you know what they are without unwrapping them. Books should be wrapped separately. Never fill the cartons more than three quarters full, as damp paper is weak and easily damaged.

When preparing books for shipment to a freeze-drying facility, support the bindings to reduce the likelihood of swelling. This will reduce the amount of rebinding required for your recovered material.

Even though your records are in a recovery facility, you must still ensure their security. Depending on the sensitivity of your data, you might want a security guard present in the drying room at all times. Now is not the time to drop your guard.

UNRECOVERABLE DOCUMENTS The destruction of any document must be carefully recorded. Be sure to clearly identify what the document was, any identifying titles or routing codes, and why (or how) it was destroyed.

Fire Damage of Paper Records

Fire damage to your records can be just as severe as water damage. Fire will char documents, cover them with soot, and make them more brittle. They may also be wet and smell of smoke. Even portions that are not burned may be darkened by heat and smoke. If you can do without the original document, make a photocopy and discard the original. Handle these documents as little as possible as they may be quite brittle and crumble in your hands.

Place every fire-damaged document on paper towels or absorbent paper. Move these documents by picking up the absorbent paper, not by touching the document itself. The absorbent paper will also pull some of the moisture out of the document.

Microfilm

Wet microfilm must be delivered to a film duplicator as soon as possible. Line containers with clean trash bags and fill them with clean cool water. Submerge the film in the water and deliver them to a professional recovery service within 48 hours. The recovery service will professionally wash the media and dry it.

Optical and Magnetic Media

Wet magnetic media should be placed in bags of cold water for transportation. The media should never be frozen. Use distilled water when rinsing magnetic materials. Tap water may contain chemicals or other materials that would dry on the media. Air-dry the magnetic media in a clean room within 48 hours. Conduct a quick check of the recovery area and ensure no magnetic sources are present, including magnetized tools.

Once magnetic storage media is dry, promptly copy it onto fresh media. Clean the read heads frequently.

Tapes. Immediately rinse dirty water and mud off magnetic tapes. Be sure to never touch the magnetic media with your bare hands. When touching the media, use lint-free gloves and handle as little as possible. Whenever possible, handle the tapes by the hubs or the reel. Air-dry in a clean room to prevent the settlement of dust and other particles on the media.

Compact Disks. Handle the CD carefully to avoid scratching. Air-dry to remove moisture.

Floppy Disks. Pack wet disks vertically in bags of cold water. Rinse thoroughly before air-drying.

CONCLUSION

Vital records protection is not difficult, but requires some thought and action before a disaster strikes to keep the damage to a minimum. The key is a good records retention policy, so that you are storing as little as possible and destroying records you no longer need.

CHAPTER 17

DATA

Your Most Irreplaceable Asset

640K ought to be enough for anybody.

—Bill Gates, cofounder of the Microsoft Corporation, 1981

INTRODUCTION

Most of what you lose in a disaster is relatively easy to replace. Buildings can be rebuilt or new offices leased, furniture is easily replaced, and even new computers can be purchased at the click of a button. What is not easy to replace is your competitive advantage, which is stored in the files and databases within your computer systems. This critical information is in accounting files, customer lists, part lists, manufacturing drawings, etc. This information is unique to your company; it is what makes your company special to your vendors and customers. It is the very essence of your company. Unlike physical assets, this information is difficult, if not impossible, to re-create once it is gone.

There are two types of risks to the infrastructure that supports your data assets: (1) physical loss due to a device failure or a disaster at your location and (2) logical loss caused by an application or user error. Physical loss is the less likely of the two, but it is potentially the most damaging. It includes incidents such as a hard disk failure, server failure, or an environmental disaster, such as a fire or flood. It can affect just a single device or your entire location. Physical loss accounts for approximately 20 percent of all incidents affecting information technology resources. In contrast, logical loss includes incidents such as application errors, user errors, or a security breach. Logical failures account for approximately 80 percent of all incidents. A logical failure can be easier to repair, but it may also not be noticed for some time.

COMPONENTS OF AN INFORMATION

TECHNOLOGY INFRASTRUCTURE

A modern corporate computing environment consists of components that build on each other to support the functions of the business. You must understand each of these components and how they relate to your business process to create an effective recovery strategy. At the foundation of this infrastructure are data. Figure 17-1 shows the typical components of an information technology (IT) infrastructure.

Each layer builds on the layer below, building up to the application that the user sees. The applications interact in varying degrees depending upon the requirements of the organization. But no matter what the specific architecture, the foundation is the data that are stored on various media somewhere within the organization.

RISK ASSESSMENT

Your data are susceptible to loss or damage or both from several sources. Some key causes of data loss include:

Viruses. These malicious programs can get into your system at any time and strike when you least expect it. Once you’re infected, the virus can spread from system to system, destroying data along the way.

Natural Disasters. Fire, flood, and high winds can all cause physical damage to systems and make your data unavailable or unreadable.

Human-Created Outages. Systems can be damaged by a sudden loss of power, or worse yet, a small part of a data stream can be lost, causing damage that may not be readily apparent.

Hard Drive Crash. It’s not if a hard drive will fail, but when. A hard drive is most likely to fail within 90 days of being placed in service and after about three years of average use (see Figure 17-2).

Laptop or Smartphone Loss or Theft. The value of the data stored on a laptop or other portable device usually far exceeds the cost of replacing the hardware.

Market research firm IDC estimates that approximately 60% of all corporate data reside on laptop and desktop PCs.

Software Failures. Operating systems and storage area network software can fail, corrupting existing data.

Application Failures. Applications are not guaranteed to be bug free; a bug in an application can cause incomplete or incorrectly formatted or calculated data to be written into your files.

FIGURE 17-1: Information technology (IT) infrastructure.

Vendor Failure. If you are hosting e-commerce or other applications with an SaaS (Software as a Service) vendor, your data could be at risk if the vendor suddenly goes out of business.

Approximately 20% of application downtime is attributed to a “disaster.”

40% is caused by application code failures.

40% is caused by operator error.

20% is caused by system/environmental failure or other disasters.

Source: Legato Systems

FIGURE 17-2: Hard drive failure rates

There are both tactical and strategic issues surrounding the loss of critical corporate information. Tactical issues include:

Compromised Information. Your valuable information could fall into the hands of competitors if stolen by hackers or by losing a portable device. Your competitors having this information could be more damaging than if it were simply destroyed.

Lost Productivity. Re-creating lost data can be very expensive, especially if it must be re-created from paper records.

Employee Downtime. Employees need their information to do their jobs; this includes people in the sales, customer service, and accounting departments.

Loss of Customer Information. Loss of important customer records can seriously hinder your ability to serve your customers.

Increased Help Desk Support Required. Not only might your help desk people be needed to help restore your data, but they will be bombarded by users requesting assistance and information.

In 2007, TJX, the parent company of TJ Maxx and Marshalls, reported that hackers had gained access to approximately 45.7 million unencrypted credit and debit card numbers. Estimates of the cost of the breach range from $250 million to as much as $1 billion.

Strategic issues surrounding data loss are those that have an impact on some critical operation within your business processes. This might include:

Loss of Opportunity. Without up-to-date and accurate information about your customers and your company, data loss can result in lost sales. If you don’t have accurate inventory information, customers may order from someone else who can guarantee delivery from stock. Follow-up calls to customers might be missed if your CRM data are lost; this may also resulting in lost sales. Future sales could also be in jeopardy.

Decreased Operational Efficiency. The lack of access to data will result in a greater reliance on manual processes, which will drastically decrease your operation efficiency.

Inability to Support Customers. Without access to customer data, you will have a difficult time supporting your customers or will incur unnecessary costs providing support to which they are not entitled.

Increased Systems Costs. Your total cost of ownership (TCO) will increase, making it more difficult to make money if margins are thin.

Noncompliance Issues. Without accurate data, you might not be able to prove compliance with government mandates, resulting in fines and legal fees.

Other costs you may incur from a serious data loss incident include:

Customer Notification. Many states now require that companies notify all customers potentially affected by a data breach.

Litigation Expenses. Lawsuits resulting from a data loss incident can be very expensive.

Internal Investigations. Time and resources will be required to clean up after a data breach.

Forensic Experts. You may need to hire outside forensic experts to help identify any existing security weaknesses.

Software Updates. In many cases numerous software updates may be required to patch security holes.

Subpoenas by Government Authorities. You may be required to respond to subpoenas from state Attorneys General or by the Federal Trade Commission.

Stock Price. If you are a public company, your stock price may go down after a data breach becomes news.

Reputation. Data breaches affecting credit card information can be especially damaging to a company’s reputation with its customers.

CREATING YOUR DATA RECOVERY PLAN

Just like any other project, there are several distinct steps required to develop your plan to successfully recover your data after a disaster. The recommended steps, as shown in Figure 17-3, are:

FIGURE 17-3: Data recovery steps.

1. Planning.

2. Identify critical data.

3. Create appropriate policies and procedures.

4. Determine type of backups.

5. Develop recovery processes.

6. Plan testing and maintenance.

PLANNING

As with any project, a successful data recovery plan begins with proper planning. Your first step should be to review data recovery expectations with key stakeholders. Find out what their business needs are, and if there are regulatory requirements about which they are concerned. Few organizations have not done any data recovery planning, so your next step should be to review the existing backup and restoration strategies. Find out what is currently being backed up and how often. Are there procedures in place to periodically test the backups? How are the backups transported and stored? What new systems have come online since the backup documentation was last updated? Are old files being backed up that could be archived and removed from the live systems?

Begin researching the most efficient and effective ways to store your backed-up data. Do you have multiple facilities that can store each other’s data? Make sure that the data being stored cannot be destroyed in the same disaster; they should be at least 20 miles apart.

You must also plan for an analysis and classification of data. What is the importance to the firm of each file being backed up? The cost of protecting the data should be proportional to the value of the data. You don’t want to spend a lot of time and resources protecting data that are easily restored by other means.

There are numerous strategies for backing up and restoring of data. They include traditional offline storage methods, such as hardcopy printouts, magnetic tape, CD-ROM, portable hard drives, and microfiche. However, online methods, which include disk mirroring, storage area networks and Internet backups, allow for faster restoration of your data. The evaluation and selection of the appropriate strategies are critical to the success of your recovery plan. Other things to consider are whether your backup hardware and software tools are the latest versions from the manufacturer. Many firms have had a disaster only to discover that the software needed to read their backup media was outdated and difficult and expensive to obtain, or simply no longer available.

Where your data will be stored is also an important consideration. It’s most convenient if your firm has multiple locations that can store each other’s data. You might also have a reciprocal agreement with another noncompeting firm to store data for each other. Of course, you need to be reasonably sure that both locations won’t be affected by the same disaster. You’ll also need to make sure that both locations can handle the extra workload if one site is down. This option is difficult to manage and does not always work well in practice.

Another option is to use a commercial storage company that will have an environmentally controlled facility to protect the integrity of your media. It will also have tested procedures for storing and retrieving data in an emergency and can offer advice on your disaster recovery plans.

If you lose not only the data but the hardware on which it is stored, you will also need a place to set up replacement hardware. One option is to have a contract with a vendor to have an off-site facility ready if your location experiences an incident. There are several basic types of remote sites:

Cold Site. A cold site is simply a place to store your data. It should have adequate space and infrastructure (power, communications, and environmental controls) to support your systems. This is the least expensive option, but it requires the most time to get up and running in the event of a disaster.

Warm Site. A warm site has systems and communications ready to go, but it requires that data to be restored on to them before they are ready to use.

Hot Site. A hot site is an active duplicate of your live systems, with both systems and data ready to go at a moment’s notice. Hot sites are usually staffed 24 hours a day, 7 days a week, and are prepared for immediate action if an incident occurs.

Mobile Site. A mobile site is a self-contained transportable office custom fitted with IT and communications equipment. It is usually transported by truck and can be set up at the desired location. The mobile site needs to be configured before it is needed for it to be a viable recovery solution. If using an outside vendor, a service-level agreement is necessary to make sure the vendor is committed to meeting your needs in an emergency.

Mirrored Site. A mirrored site is an exact duplicate of your production site, with data stored there in real time. This is the quickest way to get your business back up and running, but it is also the most expensive.

The different recovery site options offer different cost and recovery time tradeoffs. Your organization’s restore time requirements and the results of your Business Impact Analysis will determine which option you choose. Figure 17-4 compares the resource requirements for the different recovery site options.

Restoring data in the fastest time possible will minimize the revenue loss caused by damaged or lost data. “Time to data” is a critical metric to evaluate when creating your recovery plan, and is defined as how much time it takes for your users to have access to their data after a disaster occurs.

Rapid “time to data” is fundamental in achieving reduced downtime and maximizing productivity and system I/O rates.

Source: Strategic Research Corporation

Asset management is an important key to recovering your systems. You’ll need an accurate and complete hardware and software inventory list. You’ll need to know when and where it was purchased, as well as the warranty status. You’ll need to know where the hardware was located and how it was configured. Your original software licenses will be necessary to facilitate getting new media from your software vendors. You’ll also want to research what the vendor’s policy is in a disaster situation. You’ll want to know what to do if you need to quickly obtain replacement software.

IDENTIFY CRITICAL DATA

The first problem you’ll face in creating your data recovery plan is finding the data. The amount of data being produced by business today is growing rapidly. These are not just data stored in traditional databases, but also include graphics, word processing files, spreadsheets, sound clips, and other enhanced forms of data. In many organizations, terabyte (approximately 1 trillion bytes) databases are becoming common; petabyte (1,024 terabytes) size databases are right around the corner. And, of course, paper is still an important repository of data; these paper files are stored in file cabinets, desk drawers, etc. Microfilm and microfiche are also still used in many organizations. For data that are stored electronically, there are products available for automatically discovering files and databases throughout your network.

FIGURE 17-4: Recovery site selection criteria.

The next issue after you have found the data is categorizing the data. Like paper files, much of the electronic data that are created are never referenced again. You’ll need to identify the critical data required to restore critical business operations. Don’t forget to review ancillary data and documentation and data that must be preserved due to legal requirements.

Nonessential Data

Much of what is stored on your file servers by users is data that are not essential to the operation of the business. This includes space-wasting data such as e-mail attachments, Internet cache files, and personal files such as digital pictures. This nonessential data can add to the cost of backup and recovery in many ways. If you have a hot-site facility, it will require more disk storage space. If you are performing backups using tapes or CDs, additional media will be required for backups. If you are using replication to a remote location, additional bandwidth may be required to support the transfer of all these files.

A place to start in reducing the volume of unneeded files is to have policies in place that prohibit the storage of personal files on company servers. Strict enforcement of these policies can dramatically reduce the amount of data that are backed up. You should also consider limiting the amount of storage place available to each user, which will force them to consider carefully what to store in their personal folders.

CREATE APPROPRIATE POLICIES AND PROCEDURES

Most companies do not have policies and procedures for storing and classifying data. And many that do have policies do a poor job of enforcement. Having policies that aren’t enforced can create a false sense of security, which can be worse than having no policies at all.

The first step in creating policies for storing and classifying data is to identify the owners of information. All data in the company should have an identified owner who is responsible for understanding the importance and use of the data.

Once the owners of the data have been identified, develop a policy for determining the relative importance of data. You can then develop an information classification scheme. Some categories you might use include business critical, sensitive, legally required, and noncritical.

Business Critical. These are data that you must have to run your business. This can include customer lists, production drawings, accounting files, etc.

Sensitive. These are data that you would not want your competitors to see. This might include customer lists, employee lists, production process documentation, etc.

Legally Required. This is information that you need for compliance with government regulations, such as Occupational Safety and Health Administration (OSHA) compliance data, Environmental Protection Agency (EPA) information, hiring data, etc.

Noncritical. This is information that you can live without. Up to 90 percent of all information stored in file cabinets and databases is never retrieved, so this category can include a lot of data.

DETERMINE TYPE (OR TYPES) OF BACKUPS

Different types of data and different time to data requirements will require different backup processes and media. Types of backups include:

Regular backup to tape or other removable media.

Remote mirroring.

“Electronic vault” storage via a wide area network or the Internet.

You will probably use a combination of techniques, balancing time to data versus cost tradeoffs. Traditional tape backups are still widely used and can be effective, but they can create transportation issues, storage issues, and restoration issues. If not handled and stored properly, tapes can fail without warning. They require that the application also be reloaded, and software to read the tapes must be available. Electronic vault storage allows you to save your data over a wide area network, such as the Internet, and can be easier to restore than tape. Remote mirroring ensures that there is little or no data loss, but it is the most expensive option.

DEVELOP RECOVERY PROCESSES

The last step is to develop and document the process for both backup and recovery of data. It does no good to have a plan in your head or one that sits on the shelf. Schedules will need to be developed to ensure that backups are made in a timely fashion. Some criteria to be considered when evaluating which recovery techniques to use include:

RTO (Recovery Time Objective). How quickly must the data be restored before business is adversely affected?

RPO (Recovery Point Objective). How much data can you afford to lose before the business is adversely affected?

Availability. Can the system be down while you create the backups?

Restoration. How sure do you have to be that you can restore the data?

Value. How much is it worth to protect the data?

Performance. What are the performance requirements of the application?

You must also consider how effective each recovery technique is in protecting from the different types of loss. Each business process may have a different recovery process.

DATA STORAGE OPTIONS

There are numerous options for data storage, each with its own advantages and disadvantages.

Tape Backup

Tape backup is almost as old as computing itself. Tape has a low cost per gigabyte, and it is relatively easy to transport and store. Tape has been a reliable workhorse for the storage and archiving of important data, but it is not foolproof. Tapes can fail, so it is critical that backup tapes are periodically audited. The audit should be done by randomly selecting a tape and verifying that it can be read and restored using different equipment than that used to create it. An emergency is not a good time to discover that the tapes are unreadable or can only be read by the equipment used to create the backup.

If the data are important enough to back up, then they are important enough for you to implement the appropriate levels of physical and logical security. Ensure that the tapes are stored in a climate-controlled location free of dust and other sources of contamination. You should also make multiple copies of the tapes that can be stored in different locations to increase the chances of the data surviving a disaster.

Almost as important as how and where the tapes are stored is creating a tape rotation schedule. It is impractical in all but the smallest organizations to back up everything each time a backup is performed, so the normal practice is to perform a full backup periodically (e.g., weekly) followed by regular backups of any changes that have occurred since the full backup. The most common tape rotation strategy is called the Grandfather-Father-Son (GFS) backup scheme. It offers the following benefits:

A minimum number of tapes is required to keep all the system’s data backed up.

It is easy to understand and perform, making it more likely to be followed.

It is relatively easy to restore lost data from backups using this process.

It minimizes the wear and tear on both the tapes and the equipment.

The most common GFS backup process is to use a seven-day schedule where a full backup is created once a week (usually over the weekend). Incremental backups are then made each of the other days. Tapes can be rotated and reused at specified intervals, depending upon how many copies you wish to store. An example GFS backup strategy is:

1. Create an initial complete backup. Label this tape “Month 1” and store off-site. This is the first “Grandfather” tape.

2. Create a full backup at the beginning of the week. Label this tape ‘Week 1” and store off-site. This is a “Father” tape.

3. On each of the other days, perform an incremental backup using a different tape for each day. Label each tape with the appropriate day of the week. These are the “Son” tapes.

4. On the same day of the week that you did the first full backup, perform another full backup and label the tape “Week 2.”

5. Repeat for each week of the month, reusing the incremental backup tapes each week.

6. After 4 weeks, make a full backup and store this tape off-site. This becomes the second “Grandfather” tape. The first “Grandfather” tape can now be reused.

7. Repeat the weekly process, reusing the “Father” tapes from the previous month.

Disk Mirroring

With disk mirroring, data are written to two different disks to create two identical copies to increase the odds that at least one copy of the data is available at all times. The main disk used to store the data is called the protected disk, and the disk to which the data are replicated is called the backup disk. The two disks can be in the same location or in different locations. A WAN is used if the backup disk is at a different location from the protected disk. Installing the backup at a different location provides protection against a disaster that occurs at the location of the protected disk. While this is an effective approach, beware of its impact on your network traffic load.

Two different types of disk mirroring are available, synchronous and asynchronous. Each provides a different time-to-data recovery, and each has different performance considerations.

Synchronous mirroring works by writing to the backup disk first, then writing to the protected disk once it has been confirmed that the write to the backup disk was successful (see Figure 17-5). This type of mirroring ensures that the backup data are always up to date, but it is slower and more expensive than asynchronous mirroring. Special disk controllers are required to enable the two-way communication between the disks, and there is inherent latency between the writing of the data to the backup disk and waiting for the confirmation.

Asynchronous mirroring (or shadowing) works by sending the data to both the protected and backup disks at the same time (see Figure 17-6). It is cheaper than synchronous backup, and more than one system can write to the backup disk. It is also quicker, since the application does not have to wait for a confirmation on the write to the backup disk. The downside to asynchronous mirroring is that you cannot be guaranteed that the last transaction before a disaster was successfully written to the backup machine.

FIGURE 17-5: Synchronous mirroring.

RAID

RAID is an acronym for redundant array of inexpensive (or independent) disks and is used to provide fault tolerance to disk storage systems. RAID works by combining a collection of disks into a logical array of disks using a special disk controller that does not require all disks to be functioning to maintain data integrity. It can be implemented using either hardware or software. The RAID drives are seen as a single device by the operating system. RAID also increases disk performance and reliability by spreading the data storage across multiple drives, rather than a single disk. The terms used when describing a RAID implementation are defined below:

Duplexing. Disk duplexing involves the use of two RAID controllers writing the same data to two separate disks simultaneously. A system using duplexing can survive the failure of either a disk controller or a hard disk.

FIGURE 17-6: Asynchronous mirroring.

Mirroring. Disk mirroring involves the use of a single RAID controller writing the same data to two separate disks simultaneously. A system using mirroring can survive the failure of either hard disk. Both duplexing and mirroring can slow down system performance because the data is being written twice.

Striping. Striping involves breaking up the data into smaller pieces and writing the different pieces to multiple disks. The data may be broken up into bits, bytes, or blocks depending upon the RAID implementation used. Striping is faster than either duplexing or mirroring.

Parity. Parity is a way to achieve data redundancy without the disk space overhead of mirroring by storing logical information about the data being written to facilitate recovery. Parity is used with striping and requires at least three disks. The parity information is either stored across multiple disks or on a separate disk.

There are several levels of RAID operation, each with its own balance of redundancy, fault tolerance, cost, and complexity.

RAID 0. Disk striping. This implementation of RAID uses disk striping. The data are divided among several disks, which allows for good performance, but with no redundancy. This level offers no protection against data loss if a disk were to fail, and it is not recommended for data recovery purposes.

RAID 1. Mirroring and duplexing. This level of RAID involves mirroring or disk duplexing of the data across two or more disks. This provides for redundancy in case of a disk failure. Performance is slower than with RAID 0, especially during data writes. This level is simple and inexpensive to implement, but 50 percent of the storage space is lost because of the data duplication.

RAID 2. Bit-by-bit striping. This level stripes data bit by bit across multiple drives and is used with disks without built-in error detection. Since most modern disks have built-in error detection, this level of RAID is rarely used today.

RAID 3. Byte-by-byte striping. This level stripes data byte by byte across multiple drives, with the parity information stored on a separate disk. The parity disk can be used to restore data if a failure occurs. The parity information is at risk because it is stored on a single drive.

RAID 4. Block-by block striping. This level of RAID stripes data at the block level. Just like RAID 3, the parity information is stored on a separate disk. Performance is greater than with RAID 2 or 3 because the data are handled in block sizes.

RAID 5. Striping with distributed parity. This level is similar to RAID 4, except that the parity information is stored among the available disks. RAID 5 is a common implementation of RAID.

RAID 10. Mirrored striping. This level of RAID (sometimes called RAID 0+1) is a combination of RAID levels 0 and 1. Data are striped across multiple disks and also mirrored. It provides the best fault tolerance of all the RAID levels but is obviously the most expensive.

Load Balancing

Load balancing is used to distribute network traffic dynamically across a group of servers running a common application to prevent any one server from becoming overwhelmed. Using load balancing, a group of servers appears as a single server to an application on the network. The load balancing process is part of the network operating system; the process monitors each server to determine the best path to route traffic on the network to increase performance and availability. Load balancing also allows the application to continue running even if one of the servers goes down. So long as at least one server is available, the application will continue running. Load balancing can be implemented on different servers at a single location or at different sites. If load balancing is implemented on servers at different sites, it can act as a method to allow access to applications in the event of an incident at one of the locations.

Network Attached Storage

A network attached storage (NAS) environment is a common storage area for multiple servers. NAS environments are useful for storage or file server applications, such as mail and Web services. A NAS server runs a minimal operating system, and is optimized to facilitate the movement and storage of data. Using a NAS environment creates a centrally managed storage pool, which allows new storage devices to be added without requiring network downtime. Storage volumes from a down server can be easily reassigned, or new storage can be easily added if needed. The flexibility provided by a NAS environment increases the availability and reliability of network storage, adding value to your disaster contingency plans.

Storage Area Networks

A storage area network (SAN) is a high-speed, high-performance network that allows computers running multiple operating systems to store data on a single virtual storage device. A SAN is designed to handle backup traffic more efficiently than a NAS environment. The SAN can be local or remote, and usually communicates with the server using a fiber channel. By moving the storage off the LAN, backups can be performed without affecting the performance of the applications on the LAN.

Online Backups

Online data storage is becoming popular as software as a service (SaaS) is being used more and more to provide services to end users. With online data storage, changes to your data are delivered via the Internet to the service provider. This allows your data to be stored at a secure, professionally managed location away from any dangers to your facility. Online data storage provides the following benefits for disaster recovery:

The most obvious benefit is that your data is automatically stored to another location. Backups do not have to be manually started or managed.

Your data can be protected using a single solution that is accessible anywhere there is an Internet connection.

Remote offices and road warriors can back up their data without requiring separate hardware or complex VPN solutions.

There are lower upfront costs to implement an online backup solution—service is normally provided on a subscription basis.

No special in-house technical skills are required as this becomes the backup service provider’s responsibility.

As with anything else, there are tradeoffs you must be willing to make to implement an online data storage solution for use in disaster recovery:

Likely higher overall costs since the service is normally provided on a subscription basis—the provider recoups its equipment and software costs over the term of the subscription agreement.

There may be issues with retrieving your data from the provider.

Your provider could experience an outage which prevents you from accessing your data.

Restoring large amounts of data over the Internet is time- and bandwidth-consuming.

What happens to your data if your online data storage vendor goes out of business?

A hybrid option for online backup combines the best of onsite backup storage with the best of traditional online backup services. Some vendors will provide you with an appliance and software that allow you to use the Internet to do online backups, yet still have physical access to the backup device at a location that you control. This protects you against some of the disadvantages of service-only online backup solutions, such as losing your data to bankruptcy of your service provider. It also makes complete restorations easier, as the device can be physically brought onsite and connected directly to the local network for quick restoration.

Figure 17-7 is a comparison of the relative availability value of the different data storage options described here.

FIGURE 17-7: Data availability options.

VIRTUALIZATION

An increasingly popular technique for managing the explosion of data storage is the use of virtualization. Virtualization as it is most used today essentially uses software to mimic hardware. The typical software application wastes a tremendous amount of storage space. Many applications require that some minimum amount of storage be allocated for their use, and then in many cases only a small fraction is actually used. Storage virtualization allows the physical storage from multiple storage devices to appear to be a single storage device to the operating system or application. The storage can then be allocated as needed for use by users, applications, and servers. Storage virtualization can increase utilization to 75 percent or better. (See Figure 17-8.)

There are three basic approaches for virtualizing storage. The most common (called in-fabric) is the use of storage area network (SAN) devices connected using a high-speed fiber channel network. Software is then installed on a host server or a storage virtualization device is installed as part of the SAN; either provides the layer of abstraction between the hosts performing the I/O and the storage controllers providing the storage capacity. The second method, called host-client, uses software running on file and application servers to detect available storage and to maintain the metadata necessary to manage them. The third method, known as in-array or embedded functionality, uses a special network controller and management software to manage different storage systems as one large resource. All methods provide the advantages of storage virtualization, which from a disaster recovery prospective means:

Data storage becomes more mobile and easier to deploy.

Virtual tape library (VTL) technology can be used to decrease total backup time.

FIGURE 17-8: Storage virtualization.

Production and recovery storage environments no longer need to be strictly homogeneous.

Disaster recovery costs can be lower, as less expensive storage devices can be used at the recovery site. The less expensive devices, while maybe not ideal, might work fine until the production devices have been restored.

Easier administration during recovery as virtualized storage can be managed from a single administrative console.

Greater flexibility in managing changing application storage requirements.

Issues to be aware of when considering storage virtualization include:

If you have heterogeneous storage devices, are they all compatible with the virtualization technology you’re considering?

Applications could experience a decrease in performance if the recovery site data storage hardware has a slower response time.

Ensure that the storage metadata is protected and backed up.

Be aware that whichever option you choose, you’ll be locked into a particular vendor.

SOMETHING FOR YOUR SUPPORT PLAN

Your plan will need to be tested and updated periodically to keep it effective. Testing allows you to identify any deficiencies in your plan so that they can be corrected. This includes not only equipment and backup issues but personnel issues as well. Each component of your plan must be tested to verify the accuracy and completeness of your recovery procedures and to determine the overall effectiveness of the plan. Some areas to review when testing your plan include:

Ability to restore critical applications from backups.

Performance of recovery personnel.

Performance of backup equipment.

Communications.

To remain effective, your plan must be kept up to date with changes in your production systems. Changes in your IT infrastructure can be caused by business process changes, technology upgrades, regulatory requirements, employee turnover, or new policies. Any of these events should trigger a review of your recovery plan. It is therefore essential that your plan be reviewed and updated frequently to ensure that new processes are documented and that the recovery process is updated to reflect these changes. Items to be monitored for changes include:

Hardware, software, and peripheral equipment.

Business operation requirements.

Security requirements.

Technology changes.

Recovery team contact information.

Vendor information.

Regulatory requirements.

Because your recovery plan contains potentially sensitive personnel information, its distribution must be controlled. Copies should be stored at the home of key recovery personnel, at your production location, and at your off-site recovery location with your backup media. The individual in charge of recovery planning must maintain a list of who has copies of the plan, and a record of when changes were made (use Form 17-1, Recovery Plan Distribution List, and Form 17-2, Recovery Plan Change Record, found on the enclosed CD, as examples).

CONCLUSION

Data are the lifeblood of modern businesses; by having an effective data recovery plan you can help ensure that your business will survive an unexpected emergency. The steps are simple, but must be diligently performed to be effective:

Identify what data are important.

How soon do you need it?

What is it going to cost not to have it?

Test your recovery procedures.

CHAPTER 18

WORKSTATIONS

The Weakest Link

Why is it drug addicts and computer aficionados

are both called users?

—Clifford Stoll

INTRODUCTION

At the user end of our networks are typically workstations (or personal computers). Over the years, workstations have evolved first to supplement mainframe computer terminals (dumb terminals) and then to replace them. Few companies still employ dumb terminals. PC-based workstations have become so inexpensive that they can be found all across companies performing a wide range of functions. And, from a business continuity point of view, this is exactly the problem.

Mainframe computers centralized computing power and also centralized data storage. To view data stored by a mainframe required a password, and data files had various levels of security to protect them. Important data were stored in a central location, making backing up the data relatively easy. But a problem with mainframes was that programmers could never keep up with demands for their services. Personal computers, along with their easy-to-use programs, gradually migrated this capability to the individual’s desk (hence the “personal” in Personal Computing). As this occurred, all the environmental, electrical, and physical security protections that are provided for the backroom mainframes were no longer available to the workstations and the data residing in them. You’ve got a problem!

In discussing critical workstations, keep in mind that servers (specialized systems optimized for storage speed and other services) located outside the protection of the computer room are always considered critical units and must be protected as such.

RISK ASSESSMENT

As always, begin with your risk assessment. Normally workstations are listed in the departmental risk plans. Are there any critical workstations in your department? Before answering, think about what a workstation can be today. It can be a desktop unit. It can be a notebook PC that a manager uses at work and then carries home at night to catch up on urgent projects. It can be the PDA (personal digital assistant) unit carried around by the inventory manager to track shipments. It can even be a Web-enabled cell phone. In a factory, it could be a machine tool controller. In a hospital, it could a testing or monitoring device. Computing power is now everywhere!

So what to do? The first thing to do is an asset inventory. Check to see how many of each type unit you have. Categorize them by what they are (notebook, PDA, desktop) or what they are used for. Every computer has several things in common:

1. At some point, the hardware must be repaired or replaced.

2. It runs a stored program, often from a hard disk. In some cases, the program is stored in a ROM chip and rarely changes.

3. In almost every case, it stores data.

4. They are delicate flowers adrift in a cruel world.

So let’s take these one at a time. We said at some point the hardware must be repaired or replaced. Funny thing about workstations is that their usefulness fades away long before they stop working. If your processes depend on the constant availability of a specific machine, then imagine what would happen if some day it stopped working. Often you can replace it, but if the machine used an older operating system, you might have difficulty getting the old software to run on the new machine. In general, if a workstation is used in your business, it should be replaced at least every 4 years. If the workstation is critical, the hardware should be upgraded at least every 2 years. Your concern is maximum availability. If a change in hardware forces a change in the software, then at least it will be a planned event with time to address the issues that arise instead of something patched together in a crisis.

The second function common to every computer is that it is running software. This is true, even if it only runs the same software application over and over again. Like hardware, software has a useful life. If it gets too old, you should reconfigure the workstation to use more contemporary hardware and operating systems. This assumes you have the source code. If you cannot find it, then re-create the same software functionality immediately. Do not wait for the other shoe to drop! If the software was purchased as a “package” from a company, periodically check with the company to ensure the software is still supported. The supplier will provide updated versions of the software that will run on current operating systems. If not, then this must also go on your controlled upgrade list. Without support, the software may not work on replacement hardware in an emergency.

By keeping your software up to date, you reduce the number of spare workstations necessary to keep onsite. If a workstation running a critical application stops working, you can quickly exchange the hardware, reload the data from backup and proceed with your business with a minimum of downtime.

In addition, software sometimes requires upgrading. This both updates the software’s function and repairs problems in the code. Unfortunately, it may also introduce new problems. Have you ever installed an operating system upgrade that killed the workstation it was supposed to save? Controlling versions and testing software is a subject for a data processing book. The key here is that you need to be aware of any changes to a critical workstation’s software.

Always keep backup copies of critical software. The ability to restore a PC’s software from a backup copy is critical to a prompt recovery.

Be sure to include on your asset list all software used in critical units to support critical business functions. When in doubt, list them.

The third function that all computers have in common is that they store data. Most data are not worth retaining or can be lost without damage to the company. Those files are not our concern. Do your workstations hold any critical data? We will delve deeper into this subject a bit later, but as you make your asset inventory, note any critical files or general categories of files these units hold. Your list of data files should indicate:

Size. Determines optimal backup/recovery method.

Format. Is it in Access? Excel? WordPerfect?

Data Origin. Does this workstation create or modify it?

Volatility. How often does it change?

The fourth function that computers have in common is that environmental stresses may weaken the hardware. Always use a properly rated, functioning surge protector for your workstation and notebook PCs. Compared to your workstation, notebook PCs have the advantage of defaulting to battery power if they lose power from the wall plug. This acts as a built-in Uninterruptible Power Supply (UPS). Critical desktop PCs also need the protection of a UPS to ensure they do not suffer a “hard stop” when losing electrical power. This reduces the likelihood of a power outage resulting in a corrupted data file. After installation, ensure your users do not burden the UPS by plugging nonessential devices into it.

In addition to clean power, workstations are designed to exist within specific temperature and humidity ranges, as established by the manufacturer. Stray outside of these and your hardware will weaken and eventually fail. A typical office environment is usually fine. But a hot, dirty factory stresses the equipment. Notebook PCs left overnight in a car during the depths of winter can be damaged by freezing—or baked in the hot sun on a scorching summer day. Problems may not immediately appear but marginal components may begin to fail.

So far, you have identified the risk assessment from data provided from each department about its critical assets and your own asset inventory made walking around and looking at what needs to be supported. Now, the question is, how can you mitigate the risks (or threats) to your equipment, programs, and data? The most basic step involves the physical security of these assets. As you work through these steps, keep in mind your focus is on supporting your vital business functions, but from time to time you will want to extend this to other machines as well.

Physical Security

Physical security protects your unit from unauthorized access, theft, and sabotage. Sabotage can have a temporary effect if, for example, a cable is unplugged, but it can also be more devastating and permanently destroy the unit. Physical security is also important to protect software and backup files. Stolen software, which can be copied and distributed across the company, can create significant legal problems.

To prevent unauthorized access, lock the computer in a room open only to authorized people, such as a computer room or an executive’s office. These are secure locations because the computer room is always locked and the executive’s office is watched by the secretary or locked when not in use. Locking the room or watching the unit provides a barrier to theft and sabotage.

Other workstations in your company also require protection. One would be the workstation that controls the electronic door locks, such as in a hotel, or to your employee entrances. Someone sneaking in to steal something may also steal or destroy the records of their entry into this unit. Therefore, this workstation is also normally under lock and key.

How about the workstation that records the time attendance for your employees? Some companies use a barcode, magnetic strip, or RFIS chip on their employee badges to indicate when each person arrived at or departed from work (punched in/punched out). This information is recorded in a database for payroll purposes. Such a workstation must be in a place where it cannot be stolen (they contain significant private information about employees) or sabotaged.

However, most workstations are not kept in locked rooms. They are out where workers, visitors, and others have access. In some companies, a lot of people pass by the desks or float through the offices. It is not always possible to be sure who belongs there and who does not. During lunch, when everyone has abandoned the office, it takes but a moment to snap shut a notebook PC, slip it into a briefcase, and go out the door! Desktop units are also fair game, but they are a bit bulky and more likely to disappear overnight or on a weekend. Handheld units, such as PDAs or cell phones, barely make a bulge in the pocket of (stylishly) baggy trousers.

One way to hang on to your equipment is to strap it down to your desk. This is typically a steel cable through a loop in the back of your workstation. You can also buy a similar device for your notebook PC. To remove it, you must have the key to the lock. True, this can be defeated with brute force, but it drastically reduces the number of units stolen. However, it will not stop the deliberate attempt to damage the unit, so do not keep anything critical out in the open when it can be hit, burned, shocked, crushed, etc.

Workstations used in warehouses and on the factory floor should be housed in locked cabinets. Factories are a harsh environment for electronic components. A monitor and keyboard are all that a user requires. Ensure that the cabinets are well ventilated to avoid overheating the components. Locking the cabinet protects the unit against theft and sabotage. It also prevents someone from loading unauthorized software onto it.

Another important physical security issue for personal computers involves safeguarding the workstations’ vital records. It is important that all PC software licenses are gathered after purchase and filed for future reference. Some companies create a software “image” of the operating system and all standard programs. This image is then copied into each workstation they own. The serial numbers on the software then all read the same. By holding these licenses in a secure location (often the same off-site storage used for your backup media), you can readily prove how many copies of a program you own.

The other vital workstation records are the backup media. These devices (CDROMs, tapes, external hard drives, etc.) all contain confidential company data. They must be correctly labeled and safeguarded as such.

Backing Up Your Workstation Programs and Data

The key to a rapid recovery from a theft, damaged workstation disk, or sabotage is restoration from your last good backup copy. Mainframe computers typically make a full copy of everything they have stored on disk once a week. Every day, they make an incremental copy of whatever has changed that day. In this way, they capture all the data and software necessary for them to function. The storage media (usually magnetic tape) is carefully transported and stored. Over time, some tapes are kept for historical copies and others are rotated back into use.

Workstations are another matter. The data are often scattered about the hard disk mixed in among programs, obsolete data, and pictures of Aunt Meg, and following a naming convention that defies most logic. Although a complete (image) backup of each workstation is desired, it is expensive and most users will not faithfully do it. In addition, there is the cost of the backup hardware, handling the backups, and all the media necessary to copy these disks.

Selecting a Workstation Data Backup Strategy

Hard disk capacities in workstations have grown rapidly over the years, and the smallest available disk sizes are more than most users can fill. Still, your backup tool must be capable of copying everything on the disk on to the media selected.

Before beginning your data backup strategy, require users to store all mission-critical files on a computer room file server. Make this a firm rule and, whenever you find such a file, stay after it until it is migrated properly. These files should not be left to chance, and the computer room staff will ensure they are included on the normal backup tapes. This category should include databases, spreadsheets, legal documents, and anything that is truly mission critical, which includes all documents that must be retained for legal or regulatory reasons. For a few individuals, this may mean that all their files will reside on the server. So be it! The end-user’s workstation can still access these data as if they were present on their workstation’s disk.

That said, your first consideration is, “What do I want to back up?” Workstations hold a lot of programs that may not even be used but could require a considerable amount of space to repeatedly back up. For example, a workstation’s operating system may require several gigabytes just to back up the software. Applications software can easily triple this number. Most of these programs rarely change. If a workstation were stolen or otherwise rendered unusable, you should be able to install your standard software image on a new workstation and load any additional authorized programs from your support staff copies. Then the data can be loaded and you are finished. Therefore, for most workstations, it is not necessary to make backup copies of the programs—just the data.

Special-purpose workstations may have specific configuration settings in the software or operating system that are necessary for it to work properly. For these workstations, you may want to make image backup copies (the entire disk).

This is important because the larger the amount of data to back up, the longer it will take. If you are copying unnecessary materials, it will make your backup strategy more expensive to no purpose.

Once you have trimmed the job down to copying only data, consider which data you want to save. If you are keeping copies of old files and correspondence on your workstation for historical purposes, consider moving them to a CD and deleting it from the workstation. Again, why copy something over and over when once will do? The archive CD should have the proper level of security on it. Either store it off-site with your other vital records or in a locked vital records storage onsite. Mark every CD legibly as to the originator, date created, and contents. Do not stuff it in your desk if it contains any sensitive information.

Another valuable space saver is to delete files you no longer need. Some people never clean out their attic and others never delete files from their PCs. If it isn’t needed—delete it! If in doubt, copy to a CD and then delete it.

So with only active data files left, containing data that are useful, the last and very important step is to store all your data under one master directory. A common choice is to use C:\Data. Under this are folders by topic and by product, such as Excel. To back up all the data, copy them to a CD-ROM for storage with a drag and drop. Click on the Data directory, drag it over to the CD-RW drive, and drop it. The operating system copies it for you and then waits for you to tell it to burn the CD. You can type away while the copy is made.

CDs are so inexpensive, on average less than 20 cents each, that they are practical for data backup so long as the older, unneeded copies are properly destroyed. For most end-user workstations, this is the easiest way to make their own copies. Be sure they understand and comply with your standards for marking backups (for easy identification when it is needed for a restore). In addition, simplify the collection and filing of backups. From time to time, check off who is handing in backups to identify the people who are forgetting their good data processing practices.

With that said, here are the factors to consider when selecting a backup strategy for your users.

Storage Volume Requirements. The backup media used should be capable of handling the anticipated volume.

The Length of Time to Make the Backup Copy. Huge amounts of data take huge amounts of time to copy. However, a typical end-user workstation should not require more than a single CD to back up its data, if storage-intensive files such as audio, movies, and pictures are stored separately from the data.

Interoperability. The backup media should ideally be compatible with the operating systems and applications in use today and in the future. For example, in 2003, most major PC assemblers announced their intention to drop the 3.5-inch floppy disk from their new PCs.

Backup Software. Ensure that users know the process for making backup data copies and that their tools are easy to use. This will improve the chances that backups are regularly made.

Backup Technologies

Many different backup technologies are available for end-user backups. Whatever is used, be sure to mark the backup with the date and the user’s name. The most popular include:

Tape Drives. These are readily available for desktop computers. Most tape backup software can be set up to run automatically. Tapes provide the greatest amount of storage space, Depending on the technology, a single cartridge can hold several hundred gigabytes. However, to fill such a large tape may require a considerable amount of time. Further, the higher the capacity, the greater the cost per cartridge. Multiply this by the number of workstations that need to be backed up and this can be an expensive proposition. Tape is best suited for server backups.

Removable Media. This includes products such as Zip cartridges from Iomega, and are used similarly to floppy disks. They are faster and easier to use than tape drives. The cartridges are comparable in cost to tape media, but they have a lower capacity.

Compact Disk Read Only Memory (CD-ROM or CD). CD-RW drives are standard on new PCs and have replaced the old floppy disks as the medium on which to load software and data. CDs are low in cost and have a capacity of approximately 700 MB of data per disk. Most CDs (known as CD-Rs) can only be written to once, but CD-RWs allow you to write to and edit the material on the disk. Given the low cost of CDs, they are the most cost-effective way to back up end-user PCs. If all your data are consolidated under one master directory, you may be able to fit them all onto one disk.

DVD-Rs (DVD Writers). These are a step up from CDs in that a DVD holds approximately 4.2 GB per disk. The disk is the same compact size as a CD. These devices are gradually dropping in price. Most PCs now have one drive that formats and reads both CDs and DVDs.

Internet Backup. This is normally a commercial service that uses the public Internet to back up data from the end-user workstation to a remote server connected to the Internet. Software is loaded onto the workstation that is used to schedule the backups, select the files to be backed up, and communicate to the backup server. Data are normally encrypted to ensure security during transmission. A major advantage of this method is that the user does not have to deal with backup media, and the backups can be run unattended. One disadvantage to this approach is time, as the speed of your Internet connection affects how fast data can be transferred to the backup system.

However, this is a very useful tool for “road warriors,” as it allows them to back up their data while traveling. Remember, only back up the critical files that have changed. Almost all hotels have installing broadband services for travelers, so this option is becoming more attractive.

Network Storage. If the workstation is connected to a network, that unit can back up its data via the network to a server. The network file servers are then backed up daily so your files eventually end up on tape. There are two basic ways to use the network:

1. Backup Initiated by the Server. The server can be configured to read the data from the workstations and store the data either to the server hard disk or to a backup medium.

2. Store Data to the Server. A networked disk can be configured for use by the end users. The end users configure their application software to write to the virtual drive rather than to a local drive. The networked drive is then backed up as part of the normal server backup process.

Backing up to a server can present several problems. The first is space. Most servers limit the amount of space available to an end user. Once the limit is reached, the backup dies. Estimate the size of your typical workstation data directory. Multiply it times the number of users. This approach, while elegant, may not be practical.

The second obstacle is bandwidth. All these workstations can clog the network with massive data transfers. Finally, the server disks can only run so fast.

WORKSTATION VIRTUALIZATION

Another option for making recovery of workstations easier is to use virtualization technology to run an image of the workstation on a protected server. If the workstation is lost in a disaster, the workstation image can be restarted on a new workstation, and the end user does not miss a beat. This also makes the workstations easier to maintain as new software is installed and old applications are removed. A virtualized workstation can provide a managed, stable desktop environment that can be accessed using a standard PC or a less expensive thin client device. The virtualized desktop software (such as VMware Workstation or Microsoft Virtual PC) provides a virtualized full client environment using a server-based hypervisor. This allows the user to have full administrative control over the desktop environment and applications. Some of the advantages of desktop virtualization include:

Easy to deploy desktops to new users.

Desktop machines can be less expensive PCs or thin client devices.

Ability to use your desktop environment from any PC with network or Internet access.

Access to typical desktop features such as multiple monitors, USB devices, etc.

Disaster recovery at the desktop is simplified as a new device can be quickly installed to use the virtualized image on the server. Backups can be managed at the server level.

Virtualization of desktop workstations also creates the opportunity to have desktops provided via the Internet, much like many vendors are doing with applications using the “software as a service” (SaaS) model. For desktops one term being used is “desktop as a service” (DaaS), while others use the term Virtual Desktop Infrastructure as a Service (VDI).

Some issues to be aware of when considering desktop virtualization:

Your “per-seat” cost may be initially higher due to the cost of the servers, virtualization software, and Windows licenses.

The OEM version of Windows that comes with most new PCs cannot be used in a virtual environment; new licenses must be purchased.

User resistance to giving up their full client PCs.

END-USER BACKUP ISSUES

It is important that the data recovery plan emphasize the availability of the data, protect the data’s confidentiality, and ensure the data’s integrity. Some processes to follow to make restoration of workstations easier include:

Train end users on the importance of backing up data on a regular basis. If the process is simple and easy to follow, they will usually cooperate.

Document vendor and configuration information for all specialty workstations.

Establish a mail slot–type drop-off for backup media in the data center where they can be dropped off securely. Provide labels that may prompt them to fill in essential information. When tapes are recycled (old backups no longer needed), provide them to users at a pick-up point. From time to time, test these backups to ensure they can be read. Sometimes data backups look like they worked but they did not.

Hard Disk Recovery

Sometimes you are just sure that whatever was on your hard disk is lost forever. This could be a workstation that was melted in a fire or submerged for days in a flood. It could have suffered a head crash. Don’t be too depressed. There are companies that specialize in recovering data from severely damaged disks. They can also recover data from deleted files (the ones that the usual file recovery software cannot rebuild).

These companies use specially trained engineers to disassemble the hard disk unit in their clean rooms and extract the data. The services can be expensive but the savings to your company can be considerable.

MOBILE DEVICES

Mobile devices are the wave of the future. The ideal is a unit that can accompany you throughout your workday. A unit that is unobtrusive, light, and always ready for use. Today we have notebook PCs, netbooks, e-readers, PDAs, and web-enabled cell phones. Once these devices depart the cozy confines of the office, they introduce a new range of issues to be addressed.

Mobile Security

Unlike desktop workstations, notebook PCs advertise how light and easy they are to carry. This portability also makes them easy to steal. Once a mobile PC is taken out of the office, it loses whatever protection your facility’s security force provides. Several steps should be taken to protect this equipment:

1. Keep it out of sight if possible.

2. Do not carry it in a carrying case that is obviously for notebook PCs. Use a standard briefcase or pack it in your luggage.

3. When it is not in use in your hotel room, store it out of sight.

4. When attending seminars or business meetings outside the office, never leave it unattended.

5. While it is in your car, keep it out of sight. Then if you go to a restaurant for lunch, it isn’t visible sitting in your car.

6. When passing through airport security, ensure no one is ahead of you before laying it on the scanner’s conveyor. This way it should arrive at the other side the same time that you do.

Essentially the same holds true for a PDA. While it is carried on your trip so it could be used, if a thief cannot see it, they cannot target it. Airports are a favorite place to steal notebook PCs, and there are many ways to waylay the unsuspecting. Targeting a company’s executives in an airport and stealing their notebook PCs is a very effective tool for industrial espionage. You think it was just a thief, but your competition may now know many hidden details of your operation.

Some managers take their notebook PCs or PDAs home to catch up on work. The problem is that if they have an accident on the way, someone must know to promptly remove the notebook from the vehicle as soon as possible (with the permission of the police). A notebook is an easy theft item that could be removed from a wreck and not be missed for several days. More than the PC itself, it is the data you are safeguarding.

Mobile Data Backup

Mobile devices present their own particular backup issues. This is even more of a problem for other mobile devices, such as PDAs, smart phones, and pagers, which are less likely to be connected to the corporate network. Users are also a problem, as most think that a disaster such as a lost or damaged device will not happen to them. These devices are much more fragile and more easily stolen than desktop PCs.

Always make a full backup or virtual image of your notebook PC before a business trip. This will lessen the impact on your job, because the replacement unit can be restored from the backup. If necessary, a new unit can be loaded and sent out to you.

If your PC has critical data files (such as legal briefs of contracts), they can be burned to a CD and stored in a different piece of luggage. Again, if the PC is lost, the CD can be loaded onto a new unit. Guard the CD carefully—your coat pocket will do!

The average IT-enabled person uses at least three portable devices and spends more than 1 hour per day trying to keep these devices synchronized.

PROTECTING END-USER VITAL RECORDS

Ensure that your department’s vital records program includes the handling of your data backups. Workstations tend to be somewhat secure, but if competitors can lay their hands on your data backups, in most cases, you would never know. Further, to break into your workstation they must hack past the password, but no such obstacle confronts someone who wants to read the backup tape. Therefore, always treat backup media as critical data and store it properly.

Another vital record is the paper copies of your workstation software licenses. These are essential to prove the number of licenses you have purchased in the event of a software audit. If your equipment was lost in a fire, the licenses can be used to demonstrate ownership and the software copied onto your new equipment (always consult your company’s attorneys if such a situation arises).

Along with the licenses, the original software media must be secured as a vital record. This reduces the likelihood that people will install unlicensed software on multiple workstations. On the one hand, they may believe they are helping fellow employees by providing programs for their use. On the other hand, they may be ready to resign and are setting you up to be turned in for using pirated software! Don’t take chances. After installing programs, promptly gather the media and store it with the vital records. Then if you are accused, you can show you have taken prudent steps to control and stop it.

Vital records can turn up in several unwanted places. Before recycling tapes from storage, be sure to erase them because they may not end up with the same user every time. Backups on CDs cannot be recycled and should be rendered unreadable, usually by crushing them or putting them through a paper shredder.

Another set of vital records to protect is found in your surplus workstations. When a PC is ready for donation to charity, remove the fixed disk and destroy it. Some people crush it; others make holes in it with a heavy drill. There is a lot of sophisticated technology in the world that can recover data from your disk no matter how thoroughly you reformat it. The charity receiving the PC will need to find someone to loan it old hard disks. Never send one out in your surplus PC.

Additional Resources

www.globalcomputer.com —Devices for securing workstations to a desk, backup hardware, and media.

www.bsa.org —Business Software Alliance. BSA educates consumers on software management and copyright protection, cyber security, trade, e-commerce, and other Internet-related issues.

(From http://www.bsa.org/usa/about/ )

CONCLUSION

When supporting workstations, it’s not a matter of if it will break, but when. Periodic upgrading of both hardware and software is necessary to ensure that support will be possible if a critical PC fails.

Although the proliferation of personal computers has produced many benefits, it has made life more difficult for those charged with protecting vital corporate assets. Physical security is now more of a problem, as these systems are scattered throughout the organization. Data security is also more difficult, as data are no longer concentrated in a central location. But proper policies and procedures for managing these devices can help you keep these assets safe and sound.