5 assign

Running Head: Wk5

WK5 77

WK5

Student’s Name

Institutional Affiliation

CHAPTER 13

TESTING YOUR PLANS

Test, Test, Test

Action is the foundational key to all success.

—Pablo Picasso

INTRODUCTION

Writing a recovery plan is only half of the challenge. The second half, the real challenge, is to periodically test it. Everyone can relate to writing a recovery plan. “Testing” a plan sounds like you do not trust it. Testing requires expensive technician time, the equipment and facility resources to conduct a test, and the expertise to plan the exercise. Gathering all of this into one place can be difficult.

Arranging for expensive technician time was tough enough to secure for writing the plans. The most knowledgeable people are usually the busiest. Getting them to give the time to sit down long enough to test a plan is difficult— yet essential. Testing validates that a recovery plan will work. A plan that is tested has a much higher possibility of succeeding over a plan that has never been proven. The many benefits to testing include:

Demonstrating that a plan works.

Validating plan assumptions.

Identifying unknown contingencies.

Verifying resource availability.

Training team members for their recovery roles.

Determining the true length of recovery time, and ultimately the ability to achieve the desired company RTO.

The Many Benefits of Plan Testing

Recovery plans are tested for many business reasons. An untested plan is merely process documentation. Testing a plan ensures that the document provides the desired results. The benefits of testing include the following.

TESTING REVEALS MISSING STEPS When people write a plan, they think about a process or IT system, and then write the plan so that they will understand what is explained and the steps to take. In this sense, the plan is a reflection of their experience. However, in a crisis, they may not be the person who will execute the recovery. Further, some people cannot break down a process to include each of its individual steps. In action, they will pick up on visual cues to take a specific action to fill a gap.

Therefore, the first purpose of a recovery plan test is to ensure that it includes all of the necessary steps to achieve recovery. Missing steps are not unusual in the first draft of a recovery plan. Other missing information may be IT security codes, the location of physical keys for certain offices or work areas, or the location of vendor contact information.

TESTING REVEALS PLAN ERRORS Writing a plan sometimes introduces misleading, incorrect, or unnecessary steps. Testing the plan will uncover all of these.

TESTING UNCOVERS CHANGES TO THE PROCESS SINCE THE RECOVERY PLAN WAS WRITTEN A plan may have been sitting on the shelf for a period of time without review. Over time, IT systems change server sizes, add disk storage, or are upgraded to new software versions. Business processes move machinery and change the sequence of steps, and key support people leave the company.

TESTING A PLAN TRAINS THE TEAM After a plan has been debugged, exercising it teaches each recovery participant his or her role during the emergency. It is one thing to read the words on a page and another to actually carry out the steps.

Types of Recovery Plan Tests

Exercises can consist of talking through recovery actions or physically recovering something. Discussion-based tests exercise teamwork in decision making, analysis, communication, and collaboration. Operations-based tests involve physically recovering something, such as a data center, telephone system, office or manufacturing cell. This type of test uses expensive resources and is more complex to conduct.

Everyone has their own name for the various types of testing. Tests are categorized by their complexity in setting them up and in the number of participants involved. These tests are listed here in a progression from least complex to most difficult to run:

Standalone Testing is where the person who authored the plan reviews it with someone else with a similar technical background. This may be the manager or the backup support person. This type of testing is useful for catching omissions, such as skipping a process step. It also provides some insight into the process for the backup support person.

Walk-Through Testing involves everyone mentioned in the plan and is conducted around a conference room table. Everyone strictly follows what is in the plan as they talk through what they are doing. This also identifies plan omissions, as there are now many perspectives examining the same document.

Integrated System Testing occurs when all of the components of an IT system (database, middleware, applications, operating systems, network connections, etc.) are recovered from scratch. This type of test reveals many of the interfaces between IT systems required to recover a specific IT function. For example, this would be to test the recovery of the Accounting department’s critical IT system, Human Resources IT system, the telephone system, e-mail, etc.

Table-Top Exercises simulate a disaster but the response to it is conducted in a conference room. A disaster scenario is provided and participants work through the problem. This is similar to Walk-Through Testing, except that the team responds to an incident scenario. As the exercise progresses, the Exercise Coordinator injects additional problems into the situation.

Simulation Exercises take a Table-Top exercise one step further by including the actual recovery site and equipment. A simulation is the closest that a company can come to experiencing (and learning from) a real disaster. Simulations provide many dimensions that most recovery plan tests never explore. However, they are complex to plan and expensive to conduct.

Validating the Recovery Time Objective

Testing recovery plans ensure that it can achieve the required recovery time objective. Since plans are tested in small groups, the actual RTO is determined by tracking the amount of time required to recover each IT system and business process. These plans fit into an overall recovery sequence (developed by the Business Continuity Manager). Once in this framework, the time required to complete each plan is added up (many plans execute in parallel) to determine if the RTO can be achieved.

Is it a “test” or an “exercise”? A “test” implies a pass or fail result. An “exercise” implies using something and is less threatening to participants.

WRITING A TESTING STRATEGY

Testing distracts an organization from its mission of returning a profit to shareholders. Everyone is busy meeting his or her own company objectives. Somehow, time must be found within each department’s busy schedule to test its recovery plans. To maximize the benefit to the company while minimizing cost, develop a testing strategy for your company. This strategy describes the type and frequency of test for recovery plans. An executive-approved testing strategy provides the top-level incentive for management compliance. The testing strategy is inserted into the administrative plan.

The testing calendar should reach out over several years. Keep in mind that different departments have their own “busy season” and trying to test at that time will be difficult. For example, the Accounting department will be occupied before and after the end of the company’s fiscal year. Payroll needs to submit tax forms at the end of the calendar year, etc. By using an annual testing calendar, it is easier to gain commitment from the various departments to look ahead and commit to tests on specific days.

Testing follows a logical progression. It begins with the individual plan. The next level is a grouping of recovery plans to test together. This is followed by a simulation of some sort. Executives become frustrated by the length of time required to properly test all of the plans in this sequence, but they are more disturbed by the cost to test them faster.

Begin by Stating Your Goals

As with all things in the business continuity program, begin writing the testing strategy by referring back to the Business Impact Analysis. If the recovery time objective is brief (measured in minutes or hours), then the testing must be frequent and comprehensive. The longer the recovery time objective, the less frequent and comprehensive the testing may be. Considered from a different angle, the less familiar that the current recovery team is with a plan, the longer it will take them to complete it.

Another issue is the severity of an incident. While the overall plan may tolerate a long recovery time, there may be specific processes whose availability is important to the company. This might be the Order Entry IT system or a critical machine tool. Consider testing those few highly critical processes more frequently than the overall plan.

The testing goal may be stated as, “Recovery plans are tested to demonstrate that the company’s approved recovery time objective of (your RTO here) can be achieved” and that all participants understand their roles in achieving a prompt recovery.

Progressive Testing

Testing follows a progression from simple to complex. Once a plan is written, it begins at the Standalone Test level and progresses from there. Any process or IT system that is significantly changed must be retested beginning at the Standalone level. The progression of testing is as follows:

Standalone Testing is the first action after a plan is written. It reveals the obvious problems.

Walk-Through Testing exercises a group of related plans at the same time, conducted as a group discussion.

Integrated System Testing tests a group of related plans at the same time by actually recovering them on spare equipment.

Table-Top Exercises test a group of related plans at the same time, based on an incident scenario.

Simulation Exercises combine many groups for an actual recovery in the recovery sites, based on an incident scenario.

Creating a Three-Year Testing Roadmap

Some tests only involve two people, while others can include most of the IT department. All tests require preparation time. This is necessary to coordinate schedules for people, exercise control rooms, and equipment. Copies of plans must be printed and distributed and exercise scenarios created. At a minimum, every plan should be tested annually. This can be accomplished by the manager and the process owner performing a Standalone Test to see if anything significant has changed in the process.

Few companies halt operations for several days to conduct a complete disaster simulation. Instead, they test “slices” of the recovery program. For example, the test might focus on a recovery of the Operations department or Shipping department. On the IT side, this would be a group of related systems that regularly exchange information, such as order entry, materials management, and billing.

Too much testing can reduce interest in the program. Practically speaking, testing is a preventative measure (all cost and no immediate payback) and does not increase a company’s revenue. Depending on the industry, testing may never progress beyond the table-top exercise stage. The Business Continuity Manager works with the program sponsor to identify the adequate level of testing for the organization and then spreads it throughout the year.

When developing a testing calendar, executives will vent their frustration. They will want something that is written, tested, and then set aside as completed. They do not like to consider that completed plans must continue to be exercised regularly. There are many plans and combinations of plans to test: business processes, IT systems, work area recovery, pandemic, etc. A typical testing schedule includes:

Quarterly

Inspect Command Center sites for availability and to ensure their network and telecommunication connections are live.

Data Backups

Verify that data backups (on each media type) are readable.

Ensure that every disk in the data center and key personal computers are included in the backups.

Inspect safe and secure transportation of media to off-site storage.

Inspect how the off-site storage facility handles and secures the media.

All business process owners verify that their employee recall lists are current.

Issue updated versions of plans.

Annually (spread throughout the year)

Conduct an IT simulation at the recovery site.

Conduct a work area recovery simulation at the recovery site.

Conduct a pandemic table-top exercise.

Conduct an executive recovery plan exercise with all simulations.

Review business continuity plans of key vendors.

All managers submit a signed report that their recovery plans are up to date.

Practice a data backup recall from the secured storage area to the hot site.

A partial plan exercise calendar might look like Figure 13-1.

TESTING TEAM

Testing a recovery plan is a team effort. The best results come from a clear explanation of the responsibilities of team members and some training to show them what to do. This enables each person to contribute expertise to the exercise while learning by doing. Training for individual team members is the responsibility of the Business Continuity Manager.

The duties for each of the testing team members will vary according to the type of test, with a full disaster simulation requiring the most time from everyone. Possible team duties include:

Business Continuity Manager

Develops a long-term testing calendar, updated annually.

Develops or updates a testing strategy.

FIGURE 13-1: Example of a three-year testing calendar.

Schedules tests.

Prepares test areas and participant materials.

Explains testing process to team prior to start of exercise.

Presents scenario.

Logs events during the exercise.

Keeps exercise focused for prompt completion.

Injects variations to scenarios during simulation testing.

Conducts after-action critiques of recovery plan and a separate discussion of the test process.

Provides a written test report to the program sponsor.

Sponsor

Reviews and approves recovery plan test calendar and testing strategy.

Approves initiation of all tests.

Provides financial support for tests.

Ensures internal support of test program.

Observes tests in progress.

Reviews written report of test results and team critique.

Exercise Recorder

Records actions and decisions.

Records all assumptions made during the test.

Drafts narrative of what happened during the test for the after-action review.

Exercise Participants

Prepare for the test by reviewing the recovery plans.

Participate in test by following the plans.

Offer ways to improve the recovery plans during the test.

Participate in the after-action critique of the recovery plans and the testing process.

Nonemployee Participants

Where practical include people from other organizations who have a stake in your plans, such as the fire and police departments, power company.

News reporters to report on the exercise and to participate by exercising your corporate communications plan.

Visitors, such as customer or supplier representatives.

EXERCISE SCENARIOS

A disaster scenario is a hypothetical incident that gives participants a problem to work through. The scenario may describe any disruption to the normal flow of a business process. The scenario should be focused on the type of problem that a particular group of people may face. For example, the problem and its mid-execution “injection of events” should encompass all participants.

Every simulation starts with a scenario, a hypothetical situation for the participants to work through. Scenarios that reflect potential threats also add an air of reality to the exercise. A good place to look for topics is in the plan’s risk analysis section or program assumptions. Another place to look is in the recent national or local news.

For example, who has never experienced a power outage or a loss of data connectivity? How about severe weather like a hurricane or a blizzard? Or consider tornados and earthquakes. Human-created situations, such as fire, loss of water pressure, or a person with a weapon in the building, are also potential scenarios.

The planning expertise of the test coordinator is crucial. The coordinator must devise an exercise schedule to include a detailed timeline of events, coordinate and place the resources involved (people, equipment, facilities, supplies, and information), establish in everyone’s minds their role, and identify interdependencies between individuals and groups.

In theory, a business faces a wide range of threats from people, nature, infrastructure, etc. In reality, few of these will occur. Some are dependent on the season and changes in the political environment. Whatever the crisis, the recovery steps for many threats are the same. A data center lost to a fire is the same as a data center lost (or made unusable) because of a collapsed roof, a flood, etc. In each of these cases, there will be many steps unique to that event. However, the initial actions in each case will be the same. It is this similarity that enables disaster recovery planning. A disaster plan is most useful in the first few hours when there is limited information, but the greatest benefit from containing the damage and restoring minimal service to the company.

Include in the scenario the incident’s day of week and time of day. The weekend response will differ from the work-time response. Consider declaring the scenario to include the company’s worst time of the year (such as the day before Christmas for a retailer). Also, the severity of the damage can at first appear to be small and then grow through “injects” provided by the exercise controller. Consider the example of a small fire. When a large amount of water was sprayed on the fire, it ran down the floor and saturated the carpet in the nearby retail show room. It also leaked through the floor into the data center below, soaking the equipment.

Ask the program sponsor to approve the scenario used in an exercise. This will minimize participant discussion during its presentation. It will also help to avoid scenarios that executives feel are too sensitive.

Some potential testing scenarios might be:

Natural Disasters

Hurricane or heavy downpour of rain

Tornado or high winds

Earthquake

Flood

Pandemic

Fire

Severe snow or ice

Civil Crises

Labor strike (in company or secondary picketing)

Workplace violence

Serious supplier disruption

Terrorist target neighbor (judiciary, military, federal, or diplomatic buildings)

Sabotage/theft/arson

Limited or no property access

Location Threats

Nearby major highway, railway, or pipeline

Hazardous neighbor (stores or uses combustibles, chemicals, or explosives)

Offices above 12th floor (limit of fire ladders)

Major political event that may lead to civil unrest

Network/Information Security Issues

Computer virus

Hackers stealing data

Data communication failure

Data Operations Threats

Roof collapse (full or partial)

Broken water pipe in room above data center

Fire in data center

Critical IT equipment failure

Environmental support equipment failure

Telecommunications failure

Power failure

Service provider failure

Loss of water pressure which shuts down chilled water coolers

Select scenarios so that the problem exercises multiple plans. Choosing the right scenario can engage the participant’s curiosity and imagination. It converts a dull exercise into a memorable and valuable experience for its participants. A good scenario should:

Be realistic—no meteors crashing through the ceiling.

Be broad enough to encompass several teams to test their intergroup communications.

Have an achievable final solution.

Include time increments, such as every 10 minutes equals one hour, etc.

Prior to the exercise, draft the scenario as a story. It begins with an initial call from the alarm monitoring service with vague information—just like a real incident. To add to the realism, some people will use a bit of photo editing to illustrate the scene. Imposing flames over the top of a picture of your facility may wake some people up!

As the exercise continues, the Exercise Coordinator provides additional information known as “injects.” This predefined information clarifies (or confirms) previous information and also raises other issues incidental to the problem. For example, if there was a fire in the warehouse, an inject later in the exercise may say that the warehouse roof has collapsed injuring several employees or that the fire marshal has declared the warehouse to be a crime scene and the data center is unreachable until the investigation is completed in two days.

Injects, like the scenario, must make sense in the given situation and may also include good news, such as workers missing from the warehouse fire are safe and have been found nearby. Unplanned injects may be made during the exercise if a team is stumped. Rather than end their portion of the test, state an assumption as fact. For example, the Exercise Coordinator could state that, “The data center fire was concentrated in the print room and no servers were damaged.”

Try to insert some humor into a tragic situation. For example, state that a fire started by a lightning strike in the boss’s office or the Board of Health condemned the food vending machines or no one can enter the building.

TYPES OF EXERCISES

There are various types of recovery plan tests. They range from easy to set up and quick to complete to full simulation requiring months of planning. Each plan starts with Standalone Testing. Unfortunately, many companies never test their plans in a full simulation.

Standalone Testing

Standalone Testing is the first level of testing for all recovery plans. It is also required when a significant change has been made to the IT system of a business process.

Standalone Testing exercises individual IT components or business processes to estimate the time required for recovery. It provides the first level of plan error checking. The scenario of a Standalone Test is to recover an individual IT component or business process from nothing. (It assumes the process or IT system has been destroyed or rendered totally unusable.) A recovery plan is written so that someone other than the primary support person can understand and follow it. It also familiarizes at least one other person with the plan’s contents.

Recovering business processes often requires that many plans work together. Standalone Testing examines the individual building blocks of the overall effort. Later tests examine the interactions and interfaces among the individual plans.

The result of a Standalone Test should be a recovery plan that is in the company standard format. This ensures that anyone unfamiliar with this process can find the same type of information in the same place. The plan should be approved as complete and accurate by the plan’s author and plan reviewer. The plan’s author also provides a time estimate as to how long the recovery plan should take to complete.

PREPARATION Schedule a conference room away from an office’s distractions. If the document is large, break it into one-hour meetings to keep everyone fresh.

MATERIALS TO PROVIDE A copy of the standard plan format and copies of the plan for each participant.

TESTING TEAM Consists of the document author and a reviewer (may be the backup support person or the process’s owner).

THE MEETING AGENDA The meeting agenda should be as follows:

Review ground rules.

This is a draft document and anyone can suggest changes.

Suggesting a change is not a personal attack.

All comments are focused on the document and not on the author.

Review document for:

Proper format.

Content.

Clarity.

Estimate time required to execute each step in this plan and the plan overall.

Set time for review of changes suggested by this test.

FOLLOW UP Continue Standalone plan reviews until the document conforms to the company standard format and the participants believe that the document reflects the proper recovery process.

Integration Testing

Integration testing (or Integrated System Testing) exercises multiple plans in a logical group. This might be an IT system with its interdependent components (a database server, an application, special network connections, unique data collection devices, etc.).

The purpose of an Integration Test is to ensure that the data exchanges and communication requirements among individual components have been addressed. These interdependent components require each other to provide the desired business function. This type of test is normally used by IT systems. For example, the Order Entry system may require access to multiple databases, files, and applications. To test the recovery of the Order Entry system, all of the three other components must be recovered first.

The ideal place to execute this plan is at the IT recovery hot site that the company will use in a crisis. If that is not available, then use equipment that is as close in performance and configuration to the hot site as practical. This will help to identify differences between the hot site and the required data center configuration.

Another advantage to using the hot site is to provide an actual recovery time for validating the recovery time objective (RTO). This result is added to other test results to see what the company can realistically expect for a recovery time, given the current technology.

Integration testing is usually conducted by the backup support person(s) for each recovery plan. The Business Continuity Manager observes the test and records the actual time required to recover the IT system or business process.

In most IT recoveries, the server administrator builds the basic infrastructure and then provides it to the recovery team. For example, an operating system is loaded onto “blank” servers and then turned over to the recovery team. The time to prepare these devices is part of the RTO calculation.

PREPARATION Schedule time at the hot site or the use of equipment in the data center.

TESTING TEAM The testing team should consist of the following people:

Backup support person(s) for each device to be recovered.

Network support technician to isolate the test network from production and to load the DNS server.

System administrator to load the operating systems and establish the domain controller.

Applications support team to load and test their systems.

Optionally, a database administrator.

Reviewer (IT Manager or Business Continuity Manager).

Business process owner to validate a good recovery.

Business continuity program sponsor.

MATERIALS TO PROVIDE These include the following:

The business continuity program sponsor approves timing of test and required funding.

Nonproduction (spare) IT equipment, based on the list of required equipment as detailed in the recovery plans.

Copies of the recovery plans to be tested.

THE TEST PROGRAM The program should include the following:

Review ground rules.

Write down all corrections as they are encountered.

Record the amount of time required to complete each step in the plan and the total in the plan. This may isolate steps that take a long time as targets for improving the speed of the recovery.

Given the amount of time required to set up an Integration Test, if time permits, rerun it after the plans have been corrected.

The support team (network, database, systems administrators) remains close at hand to address problems after the applications recovery begins.

Prepare for the test.

Set up a network that is isolated from the world since some applications may have embedded IP addressing.

Conduct the test.

Set up the infrastructure.

Set up required infrastructure, such as DNS and domain controllers.

Load a basic operating system on the recovered servers.

Provide adequate servers and disk storage space.

Using the recovery plans, follow each step.

Note all corrections.

Using these corrections, restart the test from the beginning.

Once the system is ready:

Applications support runs test scripts to ensure the system has been properly recovered.

A business process owner validates that it appears to function correctly.

Review the results.

Update any plans as required.

Determine a realistic recovery time for this process.

Conduct the after-action review.

Identify plan improvement needs.

Identify areas to research to reduce the recovery time.

Identify improvements in the testing process.

Business Continuity Manager writes a report of test results to the program sponsor.

Set time for review of changes.

FOLLOW UP Collect all plan corrections and reissue updated documents. If a plan required significant changes, then it should be reviewed in a Standalone Test before using it in another Integration test.

Just as recovery plans are exercised, so is your ability to plan and conduct a test. After the plans are updated, ask the participants to review the planning and testing processes for ways to improve them.

Walk-Through Testing

The purpose of a Walk-Through Test is to test a logical grouping of recovery plans at one time. It is similar to an Integration test except that no equipment is involved and the recovery is theoretical. A Walk-Through recovery plan exercise familiarizes recovery team members with their roles. It is useful for rehearsing for an Integration test, for testing when an Integration test is not practical, and for reviewing business process recovery plans.

Integration testing is valuable since it involves an actual recovery. A Walk-Through also provides many benefits, but without the expense of actually using equipment.

Walk-Through recovery plan testing is conducted in a conference room. Participants explain their actions as they read through the recovery plan. The goals are to improve plan clarity, to identify gaps in the plans, and to ensure that all interfaces among individual plans are addressed. These interfaces may be the passing of data from one IT component to another or the passing of a document between workers.

A Walk-Through Test does not provide a real RTO for the collective plans. However, estimates may be provided by the recovery team members.

PREPARATION Schedule time in a conference room.

TESTING TEAM The testing team should consist of the following people:

Backup support person(s) for each plan to be recovered.

Exercise Coordinator (IT Manager or Business Continuity Manager).

Business process owner.

Exercise recorder to capture action, decisions, and assumptions as they occur.

MATERIALS TO PROVIDE Include copies of the recovery plans to be tested.

THE TEST PROGRAM The program should include the following:

Conduct the test, following the recovery plans.

Set up the infrastructure.

Set up required infrastructure, such as DNS and domain controllers.

Load a basic operating system on the recovered servers.

Provide adequate servers and disk storage space.

Note all corrections.

Review the results.

Update plans as required.

Team members are now more familiar with their recovery roles.

Conduct the after-action review.

Identify plan improvement.

Identify areas to research to reduce the recovery time.

Identify improvements in the testing process.

Business Continuity Manager writes report of test result to program sponsor.

Set time for review of changes.

FOLLOW UP Collect all plan corrections and reissue updated documents. If a plan requires significant changes, then it should be reviewed in a Standalone Test before using it in another Walk-Through exercise.

Simulations

Up to this point, all tests have been based on recovering a business process or IT system from scratch. The reasoning is that if the plan has adequate information to recover from nothing, then it will have the information necessary to recover from a partial failure. However, it is this partial failure that is more common.

A simulation test brings all of the plans together. In a real crisis, rarely is the recovery isolated to a single plan. IT systems recover the data center, work area recovery plans recover office processes, and the supporting plans for Human Resources, Corporate Communications, Facilities, Security, and a range of other departments are all in play. A simulation not only invokes these many plans but forces them to work together toward the common goal.

A simulation test begins with a scenario (such as a partial roof collapse from a severe storm or a person entering the building with a gun). In both of these examples, most of the facility is intact yet may be temporarily disabled.

Simulation adds to plan exercises the elements of uncertainty, time pressure, and chaos. No situation comes with complete and verified information, yet managers must react correctly to minimize damage to the company. Chaos comes from inaccurate and incomplete information yet decisions must be made. Unlike the smooth pace of a Walk-Through Test, simulations add to the element of chaos in which events surge forward whether someone is ready for it or not.

Simulation tests can be simplistic, such as Table-Top exercises. They can also be complex (and expensive), such as relocating the entire data center or work area to the recovery site and running the business from there. Most simulations only address a portion of the company, usually a group of related business processes. This keeps the recovery team to a manageable size and the recovery exercise focused on a set of plans.

Make it fun! Send out pre-exercise announcements as if they were news elements related to the scenario (clearly marked as exercise notices for training only). At the beginning of the exercise, state the goals to instill a sense of purpose in the group. At the end of the exercise, restate the goals and ask the group how well it measured up. After all, these people gave up some hours of their lives, so show them how important it was to the company!

Simulations can also add the dimension of external agencies to the recovery. Firefighters, reporters, police officers, and other emergency groups can be invited to keep the chaos lively while educating participants of each agency’s role in a crisis. The Exercise Coordinator may also include employees at other company sites via conference call.

The purpose of the exercise is to validate that the plans are workable and flexible enough to meet any challenge. Participants will depend on the plan to identify actions to take during an incident. (However, just as in a real crisis, they are free to deviate from them.) Notes will be collected, and the plan will be updated as a result of the exercise. Participants will note areas for improvement such as corrections, clarity, content, and additional information.

There is no “right” answer to these exercises. The goal is to debug the plans and seek ways to make them more efficient without losing their flexibility (since we never know what sorts of things will arise). “Rigging the test” to ensure success should not be done. Conducting the exercise at the recovery site will minimize distractions from electronic interruptions.

About one week before the exercise, verify that participants or their alternates are available. This is also a good time to rehearse the exercise with the testing team and to handle minor administrative tasks such as making copies of plan and tent cards identifying participants and their roles.

People will react in different ways. If someone on the team will be declared injured or killed during the exercise, ensure that they agree to this prior to the start of the exercise.

Table-Top Testing

A Table-Top Test is a simulated emergency without the equipment. It exercises decision making: analysis, communication, and collaboration are all part of the plan. Table-Top exercises test an incident management plan using a minimum of resources. The size of the incident is not important. It is the fog within which early decisions must be made until the situation becomes clearer.

A Table-Top exercise tests a logical grouping of recovery plans with a realistic disaster scenario. One or more conference rooms are used to control the recovery. The primary difference between a Walk-Through Test and a Table-Top Test is that a Table-Top Test uses a scenario, midexercise problem injections, and, often, external resources.

A Table-Top exercise is much less disruptive to a business than a full simulation. A Table-Top exercise typically runs for a half day, where a full simulation can run for several days.

The goals are to train the team members, identify omissions in the plans, and raise awareness of the many dimensions of recovery planning. Each participant uses the recovery plan for guidance but is free to choose alternative actions to restore service promptly. The goals are to improve plan clarity, to identify gaps in the plans, and to ensure that all interfaces between individual plans are addressed. These interfaces may be the passing of data from IT component to another or the passing of a document between workers.

A Table-Top Test does not provide a real RTO for the collective plans. However, estimates may be provided by the recovery team members.

The exercise coordinator keeps the group focused on the test. It works best if someone else is designated as the exercise recorder. The recorder writes down the events, decisions, and reactions during the exercise, freeing the exercise coordinator to work with the team. These notes are valuable later when considering ways to improve the recovery plans and the Table-Top exercise process.

PREPARATION Schedule a conference room.

TESTING TEAM The testing team should consist of the following people:

Backup support person(s) for each plan to be recovered.

Exercise Coordinator (IT Manager or Business Continuity Manager).

Exercise recorder.

Business process owners.

External resources, such as news reporters, firefighters or police officers.

MATERIALS TO PROVIDE Materials to provide include:

Copies of the recovery plans to be tested.

Scenario and incident “injects.”

A clock projected by a PC onto a whiteboard.

THE TEST PROGRAM The program should include the following:

Explain to participants the rules for the exercise:

Time is essential; decisions must be made with incomplete information.

Everyone must help someone if asked.

Everyone takes notes for the after-action critique.

No outside interruptions are permitted—cell phones off.

If an issue is bogging down the exercise, the Exercise Coordinator can announce a decision for the issue or set it aside for future discussion.

Introduce each of the team members and explain their role in the recovery.

State the exercise goals (familiarize the team with the plan, gather RTO data, improve the plans, etc.).

Introduce the scenario to the team.

Clarify group questions about the situation.

Ensure everyone has a copies of the appropriate plans.

Conduct the exercise.

Select several of the primary recovery team members to step out of the exercise; their backup person must continue the recovery.

Inject additional information and complexity into the exercise every 10 minutes.

End the exercise at a predetermined time, or when the company is restored to full service.

Conduct the after-action review.

Identify plan improvements.

Identify areas to research to reduce the recovery time.

Identify improvements in the testing process.

Business Continuity Manager submits a written report of the test result to the program sponsor.

FOLLOW UP Collect all plan corrections and reissue updated documents.

Two types of plans are best tested as Table-Top exercises. A crisis management plan is easily tested in a conference room. The types of actions required can be discussed rather than acted out. A pandemic can range over 18 months, so a full simulation is not practical. Both can be conducted in the Command Center for additional realism.

Disaster Simulation

The purpose of a Disaster Simulation is to test a logical grouping of recovery plans with a realistic scenario. Essentially, a Disaster Simulation is a simulated emergency that includes the people and equipment necessary to recover IT equipment or a wide range of business processes. Running a simulation is expensive in time and equipment, so it should be approved far in advance. A simulation may be disruptive to a company’s normal business and should be planned for the company’s slow time of the year. It may run for several days.

The goals are to train the team members, identify omissions in the plans, and to raise awareness of the many dimensions of recovery planning. A key advantage of a simulation is that it provides the actual time required to recover a process. It also adds in the pressure of chaos to the recovery.

The Exercise Coordinator keeps the group focused on the test. Appoint an exercise recorder for each recovery team. That person writes down the events, decisions, and reactions during the exercise. These notes are valuable later when considering ways to improve the recovery plans and the exercise process.

Always preannounce a simulation; there should be no surprise alerts. Before engaging outside participants, the DR core team should perform the simulation exercise, as a dress rehearsal to “polish” the sequence of events.

Real tests provide the most realistic results. Avoid the temptation of the IT team to make a “special” set of backup media just for the test. The true recovery time comes from sifting through the many backup tapes to find the files that you need.

A simulation begins with the initial incident alert by the night watchman or by an alarm that automatically alerts a manager. Full-scale testing involves pulling the plug on some part of the operation and letting the disaster recovery plan kick in. For obvious reasons, this is rarely done.

Simulation tests should be conducted at the recovery site at least once per year. Recovery plans are used as guidelines, but participants are free to deviate from them. The goals are to improve plan clarity, to identify gaps in the plans, and to ensure that all interfaces between individual plans are addressed. These interfaces may be the passing of data from one IT component to another or the passing of a document between workers.

PREPARATION Schedule a conference room. Ensure the participants understand the exercise is a rehearsal and not a test. A rehearsal allows people to play out their actions; a test implies pass or fail. For each recovery team:

Create a log sheet to document the communication among recovery teams (see Form 13-1 on the CD).

Create an observation log (see Form 13-2 on the CD).

TESTING TEAM The testing team should consist of the following people:

Backup support person(s) for each plan to be recovered.

Exercise Coordinator (IT Manager or Business Continuity Manager).

Business process owners.

Exercise recorder.

External resources, such as news reporters, firefighters or police officers.

MATERIALS TO PROVIDE Materials to provide include:

Copies of the recovery plans to be tested.

Scenario and incident “injects.”

A clock projected by a PC onto a whiteboard.

FIGURE 13-2: RTO Hour-by-Hour Recovery Plan for a Data Center.

THE TEST PROGRAM The program should include the following:

Explain to participants the rules for the exercise:

Time is essential; decisions must be made with incomplete information.

Everyone must help someone if asked.

Everyone should take notes for the after-action critique.

No outside interruptions are permitted—cell phones off.

If an issue is bogging down the exercise, the exercise coordinator can announce a decision for the issue or can set it aside for future discussion.

Introduce each of the team members and explain their role in the recovery.

Introduce the scenario to the team.

Clarify group questions about the situation.

Conduct the exercise.

Select several of the primary recovery team members to step out; their backup person must continue the recovery.

Inject additional information and complexity into the exercise every 10 minutes.

End the exercise at a predetermined time, or when the company is restored to full service.

Conduct the after-action review.

Identify plan improvements.

Identify areas to research to reduce the recovery time.

Identify improvements in the testing process.

Collect RTO metrics.

The Business Continuity Manager gives a report of the test result to program sponsor.

Set time for review of changes.

FOLLOW UP Collect all plan corrections and reissue updated documents. If a plan required significant changes, then it should be reviewed in a Standalone Test before using it in another Integration test. In addition, update the RTO Hour-by-Hour Recovery Plan.

SOMETIMES NATURE TESTS THE PLANS FOR YOU

There are numerous incidents that pop up from time to time that are not significant emergencies but that provide an opportunity to test parts of a plan. For example, if there is a power outage at work, use the plans to minimize the disruption. Do the same for a loss of data communications, a tornado warning, or a snow storm emergency. Relocating a business process or significant portion of the data center is similar to a disaster.

Another opportunity that can trigger a test plan is facility construction. An example of this is if the electricity to a building must be turned off for work on the power main. Use the recovery plans to locate and turn off all of the equipment, noting anything found that was not in the plan. When the work is over, use the plans to turn back on all of the equipment. Then test each critical system to ensure it is operational. Following the plans for restarting equipment may uncover equipment tucked away in offices or closets that are not in the plan.

Relocation to a new facility is a great opportunity to completely test your disaster recovery plan. Many of the activities necessary during relocation are the same as those required in a disaster: new machines may need to be purchased, servers are down for some period of time, new communications infrastructure needs to be built, data need to be restored, etc. In fact, if a relocation project is not done properly, it may turn into a real disaster!

Whenever such a problem occurs:

Focus people on referring to their recovery plans. The value of a plan is to reduce chaos at the beginning of a crisis. Plans are no good if no one uses them.

Begin recording what has occurred and people’s reaction to it. These notes are used to improve our plans (and never to criticize anyone).

Conduct an after-action review the next day to gather everyone’s perspective.

A plan that is used for a real event has been tested just as surely as a scheduled exercise. Mark that plan as tested for the quarter!

Whenever a significantly disruptive incident occurs, such as a power outage, loss of external network, or a computer virus outbreak, begin taking notes during the event. These notes should be a narrative of times and actions taken—who did what, when, and the result. See if anyone thought to break out the appropriate recovery plans and follow them.

Within two working days after the incident, convene a group to conduct an after-action review. This review is intended to capture everyone’s perspective of the incident to improve plans for future use.

Debriefing Participants Using an After-Action Review

Whenever an incident occurs (such as a power outage, fire in the computer room, etc.) that is covered by a recovery plan (or should have been covered by a plan), conduct an after-action review on the next work day after the recovery. This is an open discussion of the event and how to improve future reaction.

Someone is appointed as the review coordinator (usually the Business Continuity Manager). It is helpful if someone else records the discussions, so that the review coordinator is free to focus on the discussion.

What happened – it is important to gain agreement on what occurred, as further discussion is based on this. Each person will define the problem from his or her own perspective. Sometimes agreement on a point takes a lot of discussion.

What should have happened – this is where positive things are listed, such as the recovery plan was easy to find, etc.

What went well – not all is doom and gloom. Now that the crisis has ended, take credit for the things that went well. Acknowledge those people who contributed to the recovery.

What did not go well – this is the substance of the review. Once you list what did not work out, you can move to the last step. Take care never to personalize the discussions. Focus on the action and not on a person. Otherwise, people become defensive and no one will participate in the discussion.

What will be done differently in the future – list the solution to each item identified in the previous step. Assign action items to specific people each with a due date.

An example after-action report for a power outage:

What happened – The power for the building went out and everyone stopped working. The office people flooded out to the factory because there was light there through the windows. People milled around outside of the data center to see if they could help. The emergency lights failed in most of the offices and everyone was in the dark.

What should have happened – The emergency lights should have worked. Everyone should have known where to meet for further instructions.

What went well – Nobody panicked. The UPS system kept the data center running until power was restored.

What did not go well – No one knew what to do. Different people were shouting out different directions, trying to help but really confusing everyone. We could not find the system administrators in case the servers needed to be turned off.

What will be done differently in the future – We will identify assembly areas for everyone. Supervisors will be responsible for finding out what has occurred and passing it on to their people. The emergency lights will be checked monthly.

DEMONSTRATING RTO CAPABILITY

During the Business Impact Analysis, an RTO was established by the company. It was selected based on the impact to the company, not on what the company was capable of doing. Testing recovery plans and recording the time is the place where the company proves it can meet the RTO. If not, something must change to meet it.

Some RTOs are obvious. A company that expects to recover from tape backup requires days. If the organization requires recovery in a few hours, then stop the testing and rework the data storage strategy. However, if a company has a reasonable strategy based on its RTO, then only testing can prove if this is achievable or not.

Figure 13-2 shows the first page of a possible RTO Hour-by-Hour Recovery Plan for a data center. (A similar chart should be built for work area recovery.) This chart collects the recovery times from plan exercises. In the IT world, most recoveries must wait until the basic infrastructure is in place (network, firewalls, DNS, domain controllers, etc.). The plan for recovering each infrastructure component is placed in sequence at the top of the chart. Below that are the applications, databases, etc., that must be recovered in the appropriate sequence. For example, a LAN recovery must be in place before the domain controller can be recovered.

Use this basic plan as an outline for building your own recovery timeline. As you enter the times from actual system recoveries, you can prove or disprove the company’s ability to meet its desired RTO. Actual recovery times are always preferred to estimated values.

If the RTO is not achievable or if you wish to shorten it, use this plan to identify places to make changes. Look for tasks that could run in parallel instead of sequentially. Look for the ones that take a long time and seek ways to shorten them (use faster technology or redesign the process). Of course, the best way to reduce the time required is to eliminate noncritical steps.

During a recovery, company executives can use the RTO Hour-by-Hour Recovery Plan to follow along with the recovery’s progress. Based on where the team is in the recovery, they can look at the times and see how much longer before (for example) the e-mail system should be available, or that the billing system should be operational.

CONCLUSION

No plan can be called complete until it has been tested. Beyond the initial testing, ongoing testing is critical to ensure that the plan is kept up to date. As the organization grows and evolves, the plan must be updated to incorporate the necessary changes. Periodic testing validates these changes and keeps everyone aware of their responsibilities when a disaster strikes.

There are different types of tests from simple one-on-one Standalone Tests to full simulated disasters. Tests should follow a progression from simple tests to complex. Trying to jump too quickly into simulations will result in people sitting around while muddled plans are worked through. Participants will conclude that the tests themselves are the disaster.

The people participating in the tests are a valuable source of information. After each exercise, promptly gather their ideas in an after-action meeting. They should advise the Exercise Coordinator of ways to improve the plans, communications among the testing teams, and everything that can speed a recovery. In a separate meeting, ask them to critique the testing process. This will improve their participation and cooperation in the future, as well as make your tests run smoother.

There are times when company activities or Mother Nature tests your plans for you. Immediately focus everyone on using their plans. After the event has passed, pull everyone together for an after-action meeting to collect their ideas. (This is also a great time to slip in a plug for the value of plans when disaster strikes.)

The outcome of each test should be used to update an RTO Hour-by-Hour Recovery Plan. It is one thing for a company to declare an RTO but that chart illustrates whether it is likely or not.

CHAPTER 14

ELECTRICAL SERVICE

Keeping the Juice Flowing

Nothing shocks me. I’m a scientist.

—Harrison Ford, as Indiana Jones

INTRODUCTION

This chapter is provided so you will have a basic understanding of electrical power support for your critical equipment. Use the information as a background when talking to your facility electrical engineer and your UPS supplier. While you’ll not want to work on high-voltage circuits or equipment yourself (leave that to trained professionals), an overall knowledge of how your electrical systems work will help you to write a better plan.

ELECTRICAL SERVICE

Imagine a business in which if you create too much product it is immediately lost forever. A business where people only pay for what they use, but demand that all they want be instantly available at any time. A product they use in varying amounts throughout the day. A product that requires an immense capital investment but is sold in pennies per unit. Welcome to the world of electricity. Electrical service is so reliable, so common, that people take it for granted that it will always be there whenever they want it. Electricity is an essential part of our everyday existence. Few businesses could run for a single minute without it.

Side-stepping the issue of the huge effort of the electric company to ensure uninterrupted service, let’s consider the impact of electricity on our business. Without a reliable, clean source of electric power, all business stops. We have all experienced an electrical blackout at some point. When we add together how important electricity is and that we believe a blackout is likely to occur again, we meet all the criteria for requiring a disaster recovery plan. Because we cannot do without it and there are economically feasible disaster containment steps we can take, a mitigation plan must also be drafted.

In addition to recovering from an outage, our mitigation plan will reduce the likelihood of losing power to critical machinery. There are many other problems with electrical power beyond whether we have it or not. Therefore our mitigation plan must address ensuring a clean as well as a reliable electrical supply.

In the case of electricity, we need a process that:

Monitors the line and filters out spikes.

Provides additional power in case of a brownout or partial outage.

Provides sufficient temporary power in case of a total outage.

Makes the transition from normal power supply to emergency power supply without loss of service to critical devices.

Whatever power support plan you select, keep in mind that it must be tested periodically. With luck, you will be able to schedule the tests so that a power failure will have minimal impact. With a touch of bad luck, nature will schedule the power outages for you and, again, at that time you will know how well your power support plan works.

RISK ASSESSMENT

What sorts of problems are we protecting against? In an ideal situation, North American electricity is provided at 120 volts, 60 cycles per second, alternating current. If we viewed this on an oscilloscope, the 60 cycles would display a sine wave. There are many variations from “normal” that will play havoc with your reliable power connection.

The most common electrical power problem is voltage sag, or what is more commonly known as a “brownout.” Generally speaking, this is a reduced voltage on the power line; you can see a brownout when the room lights dim. They can cause some computer systems to fail and occasional hardware damage by forcing equipment power supplies to work harder just to function.

Sags can be caused by turning on power-hungry equipment. As they begin operation, these power hogs draw the amount of electricity they need to run from the power grid. This sudden electrical load causes a momentary dip in the line voltage until the electric company compensates for it. These power drains might be anything from heavy machinery to the heater under the desk next door. Most sags are of short duration.

Brownouts can also be caused by utility companies switching between power sources and, in some situations, it can be an intentional voltage drop by the electrical company to cope with peak load conditions. An example is the summer of 2001 power crisis in California with its rolling brownouts when the electrical utility company could not meet peak demand. The demand for electrical power is growing, but the supply of electricity is not.

Once a voltage sag ends, there is typically a corresponding “spike” of “overvoltage” that can further damage equipment. Sharp or extended overvoltages can severely damage your electronic systems, which are not designed to receive and handle large voltage variations.

Another common electrical power problem is a voltage surge, which is a short-term substantial increase in voltage caused by a rapid drop in power requirements. A typical surge lasts for 3 nanoseconds or more (anything less is known as a spike). Surges are caused by major power users being switched off. For that brief moment, the power available for that item is still being supplied but is no longer needed and must be absorbed by other devices on that line. Examples of large users that may be switched off are factory equipment, air conditioners, and laser printers.

Surges frequently occur and usually go unnoticed. Some can be handled by the equipment’s power supply, some must be absorbed by a surge protector, and the rare major surge will wipe out anything in its path. A common example of a major power surge is a lightning strike that surges down power and telephone lines into nearby equipment.

Noise is seen as jitters riding along on the 60-cycle sine wave. It is electrical impulses carried along with the standard current. Noise is created by turning on electrical devices, such as a laser printer, an electrical appliance in your home, or even fluorescent lights. Did you ever see “snow” on your television screen when using an electrical appliance? That is an example of line noise sent back into your electrical system. What you see on the screen is electrical noise riding on your local wiring that is too powerful for your television to filter out.

Noise is one source of irritating PC problems, such as keyboard lockups, program freezes, data corruption, and data transfer errors. It can damage your hard drives and increase audio distortion levels. The worse part of the problem is that in many cases you aren’t even aware of what is happening when it occurs.

Voltage spikes are an instantaneous increase in line voltage that is also known as a “transient.” A spike may be caused by a direct lightning strike or from the return of power after a blackout. Think of a spike as a short-duration surge that lasts for 2 nanoseconds or less. Spikes can be very destructive by corrupting data and locking up computer systems. If the spike hitting the device is intense, there can be significant hardware damage.

An electrical blackout is a total failure of electrical power. It is any voltage drop to below 80 volts since, at that point, most electrical devices cease to function. Blackouts have a wide range of causes from severe weather to auto accidents to electrical service equipment failures.

A blackout immediately shuts down your equipment, and it is time consuming to restart most machinery from a “hard stop.” Even though most blackouts are of a very short duration, from a business perspective a momentary blackout can be just as serious as a 2-hour outage. In addition, some equipment may not have been turned off for years—and for good reason. There may be some doubt as to whether it will even start again!

Blackouts are very damaging to computer systems. Anything residing in memory, whether it is a spreadsheet or a server’s cache, is immediately lost. Multiply this across the number of people working in a single building, and you can see the lost time just for one occurrence. Compounding the data loss is the damage and weakening of your equipment. A further issue is that, from a recovery perspective, there may be network devices working out of sight in a closet deep within the building. If you don’t know these exist or where to find them, just the process of restarting equipment can be very troublesome.

When recovering from blackouts, beware of the corresponding power surge that accompanies the restoration of system power. So when a blackout strikes, turn off your equipment and do not restart it until a few minutes after power is stabilized.

YOUR BUILDING’S POWER SYSTEM

Many years ago, your company’s delicate data processing equipment was concentrated in the facility’s data processing center. Often one whole wall was made of glass so everyone could see this technical marvel in action (hence the term “glass house” for computer rooms). This concentration allowed the equipment to be supported by a few Uninterruptible Power Supply (UPS) units and power line conditioning devices.

Now, the primary computing muscle for most companies is spread all over the facility in the form of PCs and departmental servers. Instead of a carefully conditioned and electrically isolated power feed, your equipment shares the same power circuits as soda pop machines, copiers, and factory machinery—all of which add noise and surges to the power line. None of this is good for your computer systems and network devices!

This variety of computing power creates a need to monitor electrical service to ensure maximum network and computer capabilities. The emphasis is on network because while personal computers are located comfortably on office desktops, network hubs, routers, and bridges can be found stuffed in any closet, rafter, or crawl space, or under a raised floor. This makes the automatic monitoring of electrical service across your facility an important network management function.

Filtering the electricity as it enters your building is a good practice to minimize external influences. Sometimes, however, the problems are caused by equipment inside your building; this might be arc welders, heavy machinery, etc.

BUILDING A POWER PROTECTION STRATEGY

Power protection for business continuity is a five-step process.

1. The first step is to isolate all your electronic equipment from power surges by use of small surge protectors. Power surges sometimes occur internal to your building. Surge strips are inexpensive and simple to install.

2. The next layer is line conditioning. A line conditioner smoothes out voltage variation by blocking high voltages and boosting the line voltage during brownouts. This filtering should always be applied to the power line before electricity is passed to your UPS.

3. An Uninterruptible Power Supply provides electrical power for a limited time during the event of a power outage. The UPS battery system can also help to protect against brownouts by boosting low voltages. A UPS is a critical device for ensuring that key components do not suddenly lose electrical power.

4. One of the best solutions for companies that cannot tolerate even small power outages is an onsite electric generator. These backup units instantly start and begin generating electricity to support your facility. Imagine a hospital’s liability if all their life support equipment suddenly stopped from a lack of power. This added security is not cheap to install or maintain. Keep in mind that electrical generators of this sort are internal combustion engines and must conform to local air pollution and building regulations.

5. The last step, not directly related to electricity, is the physical security of the electrical support equipment. Few people require access to this equipment, and it must be safeguarded against sabotage. This equipment is unique in that if someone disabled it, the entire facility could stop with lost production quickly running into the thousands of dollars per minute. Additionally, the “hard stop” on machinery and computer servers will result in lost or corrupt data files.

However you secure this equipment, keep in mind the cooling and service clearance requirements for the UPS system. The UPS control panel must also be available to the disaster containment team in a crisis.

Surge Protection

One of the most common electrical protection devices is a surge protector power strip. Computer stores sell these by the bushel. For your dispersed equipment, a surge protector can provide some measure of inexpensive protection. A typical surge protector contains circuitry that suppresses electrical surges and spikes. All electronic devices should be attached to electrical power through a surge strip to include all your PCs, network equipment, printers, and even the television used for demonstrations in the conference room. Even if your facility’s power is filtered as it enters the building, a direct lightning strike can ride the wires inside the building and still fry your equipment.

There are many brands of surge suppressors on the market. There are places to save money and places to lose money. The old saying goes “for want of a nail, the battle was lost.” When protecting your equipment from power problems, you may not want to skimp too much. Here are some things to look for when buying a surge strip:

Joule Ratings. A joule rating is a measure of a surge protector’s ability to absorb power surges. A joule is a unit of energy equal to the work done by a force of 1 Newton through a distance of 1 meter. Generally, the higher the rating the better. A good surge suppressor will absorb between 200 and 400 joules. If greater protection is needed, look for a surge suppressor rated at least 600 joules.

Surge Amp Ratings. The amount of above-normal amps the surge protector can absorb. As with joules, the higher the better.

UL 1449 Voltage Let-Through Ratings. Underwriters Laboratories™ tests to determine how much of a surge is passed by the surge protector on to the equipment it is protecting. The best rating is 330 volts. Any voltage rating less than 330 adds no real benefit. Other ratings of lesser protection are 400 and 500. Be aware that UL 1449 safety testing does not test for endurance.

Response Time. The response time of the surge protector is important. If it blocks high voltages but is slow to react, then it is of marginal usefulness. Adequate response time is 10 nanoseconds or less. The lower the number, the better.

All-Wire Protection. A high-quality surge protector guards against surges on the ground wire, as well as the current-carrying wires.

Telephone Line Support. To protect your modem from power surges riding on the telephone wire.

Clamping Voltage. The voltage at which the surge suppressor begins to work. The lower the rating, the better. Look for a rating of 400 volts or less.

Some surge protectors provide basic line conditioning against noise on the line. This circuitry can smooth out minor noise from the lines.

An interesting thing about the ubiquitous surge protector strips is that in addition to protecting equipment, they make handy extension cords. Over time, these surge strips may silently have absorbed any number of electrical “attacks” that have eroded or destroyed their ability to protect your equipment. Most people cannot see this because they still function quite nicely as extension cords.

Surge protectors often have lights to tell you when they are energized or not. High-quality surge protectors will have an additional light to let you know if their surge-fighting days are over. This light may say something like “protected,” “surge protection present,” etc. If your surge protector has such a light and it is no longer lit when running, then you may have a false sense of security that it is functioning as something other than an extension cord, because that is all it now is!

People traveling around the country using a notebook computer would be well advised to carry and use high-quality surge protectors when plugging their equipment into the local power grid. When you consider how big a typical surge suppressor is and how tiny notebook PCs have become, you can guess at how little room there is for surge suppression circuits in the notebook PC chassis. This is especially important for international travelers, as the power in some countries is a bit rougher than it is in the United States.

A few more things to consider when using surge protectors. As great a tool as they are, they cannot stop a nearby lightning strike from damaging your equipment. When a lightning storm approaches, unplug both the power strip and your network cable from the wall. (This is good advice for any sensitive electronic equipment that depends solely on a surge protector to defend against lightning.)

Also, never use a ground eliminator with a surge strip (a ground eliminator converts a three-prong plug into a two-prong plug for use in an older building). Doing so will make it difficult if not impossible for your surge protector to resist a major line surge.

Line Conditioning

Line conditioning ensures that your equipment always receives the same steady voltage. It also screens out noise on the power waveform. Line conditioning involves passing your normal electrical service through filtering circuitry before it is used. Many people don’t realize that the “old reliable” electricity that magically comes out of the walls is susceptible to a wide range of influences. These influences take the “pure” 60-cycle alternating current and introduces fluctuations in the voltage or current as it passes down the line.

These fluctuations can have many sources but one that we especially want to avoid is lightning. Lightning can cause a localized one-shot power surge to roar down the electrical line into your equipment. When this happens, equipment power supplies and integrated circuits can quickly melt.

Line conditioning is also advised for analog telephone lines connected to PC modems. The same lightning strike that induces an electrical charge in your electrical power lines can throw a jolt down your telephone line. Unfortunately, PC modems have little protection against a power surge, and they are very easily destroyed. Many surge suppressors now include a telephone line surge suppression jack to filter these problems out.

A line conditioner should always be installed between a UPS and the electrical power source to reduce the load on the UPS batteries. Some UPS units include a line filtering capability. Check your model to see what it is capable of doing. A line conditioner reduces the number of times that the UPS jumps on and off of battery power (which shortens the life of your batteries).

A line conditioner is an essential component when generating your own emergency power. Use it to filter the electricity provided by the generator before it is passed on to delicate computer hardware. The power delivered by a generator is not as clean as that normally delivered by the power company.

Uninterruptible Power Supplies

An Uninterruptible Power Supply provides several essential services and is best used in conjunction with surge protection and line conditioning equipment. A UPS can help to smooth out noisy power sources and provide continuous power during electrical sags. Its primary benefit is to provide temporary electrical power during a blackout. Depending on the model, it may also provide some measure of line conditioning protection.

Uninterruptible Power Supplies come in three basic types, based on their features.

1. The basic UPS is a “standby” UPS. A standby UPS provides battery backup against power outages (blackouts and brownouts) and a modest amount of battery-powered voltage correction.

2. The “line interactive” UPS is a step above the basic unit. It provides voltage regulation as well as battery backup by switching to battery power when line voltages move beyond preset limits. This type of UPS converts a small trickle of electricity to charge its batteries at all times. When power fails, the line interactive UPS detects the power loss and switches itself on. A line interactive UPS has a subsecond switching time from line power to battery power.

3. The third type of UPS is an “online” UPS that sits directly between line power and your equipment. The online UPS is always providing power to your electric circuits and has a zero transfer time between the loss of line power and the start of battery power.

UPS BATTERIES UPS systems provide power during a blackout by drawing on their battery electrical supply system. Most of these batteries are sealed lead acid batteries. Unlike the batteries found in many notebook PCs, these batteries do not have a “memory” and should be completely drained as few times as possible. Depending on how often your UPS draws on its batteries, they should last up to 5 years. Remember that brownouts and short-duration blackouts all wear on the batteries, so if your local power fluctuates very much, the life of your batteries will be reduced. The speed at which your UPS batteries age is also determined by their environment. Extreme heat or cold are not good for your batteries. Refer to your manufacturer’s guide for the recommended operating temperature range.

As batteries age, their power-generating capability will decrease. Therefore, regular preventative maintenance is important. Preventative maintenance should include changing the air filters to help keep the UPS unit cool. At that time, all the batteries should be checked for damage, leaks, or weak cells. You should also consider a service agreement that includes the replacement of damaged batteries.

UPS systems use “inverters” to convert the DC battery power to AC power. An inverter is electrical circuitry to change the direct current to alternating current. High-quality UPS systems use a dual inverter system for smoother power conversion.

UPS “SIZE” The first question people ask about UPS units is, “How big does it need to be?” This all depends on several things:

What must be supported? This translates directly into how much electricity must be supplied at a given point in time.

How many minutes must the battery pack provide this level of support?

Is your area prone to power problems?

Will this UPS be managed remotely through manufacturer-provided software?

UPS units are rated according to the number of volt-amps they can deliver. Volt-amps are different from watts and you cannot equate the volt-amps provided by a UPS with the watts used by an electronic device. Typical power factors (which is watts per volt-amp) for a workstation is 0.6 or 0.7. So if your PC records a drain of 250 watts, you need a UPS with a 417 volt-amp rating (for a 0.6 power factor). Always be careful to never overload a UPS beyond its rated capacity. Doing so will severely damage it.

Most UPS manufacturers have a software tool for estimating UPS sizes. Where possible, use their programs to size your UPS. In the absence of that tool:

1. Begin with a list of all equipment for which you will need to provide electricity. This may include personal computers, monitors, servers, critical printers, network hubs, and telecommunications equipment—whatever will be supported by the UPS.

2. Determine the wattage ratings on all these devices by checking their nameplates. The numbers may be expressed as watts. We need the numbers in volt-amps (VA) since that is a more accurate number for UPS sizing. Multiply the watts by 1.4 to get the volt-amps load.

3. If the power usage is provided in amps, then multiply that number by the line voltage (120 volts in North America and 230 volts for Europe) to get a volt-amp rating.

4. Total the volt-amp requirements for all the supported equipment.

This is the amount of load you need to support. From here you check with the manufacturer for the size of unit to support this load for the amount of time you select.

SWITCHING TO BATTERIES A UPS uses power line filters to address minor power disturbances, but its main weapon against a power loss or severe brownout is a near-instantaneous switch to battery power. This is good for keeping your systems alive but hard on the batteries. If your UPS must often switch to batteries because of poor power regulation in your area, then your battery life will suffer significantly. As the batteries rapidly age in this environment, they would not provide protection for the length of time you may be counting on from your UPS.

Recharging the batteries is another issue. Some UPS systems allow you to choose between a fast recharge or a slow recharge. The frequency and duration of outages in your area should determine if you must recharge your batteries as fast as possible or use a more gentle slow recharge process. Fast recharging puts a large drain on your restored power supply.

If you switch to generator power, you do not want the batteries to recharge from the generator as it might take away too much of the power needed elsewhere. If you plan to recharge the batteries from the generator, be sure that is included in the power load plan when sizing the generator, and that the batteries are on a slow recharge cycle.

UPS LOCATIONS If you have concentrated your data processing main computers and servers into one room, then selecting a location for your UPS will be easy. Electricians will run a separate electrical circuit from the UPS to the equipment to be protected. Electrical codes require these outlets to be a different color so you will know which circuit you are plugging into.

Some critical machinery and computers will be located away from the central computer room. For these devices, consider smaller UPS units located adjacent to the equipment. These units will not have a long battery life and will be used to keep the machine operational long enough to shut it down gracefully. Be sure not to lose sight of these satellite units as they will need to be tested and their batteries maintained over time. Remote monitoring software is ideal for this situation.

ADVANCED UPS FEATURES Modern UPS units offer much more than battery backup. They possess microprocessor logic to support a wide range of services. They can provide alarms of error conditions both on the unit and through your data network. This is a very useful feature since they are often stuck in some dark back room where an audible alarm only serves to annoy the mice.

The network signaling of power conditions is a very useful feature. Depending on the capabilities of your UPS and data systems, a UPS can start the orderly shutdown of equipment to protect it before the UPS batteries are exhausted. This feature is very useful over weekends and holidays when no one is around. In some cases, it can order a restart when power is restored. A more sophisticated UPS system stores a log of the power supply status for later analysis. Do you know how noisy your power lines are? Do you know the frequency and magnitude of sags and spikes that occur on your electrical power lines?

A UPS is a critical component of a data network. Remote monitoring software allows a network control analyst to monitor the status of each remote UPS and display the current line voltage and the voltage/current draw on the equipment. This helps to track which lines seem to have the most variation and potentially drive it back to a root cause in your facility. If some electrically driven machine in your facility is causing problems in your internal power grid, it needs to be identified and provided with better electrical isolation.

UPS TESTING It is great to have a UPS system set up and running, but it needs to be tested if there is to be a credible plan. So on a weekend in your slow time of the business cycle, you should plan for a UPS load test. This will demonstrate your power support system capabilities before a blackout strikes.

To set up the test, shut down the programs on all your computers but leave the computer running. The idea is to not lose any data but to still pull each system’s normal electrical load. Bring in your UPS service technician to address issues during and after the test. Warn management you are going to do this. When all is in place, have an electrician cut the power to the UPS in that part of your facility and see what happens.

This test has several goals:

1. The first is to see what is not on the UPS that should be. Once the power is cut and the batteries are humming, you will see which server, computer, network device, etc., has been overlooked. Now look to see which low-value items are connected and wasting valuable emergency power.

2. Second, you need to know how well your UPS will support the load you have attached to it. If it is overloaded, you must plug some equipment into other power sources or get a bigger UPS unit.

3. When you shut down the servers’ operating system, bring along a stopwatch and write down how long it takes. This will tell you the minimum amount of time the UPS must hold for you to perform orderly system shutdowns. If your servers are far apart in different rooms and the same person is expected to shut them all down, that may add travel time to the time you must allow on the UPS. Plan your time for the worst case.

4. Observe exactly what information the UPS displays about the remaining minutes of power given the current consumption rate. Compare this to the operator instructions you have provided to the after-hours support team. Be sure to also train the facility electricians on how to read the UPS display panel.

5. Exercise your power shedding plan while someone observes the impact on the UPS. How much additional time do you get for each level shut off?

POWER GENERATORS

If your facility absolutely must maintain its power supply in the face of any sort of electrical problem, then you will need your own electrical generation system. This is a large leap in complexity above UPS systems and takes extensive planning. There are some industries that quickly come to mind as requiring this level of support. Hospitals need it to support electronic medical equipment, food storage sites need it to prevent spoilage, and even Internet hosting providers need it to ensure maximum application availability to their customers.

On the other hand, it is kind of nice to switch from having a problem to being in control of it. A properly sized and installed electrical generation system can return some benefits in the form of keeping your company running while other companies cope with a rolling blackout, by the potential of selling electrical power back to the utility company, and by running your generator during peak electrical usage times thereby avoiding the highest cost electrical power.

Sizing Your Generator

Once you decide the need for maximum power availability, you begin with determining what it is you need to support. If it is everything within a building or an isolated part of a building, you could contract for an electrician to monitor the amount of electricity used in that building or part of the building and use that as a starting point for sizing your equipment. If the generator is only supporting one portion of the facility, you must have a way to isolate it from the rest of the structure.

Next, you need to know how long your generators must provide electricity. If you live in an area that experiences widespread natural disasters, such as floods, hurricanes, earthquakes, or blizzards, then you might want to allow for running this system for several days at a time. A good place to start here is to use your personal experience for the frequency and length of outages in your area to plan on system size. This will help to determine the size of your fuel storage system for running the generator.

Switching Time

Some industries, like hospitals, have a standard amount of time they can be without electrical service. Their generator must switch on automatically. However, mechanical engines take some time to start and run up to operating speed (ever start your car on a cold day?). Some of the fastest generators can automatically sense the loss of electrical power and start providing stand-by power in less than 10 seconds.

The question here is how long of a gap your company can tolerate. During this brief outage, UPS systems can maintain power to critical devices. Some equipment, like refrigerators, can tolerate a brief gap since they are already chilled down. Some equipment, like lights, can ever so briefly be out if supported by a backup emergency lighting system. So when deciding how long your company can function without electrical power, be very specific of what is needed and why.

The alternative is to always run generators together with pulling power from the power grid. As you can quickly discern, this is yet another step in complexity that distracts you from your core business. Rather than take this step, most companies settle for quick-switching generators supplemented with UPS support at critical points.

Generator Testing

More than any other power support system, the engines on your generators will take regular care. Begin by running them monthly to ensure they function on demand. Next, they need to be tested under load. This can be arranged for a weekend where everything they are to support is turned on and the electricity disconnected for a few hours from the power grid. Periodic testing under load is a critical component of your power backup system credibility.

During your testing, monitor the actual fuel consumption to generate a given unit of power. Fuel consumption is also a matter of air temperature (height of summer or the depths of winter). Aside from the manufacturer’s claim, use this to determine how long your onsite fuel supply will last for delivering electricity.

Testing also exercises the people supporting your generators. By drilling them on their duties, they will be able to respond more quickly in a crisis. Be sure to rotate personnel to provide sufficient trained backup staff.

Working with Your Public Utility

Unlike a UPS or line conditioner, a generator has the potential to help pay for itself. During peak electrical usage periods, such as the depths of winter or the oppressive heat of summer, running your generators will reduce your draw on the community’s power grid. Some utilities base their year-long electrical rates on the peak usage at any point over the year. By using your generators on these days, you contribute to the overall containment of electrical rates. And even then the units don’t need to run all day, just during the peak usage hours of the day. If your generation capability is sufficient to run your entire company site, then the utility may call you and ask that you run your generators at those times to reduce peak usage.

Another aspect of running your own generators is the selling of power back to the power utility. This must be investigated with your local board of public utilities as to how much you would be paid and what conditions must be met. But if you are in an area of unreliable power, you might be able to address your own problems and cover some of your costs at the same time.

Environmental and Regulatory Issues

Like all good things, there are some downsides. Running an internal combustion engine to make electricity puts pollution into the air. Some jurisdictions limit the number of hours per month that a generator can be run (except in a crisis). Before purchasing your generator, check for any requisite permits for such things as fuel storage, air pollution, taxes, etc.

ACTION STEPS FOR YOUR PLAN

If you have a UPS, be sure that it is properly maintained. Most UPSs require regular preventative maintenance, such as changing the air filters and checking the condition of the batteries. If you skip this step, then you are destined to discover how important it is the next time your UPS is needed.

Most large UPS units come with a small display panel that indicates the condition of the UPS’s ability to supply power. Master this panel and all controls before an emergency arises. Never open the front of the UPS as the unit is an electrical shock hazard. The unit should only be opened by trained electricians.

In the event of a power outage, the front panel display can tell you how long the UPS batteries will be able to supply power to all the devices attached. This is a very important piece of information. Most computer servers take a long time to recover if they suddenly lose power. They require time to shut down “gracefully.” You need to know the typical amount of time required to shut down each critical server.

Most UPS units include an audible alarm for when they are on battery power. It is important to know what these alarms are and what to do when you hear them. If the UPS units are in a place where a security guard can hear them after hours, be sure the guard knows what to do.

EMERGENCY LIGHTING

In a large building, it can get very dark very quickly in a blackout. Even if flashlights are readily available, you need to be able to find them. Also, a sudden blackout can be very disorienting to some people. This only adds an element of panic to the moment. To address this, most legal jurisdictions require the installation of emergency lights that come on whenever power to the building is lost. This provides some light for the safe evacuation of offices and workplaces.

These lights depend on a battery to power the lights in a blackout. To be sure that the lights and the battery are ready when they are needed, they must be checked monthly according to the manufacturer’s testing steps.

SOMETHING FOR YOUR SUPPORT PLAN

Following are three notices for you to consider as additions to your power support plan.

The first is an insert for your immediate steps contingency plan to be kept at the help desk and posted on the computer room walls. When power drops, employees should execute the steps on this notice to contain the problem while the technical staff is called in.

The second is a wall notice on priorities—which equipment to turn off in what order so that your UPS and generator system can be freed to support the most critical systems.

The third is a set of instructions for making up the power shedding tags described on the Power Shedding Priorities page.

Power Outage Action Plan

1. Immediate Action.

a. Notify your facility’s Maintenance supervisor immediately.

b. Notify your Supervisor.

Primary:   (name and number here)

Alternate: (name and number here)

c. Determine the scope of the problem.

Look outside the office. Is there power everywhere else in the building?

Send someone outside to see if the electricity is on outside of the building. (Do not go yourself. You must sit by the phone to coordinate action until your supervisor arrives.) Are there lights on in any other buildings? Are traffic signals working? Are street lights on?

d. Notify the facility’s Disaster Recovery Manager.

e. Begin a log sheet of all events to include when the lights went out, who was notified and when, any communications with the power company, etc.

2. Physical Layout.

a. UPS Room

Send someone to look at the UPS. Note how long the display indicates the batteries are projected to last.

Execute the power shedding plan.

Keep monitoring the UPS and continue shedding power using devices. When the UPS time falls below 20 minutes, begin shutting down all the servers.

Call all system administrators and the network manager.

Power Shedding Priorities

When electrical power fails or when the power company notifies you that a failure is imminent, the drain on the UPS batteries is minimized by turning off equipment according to its power shedding priority. After reliable power is restored, turn equipment back on according to its priority. Start the most critical systems first.

When a power outage occurs or is anticipated, notify the Help Desk, Facility Security, your supervisor, and the Data Processing Manager. Monitor the UPS systems to see how much time is remaining on the batteries (instructions are posted on the UPS devices).

This approach uses Power Shedding Priorities A through D, with A being least critical and D being most critical equipment to keep running. Priority is set according to:

Which systems directly support facility production.

Which systems will cause widespread problems if they stop working.

Which systems are difficult to restart if they stop suddenly.

1. As soon as you lose electrical power, shut off nonessential systems and equipment identified with a green “A” power shedding label, such as CRTs, terminals, printers, card processing equipment.

Notify other company sites on your network.

Update the Help Desk, Security, and the Data Processing Manager.

2. When the UPS units show 15 minutes of power remaining, shut off low-priority CPUs and devices identified with a Yellow “B” power shedding label. When you progress to this step, notify:

The Help Desk and the Data Processing Manager.

3. When the UPS units show 5 minutes of power remaining, shut off all remaining equipment and servers identified with a red “C” power shedding label. When you progress to this step, notify:

The Help Desk and the Data Processing Manager.

4. Let equipment identified with a tan “D” power shedding label “die” on their own as power drops off. This is equipment that can tolerate a sudden power drop.

Communication is important!

Ensure that management and the appropriate support people know when you start the next step of shutting down or restarting systems. Data Processing Management will call in the required system support people for a proper restart.

Power Shedding Tag Instructions

Labeling your equipment: Make up labels on colored paper and then laminate them.

A = Green     B = Yellow     C = Red     D = Tan

Resources:
Liebert	www.liebert.com
American Power Conversion	www.apc.com
Tripplite	www.tripplite.com
Underwriters Laboratories	www.ul.com

CONCLUSION

Electricity is a powerful resource necessary to operate the modern business. As with any resource, you need to be familiar with its role in your operation and how its absence will affect your company. In the absence of clean power from the electric utility, the main sources of electrical power are a battery-operated UPS and a generator. A thorough understanding of the electrical requirements of your organization will help you to design the most cost-effective plan to protect against its absence.

CHAPTER 15

TELECOMMUNICATIONS

AND NETWORKING

Your Connection to the World

Everything is connected … no one thing can change by itself.

—Paul Hawken

INTRODUCTION

All of us are becoming more interconnected, as people and companies make increasing use of telecommunications networks to connect to suppliers, customers, business associates, and friends. Telecommunications has enabled our businesses and personal lives to be “just-in-time” no matter where we might be in the world. We use telecommunications networks to talk, share data, run applications, download files, organize schedules, exchange e-mail, and access the Internet. We use these networks to connect us with our coworkers and to collaborate with customers in doing the business of the firm. This chapter reviews the dangers that threaten our telecommunications networks and the attributes unique to these networks that require special processes to restore them after a disaster.

Most organizations have two distinct telecommunications networks: one for voice communication that uses the standard telephone system (also known as the public switched telephone network, or PSTN) and another for all electronic communication that uses one or more computer networks that are typically connected to the Internet. While the need for two distinct networks is slowly changing with the increasing use of VoIP (Voice-over-Internet Protocol) for sending voice communications over the Internet, many organizations will have to maintain both for the foreseeable future. Fortunately for our planning purposes, both types of networks have very similar issues that affect disaster recovery.

PUBLIC SWITCHED TELEPHONE NETWORK

How important can a telephone line be? What’s the cost of a telephone call? What is the value of a missed call? How much would you pay for 100 percent telecommunications reliability? Is such a thing even possible?

Consider this scenario. You are the Plant Materials Manager sitting at home in your living room. As you watch the evening news, you see a video of flames rising out of the roof of a key supplier’s main factory. You don’t need trouble like this. What should you do? You try calling your salesperson and the company offices but no one answers. First thing in the morning and all the next morning you try calling them. The company is more than an hour drive away, you can’t get away from the office for that long, and your calls still go unanswered.

Looking through your contact list, you select another supplier and place an order for a 2-month supply of goods to replace the supplier that had the fire. The set-up costs are a killer but it is worth the money if you can keep your factory running. At least this problem is contained. Trouble is, the fire you saw on TV was isolated to the front offices and the factory is fine. The warehouse is bulging. At a time like this, the supplier needs the cash, but employees can’t call out until the telephone switching room is replaced! The company took care of the fire but forgot all about its customers.

All companies strike a fine balance between the cost of reliable telephone service and the cost of downtime. In a time of tight budgets (which is always), you balance the cost of premium services against the potential loss of telephone service. The more reliable that you want your telephone network to be, the more you must spend. Telephone service is central to the conduct of business in most companies. A failure isolates customers and suppliers from your company—and that can quickly become a very lonely feeling.

Some companies are even more dependent on telephone service than others are. If you work at a factory, you use the telephone to conduct business, but your main concern is moving products down the line. The loss of telephone service does not slow down the assembly line one bit. However, if your facility is a call center for sales or service, then 100 percent reliable telephone service is crucial to your ongoing business. In the factory example, if the conveyor on the main assembly line breaks, the workers have nothing to do. In the call center, if the telephone service is interrupted, then they are likewise idled. How your facility utilizes its telephone service has a great deal to do with how crucial it is for your operations.

The North American telephone network is designed to carry traffic from about 10 percent of all telephones in a given area. This approximates its load at its busiest times. In a wide-area disaster, this capacity is quickly swamped, as people everywhere call to check on their loved ones. The cellular network doesn’t fare any better because portions of it also use landlines and it has its own capacity limits. In today’s fast-paced business environment, how long can your business afford to be without telephone service?

The disasters that can befall your delicate communication lines are legion. Anywhere along its path the wire can be broken, switching equipment can lose power, or problems can even occur within your building. The high level of telephone communications reliability of our modern telecommunications networks is the envy of the world. You must ensure that your company has reviewed the risks to its telecommunications pipeline and has taken steps to reduce the likelihood of failure. Our job here is to identify those risks and build a wish list of mitigation actions to make your company’s telephone communications even more rock solid than ever before.

The next few sections of this chapter review the basics of how your telephone system works. If you are already a telephone system expert, you can skip these sections. If you are charged with writing a plan in an area in which you are not a full-fledged expert, then you should read on. A basic understanding of the main components will make it easier for you to develop an appropriate plan.

PSTN Basics

The world of telephonic communications all begins with the telephone instrument on your desk. This modern marvel connects you to the world at large. The telephone is connected by wire to the wall jack, which in turn is connected to a wiring closet. A wiring closet may be one of those locked doors on your office building floor that you never get to look into. The same function can be served by running wires to a specific place on the wall of a factory (which is generally a wide-open space). In either place, you would see brightly colored wires running in spaghetti-like fashion to a telephone wire “punch-down block” or punch block.

If you look at a punch block, it seems to be a “rat’s nest” of colored wire routed in an orderly fashion, yet going in all directions. Each office telephone has at least one pair of wires running from its wall jack all the way back to the wiring punch block. The punch block also has wires running to the telephone switch for your building. This is where the two are connected to each other. If an office no longer needs a telephone line, the punch block is the place where it can be disconnected.

Private Branch Exchange

From the wiring closet, large bundles of wires run to the company’s telephone switching equipment. In a larger company this would be a Private Branch Exchange (PBX). A PBX replaces the long-gone company telephone operator who would connect internal calls with a plug patch panel. The patch panel physically connected the wires from one telephone instrument into the wires of another. This is really a basic operation, but until electronics matured, it was the only way to do it. This is now all done electronically. In today’s offices, a PBX takes these incoming wires and, using the signal on them, provides electronic switching of calls within the building. A simple way to think about a PBX is to consider it a special-purpose computer with all the support needs of a minicomputer.

The PBX determines which calls are intended for external telephone numbers and connects to the local telephone company central office using “trunk” lines. Trunk lines are used for inbound or outbound calls. The number of trunk lines the PBX has connected to the central office determines the maximum number of external (inbound or outbound) calls you are capable of supporting at a given moment. Modern PBX systems offer a wide range of additional services such as:

Voice mail.

Telephone conferencing.

Call transferring.

Music on hold.

Based on this, from a business continuity perspective, you have a large single point of failure device. You must plan to recover from a catastrophic failure of the PBX (e.g., burned to a cinder in a fire). A close examination of the PBX room and its ancillary equipment will show that it is essentially a computer room, requiring the same electrical and climate stabilization actions as any computer room. Backup copies of the configuration data must be made for each device and handled with the same care as the backup data from your computer system. These data should be securely stored off-site and available when needed. Up-to-date backed-up data are the key to a prompt recovery.

Internal to a company, the telephone signals can be analog or digital. Digital PBX systems provide a wide range of services beyond simply routing calls. Most PBX systems use digital signals to communicate with the telephones. This allows for additional services, such as one-touch dialing, preprogrammed telephone numbers, voice mail, etc. This is significant in the case where you want to use a modem to dial out of the office. A fax machine or a direct modem connection requires an analog line. If you have a digital PBX, then you will need separate analog lines.

Rather than run a multitude of analog telephone lines, most companies access the Internet via their external data network. This works fine for most office dwellers. However, some devices still need these analog lines and you should keep track of where they are. Examples of analog dial-out lines might be for an alarm service, which dials out to notify the repair service of an out-of-tolerance condition, or for validating a credit card in stores at the point of purchase.

OTHER VITAL EQUIPMENT LOCATED IN THE PBX ROOM Other important equipment is typically located in the same room as your PBX. After the critical devices are identified, make sure they are protected and draft a plan to fully recover them. These devices may include:

Interactive Voice Response (IVR). Gives callers information based on what the caller enters using the telephone keypad. You have heard the messages—please select 1 to talk to sales, select 2 to talk to … These audio tracks should be backed up and the queuing logic documented.

Automatic Number Identification. This is the business version of what is sold to consumers as “caller ID.”

Intelligent Port Selector. Connects the incoming call to the first available line. This is used when you have multiple people answering inbound calls, as in a hotel chain’s reservation center.

Call Management System. Used to monitor the volume of telephone calls during peak periods, to identify the number of telephone operators needed, and to track operator efficiency.

Call Accounting. Tracks calls made and assigns them to a billing account. This is also known as a “Station Message Detail Recording” (SMDR). This is used in various ways. Lawyers might use it to bill their time to a specific client. Some companies use this to track employees’ long distance usage, etc.

Call Monitoring. Tracks the level of call activity by showing the status of the trunk lines, the number of calls in progress, the number of calls waiting in queues, the wait time, the number of abandoned calls, and the status of the operators.

The Telephone Company’s Central Office

Soon after telephone service was first created, it became obvious that every person could not be wired to every other person they might possibly want to call. This would result in an impossible maze of wires. To call someone new, you would first need to run a wire from your telephone to their telephone. Imagine the problems in asking a girl for her telephone number in those days!

To simplify this problem, all the telephone lines were run into a centralized building and then the switchboard operators would physically patch your telephone line into the switchboard, making the connection. These buildings would be located at various places around the city and were the central place where connections were made. Eventually, automated switching made it easier, but buildings were still needed to switch the calls. In more recent years with the advent of solid-state circuits, the floor space required for these buildings has shrunk dramatically. You still see them around: small buildings without any windows, usually neatly trimmed grass, and a small telephone company sign by the front door.

The central office provides a service similar to your PBX by switching your call to another local telephone, to a different central office far away, or to a long-distance carrier’s point of presence (POP). The long-distance company then routes the call through its switching center and back to a distant central office and down to the far-away telephone.

Interexchange Carrier Point of Presence

With the breakup of the AT&T long-distance monopoly in 1982 came the creation of independent long-distance telecommunications providers. These non-AT&T long-distance providers are known as Interexchange Carriers (IXCs). Along with this choice came the opportunity to split your long-distance service across different carriers in hopes that all of them would not be knocked out in a disaster and therefore your communications traffic could flow out on the alternative pipeline. For that to be true, a lot of careful planning is necessary.

First, you must ensure cable separation so that you have a different wire path from your facility to the IXC’s point of presence. Most companies are a mix of owning the network in high-traffic areas and renting from another carrier in a low-traffic area. This means your traffic separation may only be on paper. If you are in a high-traffic area, the carrier may run a separate cable to your facility. If you are not, then you will probably connect to the IXC in the nearest central office. Do not take for granted that because you use two companies you are on two different wires. Ask them if they share lines and ask to see the route of the cable.

Some telecommunications experts believe that route separation is much more important than using multiple telecommunications companies. They feel it is easier to manage one supplier so long as the cable issue is addressed.

When evaluating networks, the first thing to consider is whether they rent their network or own it. Most are a mixture. The more of the network equipment that they own, the more control they can exercise over it. When choosing an IXC, things to consider include:

What is their system availability time, and what they will guarantee?

What are the consequences to the IXC of downtime? You can ill afford it. A few extra free minutes of service every month is poor recompense for missed customer calls.

What is the restoration priority for the sections of the network that you will be using?

What are their alternate routes for the places you normally communicate with? Don’t automatically assume a new carrier will be a better alternative to your existing carrier. Is your service route a spur (single-threaded) service? Is it a ring architecture, which at least gives you two paths in case one has a problem?

How often does the carrier practice its disaster recovery procedures?

How easily can the IXC shift your inbound calls to another site?

OK, well, this all sounds pretty straightforward. After all, telephone service has been around for well over 100 years and its technology is pretty well known. What could possibly go wrong?

COMPUTER NETWORKS

All but the smallest firms today have one or more networks within their organization. A network consists of one or more servers that are connected to one or more workstations and allows users to share information and resources. This connection can be by copper wire, by fiber-optic cable, or even by radio frequency (RF, or wireless). Rarely are workstations connected directly into a server. Typically they plug into the wall jack, which is connected to a wiring closet. In the closet will be one or more “hubs” that will connect the local devices to a hub in the computer room. This room is often the same closet as contains the telephone wiring.

In the computer room (either a central location or possibly sprinkled about the facility) sit the servers. A server is a computer that runs software to provide access to resources attached to the network, such as printers, disk storage, and network applications. A server can be any type of computer that supports the sharing of resources: it may be a standard desktop PC, or a dedicated device containing large amounts of memory and multiple storage devices that can support hundreds of PCs at the same time.

The next section reviews the basics of how your computer network works. If you are already a network expert, you can skip this section. If you are charged with writing a plan in an area in which you are not a full-fledged expert, then you should read on. A basic understanding of the main components will make it easier for you to develop an appropriate plan.

Computer Network Basics

The term local area network (LAN) is typically used to describe workstations and servers that are all physically in the same location. A LAN can be implemented using two main architectures:

Peer-to-Peer. Each computer on the network communicates directly with every other computer. Each computer is treated equally on the network. Communication between computers is coordinated using a network hub.

Client/Server. Each end-user PC is connected to a central server, which coordinates communication and supplies resources to the end users. Typically the server contains file storage, printer, backup, and other resources that are shared by the end-user client computers.

Figure 15-1 shows a typical client/server LAN, with five PCs connected to a server. The server has attached a network printer and a backup tape drive that can be accessed from any end-user client computer on the network.

A wide-area network (WAN) is a data communications network that consists of two or more local area networks connected to allow communication between geographically dispersed locations. Most companies have some sort of WAN connection, either connecting to other locations within the organization or to the public Internet. There are several different types of WAN communication links available. The most common ones are:

Dialup. This is the most basic type of connection, using a modem and a PSTN line. This is also the slowest and most unreliable type of connection available, supporting up to 56 kilobits per second (kbps).

FIGURE 15-1: Typical LAN configuration.

ISDN. An ISDN (integrated services digital network) connection uses a PSTN line using an enhanced communication standard to speed up the connection to 64 or 128 kbps.

T-1. A dedicated connection that supports data rates up to 1.5 megabits per second (mbps). A T-1 line consists of 24 individual channels, each supporting 64 kbps of data.

T-3. A dedicated connection that supports data rates up to 43 mbps. A T-3 line consists of 672 individual channels, each supporting 64 kbps of data.

Frame Relay. This is a packet switched protocol for connecting devices on the WAN. It supports data transfer rates up to T-3 speed.

Cable Modem. Using the same digital or fiber-optic network as cable TV, cable modem Internet is capable of speeds up to 60 mbps.

ATM. ATM (Asynchronous Transfer Mode) is a means of digital communications that is capable of very high speeds; suitable for transmission of images or voice or video as well as data.

Wireless. A wireless LAN bridge can connect multiple LANs for distances up to 30 miles with a direct line of sight.

VPN. A VPN (Virtual Private Network) uses encryption in the lower protocol layers to provide a secure connection through the public Internet.

Figure 15-2 shows a corporate wide area network, with two remote locations linked to the headquarters, which is in turn connected to the public Internet. Just as your phone system has a room for patching users into the telephone network, your computer network also has a central location where individual pieces of equipment are connected into one or more servers. This might be the same room as your telephone patch panel, or it could be part of a separate dedicated computer room.

It starts with each piece of equipment on the network being connected by wire to the wall jack, which is in turn connected to a wiring closet. A wiring closet for your computer network is typically located in or next to the computer room where your servers are located. The wiring closet consists of many different cables running in spaghetti-like fashion to a network cable “punch-down block” or punch block.

The network cable punch block is similar to the telephone wiring punch block discussed earlier. It has network cables routed in an orderly fashion, yet going in all directions. Each piece of equipment on the network has at least one cable running from its wall jack all the way back to the cable punch block. The punch block also has wires running to the computer servers. This is where the two are connected to each other. If a piece of equipment needs to be removed from the network, the punch block is the place where it can be disconnected.

FIGURE 15-2: Typical WAN configuration.

Internet Service Providers

Internet Service Providers (ISPs) connect your organization to the Internet. You are connected to your ISP via one of the network connection options listed in the previous section. Most organizations will have only one physical connection between its network and the ISP for Internet access. If Internet access is critical to the operation of your business, you should consider a connection to a second ISP to give you a second pathway to the Internet. There are, however, several things to evaluate and plan for when considering a redundant ISP. (This is similar to considering multiple telecommunications companies.)

First, you must ensure cable separation so that you have a different wire path from your facility to the ISP’s point of presence. Most ISPs are a mix of owning the network in high-traffic areas and renting from another ISP in a low-traffic area. This means your traffic separation may only be on paper. If you are in a high-traffic area, the ISP may run a separate cable to your facility. If you are not, then you will probably connect to the ISP’s nearest point of presence. Do not take for granted that because you use two ISPs you are on two different wires. Ask them if they share lines and ask to see the route of the cable.

Some telecommunications experts believe that route separation is much more important than using multiple ISPs. They feel it is easier to manage one ISP so long as the cable issue is addressed. When choosing an ISP, things to consider include:

What is their system availability time, and what they will guarantee?

What are the consequences to the ISP of downtime? You can ill afford it. A few extra free minutes of service every month is poor recompense for missed sales due to the Internet being unavailable.

What is the restoration priority for the sections of the network that you will be using?

What are ISP’s alternate routes for the places you normally communicate with?

How often does the ISP practice its disaster recovery procedures?

RISK ASSESSMENT

When developing your disaster recovery plan for your telecommunication services, look for single points of failure that will adversely impact your critical business processes. These critical processes should have been identified in the Business Impact Analysis. External threats to telecommunications include cables being cut, interference from electromagnetic sources, attacks by hackers and other intruders, and damage from natural hazards, such as fire or water. Internal threats include many of these same dangers; cables can be cut by remodelers and leaky water pipes can damage cables and equipment. In an ideal world, you would have duplicate service running to each telephone or desktop. Externally, an issue with multiple Internet connections is that there are now two points of potential entry for hackers; be sure to work closely with your security team to adequately protect your WAN. Internally, it is usually not cost-effective to run duplicate cables; an alternative is to install an extra jack every few cable drops. If a problem occurs in an end-user’s cable, the extra jack would be available as a backup until the problem cable is repaired.

It is also important to review your vulnerability to problems with the devices that connect your telecommunications network together, such as routers, switches, and hubs. Your Business Impact Analysis should determine the level of threat that these devices pose if they fail. Look at having redundant devices installed that can take over if the primary device fails.

Wireless telecommunication connections can provide an effective backup strategy if the wired network fails. Wireless networks have no wires to fail and can be quickly installed if needed after a failure in the wired system. The major drawback to using a wireless network is that the transmissions can be intercepted; make sure encryption and other security measures are implemented to avoid others intercepting your network traffic.

You should also consider using network monitoring software that can detect problems on the network. Network monitoring software can alert you immediately if a node on the network is having problems or has failed, which will allow you to restore service more quickly and help prevent problems from cascading. Most network monitoring software can be configured to look for system parameters that fall out of the desired range and then generate a message to a support person to take action.

Natural Hazards

Now that you know the basics of how your telecommunications systems work, what are the risks to these systems? Chapter 3 discussed natural hazards in detail, so this section only addresses those natural hazards that have a major impact on your telecommunications systems.

Ice Storms and Blizzards. Ice can coat cables strung from telephone poles and, if the weight is great enough, potentially bring them down.

Thunderstorms and Lightning. Severe rain can weaken the ground around a pole and cause it to sag when there are high winds at the same time pushing against it. Lightning can strike telephone poles and send a major charge flying down the line, burning up wire and equipment along the way.

Tornadoes. A powerful destructive force that can snap lines and rip up telephone poles. High-risk areas are prime candidates for buried lines.

Hurricanes and Floods. Can cover a wide swath of land and not only bring down a telecommunications network but also prevent crews from promptly addressing the problems.

Human-Created Hazards

People are just as big a danger to your telecommunications systems as nature can be. Human-created dangers include:

Breaks in Buried Lines. Sometimes emergency excavation is necessary (such as repairing a broken gas main). Sometimes the local government is cleaning trash out of the ditches alongside the road. Sometimes a well-meaning person just digs without asking (including on your own property). In any of these cases, there is the chance that your tiny little cable will be dug up and severed.

Accidents. Sometimes people miss the tight turn and break off a telephone pole. If this is your only line to the central office or your ISP, your service is gone until the pole is replaced.

Central Office or ISP Failure. A problem in a central office or with your ISP can quickly shut down your telecommunications unless you are wired to a second provider.

Denial of Service Attacks. Hackers can use various techniques to render your Internet service unusable. A Distributed Denial of Service (DDoS) attack uses the computing power of thousands of vulnerable, infected machines to overwhelm a target or a victim.

Telecommunications Equipment Room

The room that houses your telecommunications equipment may not be as safe as you think it is. Dangers to your telecommunications room include:

Temperatures that are too hot, too cold and that swing widely are all hard on your telephone switching equipment. Extreme temperatures stress the printed circuits. Large swings in temperature (hot to cold) cause expansion and shrinkage of circuit cards and again can weaken components over time.

Humidity has an effect on temperature and the growth of mold on your equipment.

Loss of electrical power will definitely stop a PBX, routers, switches, and other telecommunications equipment. Pay phones and direct lines out should still be operational since the telephone circuits supply their own power.

Water pipes overhead in your telecommunications room could release water onto your equipment. The same goes for your cable panels. Overhead pipes along external walls could potentially freeze and leak when thawing. Leaks from rooftop air conditioning compressors are also a problem.

Security may be a factor. This room is not set up to accommodate tourists and they should not be allowed in. Keep the door locked at all times.

Fire is a possible danger. This equipment generates heat and is in danger from fire or the sprinklers used to extinguish a fire. Typically this room is unattended and a slow-starting fire may go undetected. Gas fire suppression is expensive but may save your equipment.

Cabling

The discussions earlier about wiring closets had a purpose. Imagine the mess if an isolated fire in the wiring closet melted all these wires. In terms of structural damage, you got off pretty easy. But in terms of damage to the telecommunications system, that entire area of the facility will be without service for some time. New cable would have to be run from the PBX or Internet server to the closet and from the wall to the closet. The alternative is to splice an extension onto each cable and run it into a punch block. Either way is time consuming and expensive. So you can see how the wiring closet can be a single point of failure for your telecommunications systems. This makes it a good idea to keep everyone out of it and to not store anything in the closet that might cause a problem.

INTERNAL CABLING For disaster avoidance, concerns begin with the wiring closet and patch panels. Take a walk with your telecommunications specialist and identify the location of all telecommunication patch panels in your facility. Things you are looking for:

1. In a closet

The door and any windows are kept locked.

There is fire suppression equipment (usually fused link sprinklers).

There is nothing else in the closet except telecommunication and/or data communications equipment. Combustible materials stored in the closet threaten your equipment. This is not the place to store holiday decorations, old files, office supplies, janitor supplies, etc.

Very few people have keys—telecommunication support staff, security, and no one else.

There should be sufficient light to work in the closet.

There will often be data network wiring and telephone switching equipment together in these closets. They are a natural fit here. The issue is that where the patch panel is simply a physical connection for wires, the network equipment is energized electronics, which will generate heat and introduce a potential fire source.

2. Outside a closet

Be sure it is covered and the cover is locked. Same rules for the keys as for the wiring closets. Enough keys for the right people and none for anyone else.

All cables and wires leading into the external panel are encased in heavy conduit to inhibit tampering.

If this is located in a warehouse or factory, be sure it is strongly protected from environmental influences (leaky ceilings, etc.) and from being crushed by a forklift or toppling stacked material.

EXTERNAL CABLING The first rule of outside cabling is “cabling and backhoes don’t mix!” A chain is only as strong as its weakest link. Your servers and PBX are snugly locked up in their room, and the ISP’s office and the telephone company’s central office are also secure. But the wire in between is exposed to the ravages of weather, people, and machines.

Experienced telecommunication professionals have their own name for backhoes—cable locators!

External cabling, which is the cable that runs from your wiring closet to the telecommunication provider’s office, is your biggest concern. The wire runs from your building to an access point along the road, usually on a pole. Then the wire runs through the countryside (usually along a road or railroad) to a point of presence or central office. In the city, it might run through underground pipes to the point of presence or central office. You have no control over where the wire is run and no capability to protect it! In some areas, you will even have separate cable runs to be concerned about—one for the local telephone company, one for the Interexchange Carrier (long-distance service), and yet another for your Internet service.

A common term used when describing part of a telecommunications network is “the last mile.” The last mile describes the wire from the telephone company’s central office or the ISP’s point of presence to your structure (and is usually more than a mile). This is also known as the “local access” or the “local loop.” This part of the network is the most vulnerable. It is often carried on telephone poles (susceptible to ice storms and errant vehicles) or underground (susceptible to damage from digging).

ROUTE SEPARATION The best path to telecommunications reliability is redundancy. This can include redundant equipment, redundant technicians, and redundant cabling. The more alternate paths that a signal can be routed, the more likely it is to get through. The principle of cable route separation should be an integral part of your telecommunications network design. Essentially this means that you have more than one cable between your building and your telecommunications providers. This prevents a total communications outage from a single cable cut.

Few companies can afford to take this to the extreme, but you can look into some steps. Begin by asking the telecommunications provider to show you the route that your cable takes from the wall of your building to their office. This will also show you how exposed the cable is to auto accidents (are the poles close to the road), to backhoes (is the cable buried along the road in some places), or any other number of threats.

With this experience in mind, negotiate with the telecommunications provider a fresh cable run from your building (exit from the opposite end from the other cable) to a different central office or point of presence. This will keep you operational in case your usual point of presence is damaged or experiences an equipment failure, or if the cable to the point of presence was broken. How likely is that to happen?

In May 1988, there was a fire in the Illinois Bell central office located in Hinsdale, IL. The two-story building was completely gutted. This building was an important hub for Illinois Bell, as well as for major long-distance carriers, such as MCI and Sprint. In addition to cellular service and data networks, approximately 40,000 subscriber lines and 6 fiber-optic lines lost service.

To reduce the chance of a total system failure similar to the Hinsdale disaster, the telephone companies have gradually migrated their central office structure from a spoke-and-hub approach (with its obvious single point of failure) to a ring or mesh approach, in which multiple central offices are connected to each other. In this scenario, calls are routed around the damaged central office in a manner that is transparent to the caller. This is an ongoing process and may not yet have been completed in rural areas or small towns. If possible, you want routing separation with the wire running from your facility running to two different central offices on separate routes.

Route separation is more important than having multiple vendors. Most cable routes follow railroad rights of way and the major carriers’ lines commonly converge at bridges. Imagine the number of places there are to cross a major river, highway, etc. There aren’t many to choose from. The lines from various companies often come together here and cross under the same portion of the bridge. What could go wrong? Vehicles crossing bridges might catch on fire, river barges can break free and strike bridge pilings, and major bridges are tempting terrorist targets.

Many telecommunication providers define cable separation as a distance of 25 feet or more between cables. Others ensure cables are at least 100 feet apart and have at least a 2-foot separation at cable crossovers. Ask your Internet, local, and long-distance telephone carriers how they define cable route separation and how faithful they are to that standard. When using multiple vendors, even if their cables are separated, they share a common weakness if they join at the same point of presence or cross a river under the same bridge. Upon close investigation, you may even see that where you are using two companies, one is leasing part of the same wire from your other provider!

What to do? Find out how many access points there are for your telecommunication service. Does it enter the building at more than one place (again, trying to separate the cables)? Ask the telecommunication provider to identify the route that the cable takes from their office to your facility. Drive along this route and look for potential problems.

In terms of your long-distance provider, where is their point of presence located? Where is the point of presence of their competitors? You can consider running a separate connection to a second point of presence to provide for multiple circuits. Depending on the distance, this separate line may be quite expensive to run and maintain.

MAP IT OUT Now that you know the cable route to the telecommunication provider, map it out—from where it enters your wall to the provider’s point of presence. You might think, “Isn’t that the telecommunication provider’s responsibility?” The answers is yes, but type up that excuse and paste it to your wall in case the line is cut. Try it on your boss! If you drive to work using the same route as your cable, you might see construction crews digging near where the cable is to repair a water main, snapped poles, or sagging wire due to accidents or severe weather—any number of threats too close to the wire for comfort.

So draw out a map of where your wire runs from the telephone pole outside to the telecommunication provider’s office. Then make a detailed map of the run from the pole to the wall and on to the wiring closet room. Indicate which lines terminate in equipment provided by the telecommunication provider and which lines provide essential services.

DEVELOPING A PLAN

This sections describe issues to consider as your develop a disaster recovery plan for your telecommunication assets.

What Are We Protecting?

As you know, the first step in building a plan is to make an inventory of your telecommunications assets. You will assemble at least three lists. The first lists every major item and who to call if it breaks. The second shows where the main cable runs are located in the building. The third includes all the telephone numbers used by the facility. In a crisis, you may need to reroute some of these to another location.

1. Begin with a list of all major devices in your telephone switching room and network room, such as the PBX, IVR, Internet server, etc. Include on this list:

A description of each device.

The serial numbers of the main equipment and any major components.

The name, 24-hour telephone number, contract number, and contract restrictions for whomever you have arranged to service that item.

The location of every item, including a simple floor map of all telecommunication rooms.

Be sure to back up your entire configuration data, either on magnetic media or, if the file is small, print it off and store it safely away.

2. Now make a wiring inventory of all the cable runs within your facility. You do not need to show individual runs to the offices. In a crisis, you can always shift someone to another office. This is best accomplished with computer-aided drafting software on a digitized version of the floor plan.

Indicate the runs on maps of each building or floor of a multistory building. Knowing the location of these cable runs is important to quickly assess damage.

Indicate where the telecommunication service enters the building and its route to the telephone exchange and network rooms. Note any hazards along the cable path.

If you have pay phones, indicate them on the map.

If you have any independent direct lines that bypass your telephone switch, mark them on the map.

3. Make up a telephone number inventory with all the telephone numbers assigned to your building/facility.

DID (Direct Inward Dialing) lines.

Dedicated telephone lines that bypass your PBX, such as fax machines.

Pay telephones.

Foreign exchange lines.

Identify Critical Telephone Circuits

Telephones are used to communicate, and every part of the company uses them differently. In an emergency, salespeople will need to contact customers, the warehouse will need to call suppliers, and the people at the disaster site will want to call home. In a disaster, you won’t be able to please everyone right away, so you need a restoration priority guideline to know which circuits to recover (or protect) and in what order.

Start this analysis by reviewing the critical functions identified by your executives at the beginning of the business continuity planning project. Next, meet with representatives from each department and ask them to identify their critical telecommunications needs and at what point in the disaster containment/recovery they would be needed. For example, early in the emergency, the Human Resources team might need to notify employees to not report to work or to come in at certain times, etc. Each department must prioritize its communications needs.

With your critical communications functions identified, you can determine which circuits support these key functions. If you trace out the top three or four circuits, you may see some of the same hardware, some of the same cables, and circuit paths common to them all. Working down your list, you can see which of your hardware devices (or cable bundles) has the greatest benefit to being restored first.

You must also include in your plan how you will relocate operations to another site. This is a twofold issue. In the first case, if a department must be moved off-site due to damage to its offices, you need to be able to shift its inbound calls to the new location quickly. In the second case, if your telephone equipment room is destroyed, you must quickly restore a minimal level of service. Some people might plan to build a new telephone room onsite. Others will contract with a service company that will bring a telephone equipment room to your site already set up in a large trailer. The company needs to plug into your wiring (no trivial task). This could buy you time until your equipment room is rebuilt. If you elect to bring in the configured trailer, this agreement must be made in advance, and be aware that these might not be available in a wide-area disaster.

Review Network Security

Your network has some unique security issues to consider. The first is physical security. No one but the network support staff should be allowed in the network closets. The network administrator must work with the facility’s security manager to ensure that these doors are always locked. Depending on the airflow in the closet, some vents may be added to the doors. These rooms are not for storing holiday decorations, old documents, unwanted furniture, etc.

Sometimes network equipment is in freestanding cabinets instead of closets. These cabinets must be locked just like the closets and the keys held by only a few people. In a large facility, this can add up to a lot of keys. Your goal is to minimize the number of keys by using a submaster key for all closets and, if possible, a submaster key to all cabinets. These keys must be tracked and, if lost, a determination made whether to rekey the locks.

The next area to consider is logical security. The network software on the servers and in the network devices will be, in some cases, password protected. These passwords should be protected like any other and known only to the key network support staff. However, they should also be written down and locked in the data processing manager’s office in case the network staff is unavailable. (In a wide-area emergency, the network manager or key staff members may not be available or able to come in.)

Another issue to consider is having a policy that no one is permitted to plug anything into the network. That is the exclusive job of the network support team. This is to stop people from plugging in their home notebook PCs and bringing down your network. In addition, contract employees should never be permitted to connect into your network. The same policy should cover your wireless nodes.

Because servers can support a large number of users and are used to host critical applications, loss of a server can have a severe impact on the business. Some processes to ease the restoration of servers include:

Store Backup Tapes and Software Off-site. Follow the procedures discussed in Chapter 17 for handling and storing backups. Backups of data and application installation media should be stored off-site in a secure, environmentally controlled facility.

Standardize Hardware, Software, and Peripherals. Standard configurations of these items will make restoration much easier. The standard configurations for hardware, software, and peripherals should be documented in your plan.

Document the Network. The physical and logical network diagram must be kept up to date. The physical documentation should include a diagram of the physical facility, where the primary cables are routed. The logical diagram shows the network nodes and how they interconnect. Both diagrams are critical in restoring the network after a disaster.

Document Vendor Information. Maintaining up-to-date information on the vendors you use for hardware, software, and peripherals will make it easier to restore your operations.

Work with Your Security Team. The more secure your systems are, the less likely you are to have a data loss due to a security breach. You’ll also want to be sure you can restore the latest level of security if you must configure new servers after an emergency.

Telecommunications Mitigation Plan

With these risks in mind, along with due consideration of the identified priority circuits and equipment, you can assemble a mitigation plan to reduce the likelihood of a threat or its impact if it occurs. The key to telecommunications mitigation is redundancy. Redundancy in equipment in case one machine must be repaired. Redundancy in communications routes in case a cable is severed or interrupted. Redundancy in communication methods, such as radios, cellular telephones, or satellite communications, will provide at least basic communications support.

CABLE MITIGATION PLAN Your cable mitigation plan should consider the following.

Multiple paths for the “last mile.” Investigate the path from your telecommunications equipment room all the way to the central office or point of presence. Ask the telecommunications provider to make another connection from your PBX or Internet server out through the wall of your building at a point distant from the other exit point and on a different route to a different point of presence. Be sure that your in-house wiring staff understands what you are asking for before they start.

Multiple paths for Interexchange Carriers. Investigate the path from your telephone equipment room to your service provider’s point of presence. Avoid the same route as used for your local telephone service.

From the pole to your wall, if the cable is underground, ensure it is clearly marked with “do not dig” indicators or other obstacles to keep digging equipment away (or at least delay them until the cable can be marked).

TELECOMMUNICATIONS EQUIPMENT ROOM MITIGATION PLAN Think of your telecommunications room like a computer room. Their environmental and security needs are almost identical.

Uninterruptible Power Supply (UPS). When you lose electrical power, this is what will keep your PBX or Internet server active until external power is restored or until your facility’s electrical generators kick in. Conduct a power loss test to see which equipment is not connected to the UPS. If the device isn’t essential or time consuming to restart, take it off the UPS. The fewer machines on the UPS, the better. When conducting the power test, see how long the batteries can support the load. Be sure the UPS is properly maintained by your service company.

When Hurricane Hugo struck the southeast United States in 1989, it surged hundreds of miles inland with huge amounts of rain and wind. The storm downed so many trees that it was days before electrical power was restored to some areas. Even the emergency batteries at the central offices were eventually drained. So, do not depend solely on your UPS for emergency power. Consider other power sources as well.

Fire. This room requires the same fire protection as a computer room. An early-warning fire alarm system and gas fire suppression system are highly recommended.

Security Access. This room is normally unattended. It should be locked as no one has any business strolling around in this room. The telecommunications system administrator can normally perform his or her administration duties via terminal over the network.

Structural Investigation. Although you can rarely select the room, a close inspection can identify problems. Water pipes running along the walls or along the ceiling are a potential source of problems. They may freeze or leak. They should be watched carefully. Consider installing a plastic shield attached to a drain placed under them to catch condensation or leaks. A roof-mounted air conditioner may cause a roof leak. External walls may stress the air-handling equipment as outside temperatures heat or cool the walls.

Temperature Variability. Your equipment must be in area that maintains a specific operating temperature and humidity range for maximum life. This is also essential to maintaining your service agreement coverage. Proper air conditioning, heating, and humidity control equipment must be in service and well cared for.

Alarms Such as Humidity, Temperature, Fire, and Electric. A bank of alarms will help you monitor the condition of the room. These alarms must sound within the room, as well as at the security guard station as the room is normally unattended. Early detection will reduce the likelihood of significant damage. Include automatic paging equipment for notifying the after-hours support team of problems. Consider installing automatic shutdown software for your equipment. This signals your hardware to shut itself down gracefully in the event of a problem.

Data Backups. Like a computer, your switch and configurable devices need to back up their configuration data whenever these change. Store copies of these files in a secure off-site location.

Housekeeping. Keeping the door locked will help to prevent others from using the switch room as a storage facility. Nothing should be stored in this room that does not pertain to the equipment. This will reduce the amount of combustibles available to a fire. Do not let this room become another storage or janitor’s closet.

ALTERNATE COMMUNICATIONS METHODS You must have written procedures for quickly routing Interexchange Carrier traffic to the local telephone service in the event that your IXCs have major problems. How hard is this to do, and then to switch it back again? Is it easier (or cheaper) to do this by splitting the load across two carriers and then shifting the entire load to the functional carrier? Other items to consider include:

Do you have company-owned cell phones and cellular modems for communicating when the telephone system is inoperable?

Develop written procedures for how to work with the telephone company to shift specific telephone numbers or all incoming calls to a different company site.

Do you have any employees who are ham radio operators? It is nice to have alternative channels when the primary ones are not available. In a wide-area emergency, radio communications will also get congested. Radio conversations are not secure and anyone with the proper equipment can listen to them. Radio communication is also slow and not suitable for large data volumes.

Other communications alternatives include satellite and microwave, both susceptible to problems if the antenna has been shifted by a storm or earthquake.

ACTION STEPS FOR YOUR PLAN

Most companies contract emergency recovery of their telecommunications equipment through the same company that services their equipment. A problem arises if there is more than one service company involved. An alternative is to arrange with a company to come onsite if your telecommunications equipment room is destroyed and to set up a trailer adjacent to your building with a ready-to-go telephone switch and Internet connection. To support this, companies often run cables to the front of the building for a quick connection.

Another consideration is that pay phones do not route through your PBX. They are direct lines to the outside service. So, if your PBX is inoperative, use pay phones. Keep a list of where they are and their telephone numbers.

Cell phones are also common. Cellular traffic can be somewhat limited in a wide-area emergency as the local cell towers become saturated with calls. Use this as an alternative communications channel, not as the primary backup.

A branch office is an ideal place to shift operations to until the disaster-struck site is recovered. The key is how far away the backup facility is located from the affected site. It should be far enough away to be unaffected by the same disaster. There is no set mileage distance but it should at least be on a different power grid and telephone central office. If it is too far, then you must also provide lodging for the relocated people.

One problem is how to relocate to these sites in a crisis. If this is during an area-wide disaster, mass transit such as air travel may be disrupted and driving there yourself may be difficult. One side aspect of the attack on the World Trade Center in September 2001 was that it shut down all air travel. Companies scrambling to activate their out-of-state recovery sites had a difficult time shifting key personnel and material to the site. No one had planned for a complete shutdown of the air transportation system.

Things to consider after a disaster:

Do not make any unnecessary calls—only for emergencies.

When calling, you may need to wait several minutes to get a dial tone. Do not hit the switch hook, because every time you do, you are placed at the end of the line for the next available dial tone.

When you receive a dial tone, quickly dial your number. In a time of low telephone service availability, the dial tone is offered for a much shorter time.

TESTING

Plans are written by people with the best of intentions but, unless they are tested, you will never know what you don’t know. Testing a plan exposes gaps and omissions in the process steps. It points out incorrect emergency telephone numbers and emergency equipment that no longer exists.

Test exercises are a great way to train the people on what to do. The more that they practice their emergency steps, the faster they will be able to perform them as they become familiar actions instead of something new. Where possible, include your emergency service providers. They will be a great source of recovery information.

When testing your plan, include some of your contract service providers. They may be able to point out some gaps in your planning or things that can be done in advance to make their assistance to you flow much easier.

Also, over time, your communications flows will shift. As this occurs, update your testing accordingly.

CONCLUSION

The modern business relies on telecommunications to perform its role in the marketplace. Like any resource, we need to be familiar with its role within our operation and how its absence will affect us. Redundancy is your best defense against a disaster removing your ability to communicate with customers and suppliers. A thorough understanding of the telecommunication requirements of your organization will help you to design the most cost-effective plan to protect against its absence.