IT Strategic Plan Part 2 Continuation

Chapter Twenty-Seven: Business Continuity Planning

Dave McCandless

OVERVIEW

Sage wisdom tells us that "Life is what happens when you're busy making plans." A more recent version of this same sentiment can be politely phrased as "Stuff happens." No matter how it is said, the message is clear: As we go on about our daily business, both personal and professional, disruptive events will take place that have the potential to substantially change what we had planned for our future.

For business, undefined change is very likely a bad thing. Business plans define key objectives that must be met for the business to survive. Deviation from these plans will likely result in higher costs and lower revenues, sometimes to the point where the business must pause or even cease to exist. Stable businesses exist because their leaders are able to understand the nuances of whatever industry they are in—they find the path and navigate the obstacles to success. The good leaders show us everyday how they can manage what they can control.

But what happens when that control is threatened? How do these leaders react to sustain a business when faced with disruptive events? How do they ensure continuity of the business plan to be able to achieve their key objectives? Leaders make this happen by creating a company culture that methodically plans and executes to sustain business operations when faced with disruptive challenges.

Business continuity planning (BCP) is the self-imposed discipline that defines how well a business can sustain operations when faced with disruptive events. As a CIO you will be part of this process, since in the information age a substantial number, if not all, of a company's business processes will rely on IT systems. This chapter will give you insight into how to navigate the journey that is the discipline of BCP.

DEFINING THE NEED FOR BCP

While leaders universally agree that continuity planning is a vital component of strategic business process delivery, they have widely varied opinions on how to define and justify the actual function that is BCP. Successfully defining the problem means creating a crisp yet persuasive value proposition that business leadership will endorse and champion.

Why We Plan

If all went according to plan, we would not need alternatives. No spare batteries for the flashlight. No spare flashlight. No candles for when you cannot find the flashlight. And no matches for the candles, since we no longer use candles. But things do not go according to plan—which means we are stuck with figuring out how to plan for events that happen that we cannot predict.

This gap creates the challenge to create a collection of manageable business processes that can lead a company through the resolution of disruptive events. There is a clear choice of words here: disruptive meaning we will know what to do when disruptions happen, but we likely will not know when the disruptions will occur. Words like unforeseen and unexpected could both be interpreted as events for which the company was not prepared. Disruptive means events that change the course of the action, but if planning is complete, the corrective action is defined and methodically initiated. Planning for the continuity of operations becomes the exercise and includes all operational areas of the organization. Any foreseeable disruptions need to be defined and then analyzed for their impact and eventual remediation. When completed, operational procedures are updated to include these new changes. To ensure correctness, recurring testing must take place. As an ongoing process, these changes likely represent a tremendous cost to a business—for many foreseen disruptive events that may never happen.

Which leads to the question: At what cost? How much investment must be funneled into this process to ensure continuity? Answering this fundamental question is the responsibility of senior management, since they ultimately are responsible for the ongoing operation of the business. Management has the final say in what processes are critical to ongoing operations. Through this leadership process, management communicates to the organization what is important and sets the expectation for all business and operations leaders to find sustaining solutions. Without agreement on what is important at the time of the disruption, chaos is the likely outcome.

Taking an example from history, one of our great planned achievements has been to send men to the moon and return them safely to Earth. These two simple objectives drove an unprecedented continuity planning process that sought to uncover every foreseeable disruptive event in NASA's Gemini and Apollo programs. This led to an unfathomable price tag. What percentage of this cost is attributable to contingency planning? Can we make the argument that any of the planning or the subsequent redundant systems deployment was not required to meet the objective? Certainly few of these contingency systems were ever used, but would we ever have sent men into space without them?

Executive strategy requires that a choice be made of what kind of BCP an organization will embrace. It must start at the top.

Expecting the Unexpected

Earthquakes. Hurricanes. Global economic collapse. Loss of network connectivity at a critical call center. These and thousands of other events like them jeopardize delivery of a company's global business every day, all day. With rigorous planning followed by methodical execution, these events will not impact the business—in fact, they may add value. A Thursday afternoon blizzard will not be nearly as well received in Chicago as is will be in Aspen.

The Web has shaped all of us to expect always available, fully functional services. Online shopping available 24/7. Customer support available anytime via a text message, mouse click, or iPhone. Instantaneous e-mail and chat responses. To achieve these service levels, companies are increasingly less tolerant of outages and have expectations that outage durations will be minimal.

Dwight Eisenhower, a master planner, once said: "Plans are nothing; planning is everything." This concept is the basis for BCP: An organization that has taken the time and committed the resources to plan for the unexpected will have a greater chance of surviving than one that has not. An untested strongbox full of backup tapes, user manuals, and contact lists could very well be useless the minute it is opened. Yet in many companies this is exactly the expectation—that some magic "survival package" will be ready when needed at a time of crisis.

Internal expectations for high availability are not unique; requirements are also on the rise for companies to be able to demonstrate externally the capability of resiliency in the face of impact events. Each year, more pressure is created from business partners and regulatory agencies to show documented proof of a continuity plan and its subsequent successful test execution. The regulated permission to stay in business may be dependent on a thorough and successful BCP process.

Adding Business Value

Is it reasonable to make a case that a strong BCP process can add business value? Is avoiding unavailability of the business a strong enough argument to gather support for BCP? Or do we also need to make the case for tangible business value?

Risk professionals can demonstrate that, yes, there are sufficient examples of businesses that are severely impacted by business interruption. But for business leaders who require more convincing, there are ways to make a case for how BCP adds tangible business value. The details are not found in widgets sold or expenses reduced; they are accumulated over time via reduced incidents of outages and consistent costs of operation.

PROCESS OF CONTINUITY PLANNING

Many methodologies exist that explain the BCP process. This section covers the basic concepts common to each. Any organization pursuing BCP needs to craft a process that matches its specific business needs.

Commitment to the Process

The first step in BCP is critical—recognition by leadership that the organization needs a BCP strategy and commitment of resources to support it. Executive sponsorship not only ensures the commitment but demonstrates to all employees the importance of the BCP process. Sponsorship means delegating the BCP responsibility to an internal team of professionals that will deliver on the objectives.

How should the BCP team be created? To add the most value, the team should include a mix of journeyman members of both business and services parts of the business. Members should also have previous exposure to a BCP implementation or be provided with formal training. Finally, a team leader with solid project delivery experience is needed, preferably an internal resource with sufficient authority and finesse to make change happen.

Assigning responsibility without proper authority will destine the process to languish with no chance of success. Also assigning "any available resource" demonstrates a lack of commitment to complete the task right the first time through. External experts can assist to make the process happen, but the right internal staffing is crucial to success.

Defining the Objectives

Once BCP oversight is created, the next step engages the BCP leadership team with the company executives to define what are the strategies and priorities for the business. Some examples: Online companies need their commerce web sites available 24/7; grocery stores that survive by just-in-time inventory need the doors open and the supply trucks rolling in; airlines need not just planes and crews readily available but decent weather in which to fly them. Each of these companies has the same goal—to keep the goods and services moving that bring in the revenue. Yet each has vastly different requirements as to what are the priority activities of the company.

The responsibility belongs to the senior management team to define the operations that are important. These are not low-level decisions about e-mail and enterprise resource planning; these are high-level, what-is-important-to-the-business decisions. Once these critical business operations are identified and their priorities of importance are established, the process cascades down the management chain to subsequent levels as details are added. Each business team participates as needed to add depth to the requirements. When finished, the company has a clearly defined business process classification—a target list of business priorities.

Of course, none of this planning can be done in a vacuum—BCP needs to coexist with other risk mitigation processes. Viable organizations will likely have risk management policies in place, so BCP objectives and risk mitigation objectives need to be in alignment. Often businesses already will have recognized this need and will have created a framework to bring similar processes under one objectives-defining umbrella.

Determining Impact

The next step involves the tasks to identify and qualify the impact on the business processes posed by the disruptive events. To begin, the analysis team must determine the various event categories to consider and, for each type of event category, the specific events themselves. For example, in the case of natural disasters, the specific events could be floods and tornados. In the case of malicious attacks to computing systems, the specific events could be denial of service attacks, intrusion attacks, and spam e-mail injection overload attacks. Businesses must assess their own unique requirements to define events and categories.

Once all the event details are fully defined, each event is rated—both for the magnitude of the potential disruption as well as the likelihood of the event actually happening. The smaller the disruptive impact, or the smaller the likelihood of occurrence, or the smaller the importance of the impacted capability means the overall business impact is low.

The common tool used in BCP to facilitate this process is the business impact analysis (BIA). This inspection of each critical business process will identify scenarios where interruptions or failures may happen, the likelihood of them happening, and the damage they will likely do to business operations when they happen. The BIA will also establish how quickly a business process can return to operation (recovery time objective), and what levels of functionality and data availability must exist within the business process (recovery point objective). Once completed, the BIA provides a complete specification of requirements on how to enhance the business processes that will result in the most effective mitigation of impact of disruptive events.

Knowing how to measure impact becomes an interesting exercise. For example, in the area of IT, an interesting paradox has emerged: Just how important is e-mail? Seldom does the process of sending and receiving e-mail account for any substantial amount of revenue, yet it is hard to imagine a company's staff completing useful tasks if e-mail is not available. But now consider business-to-business or business-to-consumer messaging: Any company that relies on its internal e-mail infrastructure for significant communication with customers and partners substantially raises the impact potential of loss of any part of this infrastructure and operation. So now how important is e-mail? This is a good example of the kinds of questions the BIA works to answer.

Creating the BC Strategy

Armed with the management priorities and the business impact findings, the next process is to create a strategy for business continuity that ensures ongoing business operations. Two remediation types are possible: corrective remediation that minimizes impact and restores operations, and preventive remediation that expands operational capabilities to withstand outages. Additionally, multiple strategies can be created that will provide resumption of some aspects of business operations. Equally possible is that the same impact can be resolved by differing recovery solutions. Total cost of the various possibilities will play a significant part in defining the strategy.

A quick analysis of alternate data center strategies is a good example of how to apply these principles. Companies with a single data center that rely on an outsourced provider for an alternate data center have chosen to correct the disruption of the loss of their only site by moving to the alternate site. Companies with multiple data centers with integration of systems and operations across sites have chosen to avoid or prevent disruptions by managing transaction activities through distribution. Which remediation type to choose is driven by a compromise between business needs and available funding.

Creating the BCP strategy also means engaging the various business units to elaborate on their technology dependencies. Many business units may expect IT to own this investigation on their behalf. However, this expectation is flawed, given that IT only provides the platform and is in no position to judge the critical importance of the various capabilities of business systems.

Once the importance of all critical systems and their outage impact are established, the final steps in creation of the strategy are to define which systems are selected for operational and technology changes. While simple in concept, this process defines where the investments in the business should be made to fulfill the requirements of the strategy.

Implementing the BC Strategy

The implementation process is the execution of the details described in the strategy. Simply stated, this is the process that implements value changes to existing operations and infrastructure that will result in higher resiliency to impact events. Up until this stage, all previous BCP activity is focused on analysis. BCP implementation forces a change process and creates the potential for disruption to these critical business systems due to the change itself.

Implementation also includes other areas critical to the business: updated emergency response procedures, the creation or upgrade of an emergency operations center capability, updated crisis management capabilities, and updated BCP training and education procedures. A fully-defined BCP testing process must be deployed and validated to ensure the changes are implemented and will deliver the expected results.

Ongoing Plan Maintenance

The final step in the BCP process is to ensure that this strategic planning exercise is not a one-time event but continues on through regular and predictable execution. This ongoing process will provide additional business value if continuous improvement techniques are performed as part of year-to-year program management.

It is very likely that new requirements will emerge as the BCP process is in execution. Like any other large project, scope creep can be assured. Testing with partners and outside agencies? Review of test plans to meet BCP objectives? Review of management objectives for comparison with BCP objectives to ensure they are still in alignment? Adding new lines of business or retiring old ones? Adjusting to substantial changes to the market or customers? Many questions like these will emerge during the execution of the analysis and deployment processes.

Business processes will undergo constant change. New types of disruptive events will emerge to threaten the business. Executive objectives and priorities will change. The BCP process must continue to adapt to these changes to avoid the impacts of disruptive events.

STRATEGIC VALUE OF BCP

Measuring the business value of BCP is tricky. In the classic sense, the question has been What is peace of mind worth? But as the process of BCP becomes better understood, organizations discover how the process creates value to distance them from the competition.

Value before a Disruptive Event

In a world where every interconnected organization deploys some amount of BCP, regulatory and certification agencies will continue to require more resiliency capability before granting permission to participate. For companies with strong BCP, this process will prove to be a filter to slow weaker industry competition, creating a competitive advantage.

Having the appropriate BCP processes in place at the right time enables a company to focus on the core business. Having insufficient BCP in place could mean diverting key resources to the BCP process to bring it up to a minimal industry standard. Perhaps the business intends this catch-up as part of the strategic plan—a reasonable business strategy. But it raises an interesting question: Without sufficient investment in the BCP process, how would a company know what is a minimal acceptable level of BCP, and how would it know when it has caught up?

For companies that struggle with tracking the maturity of their various business units and their interactions, a strong BCP process can add management value. Since the basis of BCP is to ensure continuity across the entire business, BCP participation will identify business capabilities in need of improvement. This leads to resiliency across the entire organization to withstand impact events and create a healthier business process environment.

Value after a Disruptive Event

As soon as a substantially disruptive event occurs, the first reaction from all involved will be "Good thing we made the investment in BCP." Of course, if there was minimal or no investment in BCP, the reaction will be much different.

A well-crafted BCP process will go into execution automatically—procedures will be followed and operations teams will execute according to plan. Companies that have planned well will be seen as strategic thinkers and will likely see their brand value and possibly their financial position improve due to their ability to sustain operations. Their immediate and well-honed messages to customer, partners, and the general public will instill a sense of respect and feeling that "I want to be in business with this company." Not only will the company sustain its current business, but in all likelihood it will generate new business.

But the added value does not stop there. Disasters as impact events may put businesses in various market segments or geographic regions temporarily or permanently out of action. Companies designed to adapt quickly can aggressively take on a greater share of an existing market or get a foothold in a new one. BCP in this case becomes an enabler of business opportunity—what would have been lost opportunity is now possible because of a previous commitment to BCP.

WHAT BCP IS NOT

Now that we have talked a bit about what BCP is, it is worthwhile to spend some time on what it is not. There is often confusion in the business world on the value of BCP. Those companies that do not recognize the value will not understand the discipline and will perpetuate the misconceptions.

BCP is Not Disaster Recovery

One aspect of BCP is the ability to recover from disasters, or disaster recovery (DR). While the exact numbers will vary, the industry consensus is that fewer than one in five business-disruptive events are caused by disasters, such as hurricanes, floods, and earthquakes. With 80 percent of the impact posed by nondisasters, wouldn't a company be better off spending time on nondisaster recovery (NDR)?

Ironically, no one ever asks about an NDR plan, nor do any CIO peer-level discussions ever address the value of an NDR plan. Even a recent Google search found zero results for "nondisaster recovery plan" but over 400,000 results for "disaster recovery plan."

This one area creates so much confusion in the industry that it is worth addressing in detail: Disaster recovery is not business continuity. Years ago in a less mature IT world, the methodology was called DR. DR may be an integral part of BCP, but only a small part, and depending on the nature of the business, it may be a minor part of the BCP process. A future CIO's best approach is to learn to use the terms BCP and DR correctly. And when faced with a DR-centric organization, it will make for quite an education challenge.

BCP is Not Insurance

As a business discipline, insurance is the end result of defining a monetary value to an asset such that in the loss of that asset, your insurance partner will pay you the agreed value of that asset. The process of obtaining the insurance includes the process for assigning value to the assets—which in turn implies some discovery process for identifying of all the company's assets, followed by the determination on which assets are worthy of a declared a value.

While insurance may be part of contingency planning, the BCP process is too important to the survival of the business to relegate to another annual finish-and-forget-it process. Insurance is normally a process the finance team uses to mitigate the cost to replace key assets without incurring the costs of the replacement.

True business leaders understand that BCP is not a loss mitigation computation but a strategic business-sustaining process that needs ongoing attention. They recognize that business operations will have continuity procedures interwoven into daily activities. They are willing to accept the cost of BCP beyond that of simple asset replacement.

BCP is Not a Part-Time IT Job

Companies that expect to stay in business when faced with disruptive events understand the need for BCP. Those that do not understand the value will often do just enough to get by—and that means doing what is minimally required to pass the many audits imposed across the company's administrative functions.

Take an example of a company with a web site as its primary platform for business transactions that is suddenly faced with a disruptive event. Most likely the IT team has created a solution with the appropriate alternate site capabilities and 24/7 on-call procedures for systems administration. But is the same true for the customer service teams that support the customers: Do they have alternate plans for e-mail, phone, chat, and other tools—even a place to go to work—to execute their jobs? Are the communications and marketing teams coordinated in advance to be able to quickly and effectively get the word out about the disruptive event? Does senior management have confidence that the entire process will be put into motion with the same effectiveness as normal daily operations?

Most small and many medium-size businesses choose not to allocate sufficient funding to sustain an enterprise-wide BCP process. Instead these companies may choose to have a minimal few members of the IT team keep a current DR plan. This of course creates BCP only if all your problems are planned disasters and if the only impact to the business will be to IT.

EMERGING TECHNOLOGIES AND BCP

Each tick of the technology evolution clock brings IT planners a variety of solutions available to them with the potential to change the nature of systems recovery. BCP experts who for decades practiced the concept of backup tape creation, off-site media management, and emergency data center failure now have a range of modern alternatives that will rewrite industry best practices. Years from now even these cutting-edge capabilities will seem archaic, but today they represent potential for substantial value.

Virtualization

Enterprises large and small are reaping the benefits of using tens or hundreds of standard computers to run thousands of server images. This concept—creating fully encapsulated application environments that are transient and adaptive across hardware infrastructures—makes the process ideal to support rapid recovery from the disruptive nature of impactful events.

Which is a better way to re-create a server: Spend hours installing operating systems, then patches, the applications, then more patches, then adjust configuration, or in minutes instantiate another instance of the same server? And do so anywhere in the global infrastructure? IT plans that incorporate virtualization into daily operations will find that minor changes to the process will simplify how the process expands to allow fail-over and fail-back recovery operation activities between multiple hot sites.

Systems management tools already exist that simplify the rapid migration of workloads across data centers. Need to move processing out of a data center threatened by a hurricane? Very easy to accomplish if the enterprise core infrastructure is already enabled for this capability.

As virtualization technologies mature, the product extensions seen as unique to BCP will likely become transparent features within product offerings. Planners will be able to use configurations in the standard solutions to enable preventive and corrective actions that kick in once disruptive events are discovered—or, even better, anticipated.

Supernetworks

Gone are the days of high-cost, low-bandwidth networks. Innovative communications technologies are driving new methods for moving large quantities of data to multiple global locations. Increasingly more intelligent data routing and caching technologies allow for critical data to be replicated as part of common business process to facilitate rapid remote site recovery. As the concepts of "local" and "remote" blend, planners will change the decision processes that have traditionally defined global operations.

Cloud Computing

Cloud computing by its nature has the potential to change the face of BCP for IT. The ability for business to quickly move its computing resources across infrastructure capabilities the way business travelers switch hotels creates massive flexibility unheard of in the current world of event recovery.

Consider the current model for popular recovery services used by the large providers. BC planners contract for emergency services that match existing corporate infrastructures, with the expectation that when the event happens, the corrective recovery process goes into action. The actions taken are those defined by the contract— an agreement that is infrequently tested and comes with a specification that is very likely a distant reflection of the company's current and ever-changing production infrastructure.

The infrastructure-as-a-service and platform-as-a-service models of cloud computing provide organizations new opportunities for innovative partnerships that can be kept current with the rate of change of the business needs. BC planners will see an end to the countless hours focused on network and server provisioning and will instead focus on how to build application management and recovery solutions that span a multitude of diverse service offerings.

As is the case with all disruptive technologies, early adopters will be the guinea pigs that will flush out the problems for everyone else. Cloud is no different—and in the world of BCP where resiliency is everything, the evolution will take many years.

De-Duplication

The massive glut of information—what we have and what we have yet to create—has the potential to overwhelm organizations that need this information to feed systems critical to always-on business operations. Increased demand to push data efficiently to distributed global locations has forced companies to rethink how to manage the movement and quantities of this data.

Fortunately, the data storage industry has within the past few years created substantial innovation in data de-duplication, or the process of removing duplicate bytes of data of images for storage and transfer. These new duplicate-free image files become a more efficient way to manage data copies—fewer bytes to save to disk, copy to tape, or move across networks.

The value of de-duplication solutions will be recognized in the efficiency of backup operations: Full backups that now take days to complete are reduced to hours, and incremental backups that take hours to complete are reduced to minutes. This creates recovery scenarios such that site-to-site image copies will complete in fractions of historical times, enabling remote-site activities to start with reduced delay time. Coupling this capability with other technologies like virtualization and cloud computing, de-duplication provides BC planners the chance to apply new thinking to process of solving recovery challenges.

SUMMARY

BCP has become synonymous with emergency response—the ability of an organization to withstand the unexpected. BCP creates a process that documents two critical capabilities of a company:

1 The essential business processes that sustain ongoing operations

2 The automatic response activities that will sustain these business processes when the unexpected happens

BCP becomes the backbone process that breaks this knowledge down until preparedness becomes just another brick in the foundation for everyday operations.

Because companies will continue to build business solutions based on IT technologies, CIOs will continue to be called on to impart greater influence in their companies' BCP strategy.