TOP 3 REASONS YOUR DISASTER RECOVERY PLAN WILL FAIL

Monthly, I will choose a specific area of information technology or digital business to present three topics of discussion to invoke thought and hopefully, educate. For my first Top Three Blog, I decided to write about the “business continuity and disaster recovery planning process” prompted by the realization that most business owners, or those charged with assuring the availability of their organization, had a misplaced level of comfort with their existing Business Continuity Management (BCM) and Disaster Recovery (DR) planning process.

Often, organizations look for turn-key solutions for Business Continuity Management and Disaster Recovery efforts, resulting in failed implementations when the need arrives and misappropriated IT spend on technology that may be excellent configuration items but not necessarily the end all be all to a comprehensive Business Continuity Management and Disaster Recovery program. 

Before I start to explain what the reasons for failure may be, let’s look at several key indicators of the importance of business continuity and disaster recovery from a published CloudEndure survey of 118 respondents from organizations small (less than 100 employees) to large (over 1000 employees):

By the numbers reveal that over 92% of business owners/leaders consider availability of their business operations very important

By the numbers reveal that over 36% of business would lose over $100K for each day of downtime, with over 72% in the $10K-100K range.

Now that we have seen a snapshot of the importance of BCM and DR planning for organizations small and large, I have listed some reasons below that DR plans fail, and should be considerations to prevent your own BCM and DR plan from failing.

Reason Number One: You didn't even create a plan, it’s all based on technology.

Often businesses charge a senior member of management, normally in the IT space, to develop a business continuity and disaster recovery plan for the company. Sometimes, these newly elected "Masters of Disasters" are not so savvy when it comes to planning for timely recovery of business resources, which goes beyond the technology and also includes identification of core business functions and/or processes, along with associated recovery time frames, estimated financial impacts, operational impacts, and estimated personnel required for realistic business continuity and recovery operations. These leaders, while well intentioned, look for turnkey solutions and tend to invest a significant amount of spend on backup and recovery technology solutions for business recovery efforts as a measure of risk mitigation and\or remediation, coupled with a non-comprehensive, all technical process for governance of these recovery activities, or in more egregious instances no process at all.

While there is nothing wrong with spending a significant investment in backup and disaster recovery (BDR) technologies, it is not the end all be all to assure availability of the organization core business processes and it’s data. Some BDR solutions offered by service providers include replication technologies to cloud targets so that data can be accessed in the event of a disaster, but what’s most often times missing is a plan for transitioning systems depending on that data and file access to their alternate data source with minimal to no manual intervention, while also taking into account the required performance level of the online customer facing and internal systems that will be considered satisfactory to the people working with the systems and data; after all we are most times dealing with more than just standard document files, but also online database files, email databases, software configuration items, etc. It should be realized that it’s the plan, not the technology, that assures your business meets those all so important Recovery Time and Recovery Point objectives that should have been thought through and specified at the very beginning of your planning process. If not comprehensively planned, implemented holistically, and tested frequently these plans typically fail during the time of need. Avoid the disruption by working with a Senior IT Consultant or Virtual CIO who has extensive experience and knowledge planning for a comprehensive BCM and DR program for your company.

Reason Number Two: Your focus was always recovering from a disaster instead of keeping your services available and avoiding the disaster

In my observation of the services offered by many technology service providers in regards to business availability it is almost exclusively containing services titled “Disaster Recovery”. While most businesses have a tolerable level of downtime, most would rather avoid the disruption altogether. This practice which addresses these needs is known as business continuity management, which should always be a guiding principle and precursor to planning for recovery from any disaster. There are many risk mitigation steps which can be implemented by your technology service provider in advance of any impending disaster. A few examples are:

• High-availability servers (server clustering or mirroring)
• High-Availability data centers (active-active geographically independent configurations)
• High-availability configuration of telecommunication circuits
• Link aggregation for network configuration items
• Redundant power sources (grid to rack) for technology configuration items
• and a lot more

Furthermore, companies do not put together a plan to test their recovery environments and failover procedures for efficacy in continuing their business operations as if there were an actual disaster happening. With the sea of changes that can occur in a business’s technology environment, this is but another failure of planning effectively. Disaster Recovery and Business Continuity tests allow everyone throughout your organization, including those with roles in the DR process to find and remedy process breakdowns before it happens when you need it most. Always test your DR plan on an annual basis, and not just the systems, but also the logistics of said plan.

Inevitably, it is up to the senior leadership team to risk manage all the possible technology risks evident during different types of disaster events as well as the appropriate risk mitigation methods available to effectively plan for business continuity and disaster recovery. 

Reason Number Three: You did incorporate a comprehensive Business Impact Analysis in your planning process

While organizations often focus on planning for server failures, failures in the data center, loss of data and compute, and other technical infrastructure issues, most times they fail to plan appropriately for non-technical disruptions, such as threats posed by natural disasters (loss of facility, loss of workforce, loss of communications with FTEs, loss of life, etc.), which most times, if not always, results in significant business disruption followed by cash flow, payroll and employee morale issues which will eventually lead to a company closing its doors indefinitely.

A business impact analysis is a risk assessment process which incorporates your key business stakeholders into the planning process to understand the activities within each individual business unit to include the resources necessary to conduct their operations. These planning elements usually include key personnel, resources and technology, required inputs and outputs, and absolute critical technology systems needed for their business operations. Due to the concept of shadow IT, business units tend to use systems for critical business processes not known by IT, and as such, these aren’t appropriately planned for in a DR scenario. Furthermore, there are usually many dependencies for continued operations of the business unit that is usually off the radar by those conducting disaster planning (i.e. Hardcopy books used for references when systems access is interrupted, shared spreadsheets stored locally on someone’s desktop used by the whole department for a key business process, etc.) If you are only planning for business continuity and disaster recovery at a corporate level, then why even have a disaster recovery plan in the first place if it won’t help your core business units execute their business processes.

In closing, while there certainly are many more reasons why an organizations business continuity and disaster recovery planning process may fall short I hope this article makes clear that there are no turnkey solutions for effective and comprehensive disaster recovery and business continuity. As a metaphor, when a human being plans for survival, they don’t plan for near death activities, but rather the activities that will prevent them from getting into those near-death instances in the first place. BCM and DR planning is a huge undertaking, and for instances where failure is not an option, do your business a favor and work with a senior IT consultant or virtual CIO who has the required background, experience, and knowledge for planning for and leading these programs.