Maintaining a disaster recovery worksite is easy. In the case of a disruptive event, moving operations and people to a DR site can be chaos without solid, tested procedures in place. Here’s an inventive idea on how to ensure business continuity procedures are executed flawlessly.

A FIBERTOWN oil and gas customer based in Houston uses an A and B team deployment strategy to ensure disaster recovery operations run smooth and seamless.

This company has an A Team of 15-20 staff members (including IT, operations and HR), who are the “first responders” for any disaster recovery/business continuity deployment.  They head to their DR site – 100 miles northwest Houston – in advance of a storm to prepare for transfer of operations from the corporate office.

The A Team is deployed when a storm is 72 hours off the coast to assemble DR operations. They complete a checklist that includes anything from making sure phone systems are properly routed, organizing docking stations and work areas, and even ensuring there are plenty of coffee and supplies.

While this plan of attack is not uncommon, what’s unique about it is the way this company treats this on-call, A Team staff.

If needed, the company will actually house the families of the staff during a deployment, even finding a kennel for the family pets. The company also has plans in place for a contractor to go to the homes of the families and board the windows and doors. It also has a security contract to monitor the homes while these mission critical staff are away.

DR/BC deployments can be stressful on the business but even more so on the staff as they have to leave family behind to do their jobs.  This company does everything it can to keep the staff worry-free during deployment by ensuring their family (even Rover) is safe.

After the A Team readies the site and a storm is approaching the coast, the B Team will activate. This staff will head to the DR site in Bryan/College Station, set their families up in corporate assigned hotel rooms and activate operations.

For large organizations, this A/B team strategy is an inspiring way to ensure DR operations run without a hitch.

Every business has some kind of insurance these days. We have monetary insurance for theft, lawsuits, natural disasters and employee misconduct. What about business continuity? Why are companies so hesitant to invest in BCP and disaster recovery unless it’s mandated by shareholders or for compliance? If business continuity were looked at as insurance for your business, it would never go unpaid, unplanned or untested.

Think about this in terms of disaster recovery. If your applications and mission critical data are wiped out or even lost for a period of time, is your company out of business? According to the Houston Area Research Center, every dollar spent on disaster preparedness saves seven dollars in recovering disaster-related economic losses.

There is not a cookie cutter business continuity plan for all. Companies should consider best practices while preparing and customizing their own plan.

Get started by building a checklist. Put each of the checklist items in one of three categories – Completed, In Progress or Not Started.

Planning for the impact of an unexpected or catastrophic event on your business

- Identify a coordinator and/or team with defined roles for preparedness and response planning. Potential team members may include: Information Security, Operations, Systems, Police/Security, Physical Plant, Insurance, Legal Affairs, Public Affairs, Personnel Department, Comptroller, Audit Division, Safety Office and/or Emergency Response Team.

- Conduct a business process and services inventory to understand which processes are mission-critical to the survivability of the business.

- Determine acceptable levels of service during the recovery period, and what processes need to be maintained or restored first to keep the business running.

- Identify essential employees and other critical inputs (sub-contractors, services, logistics, etc.) required to maintain business operations by location and function during the event.

- Conduct a technology asset inventory to determine and document the mission-critical technology components, their location, how they’re configured, and who is responsible for management.

- Once key components are identified, determine what measures should be taken to protect and recover them.

- Understand the rules or regulations governing your business operations. If you had a business failure, would you be able to maintain compliance? (Sarbanes Oxley, HIPPA, privacy, etc.).

- Understand customer or business partner performance metrics/service level agreements to assess risk for breach of contract, or to put in place performance remedies for your customers.

- Identify a budget: Quantify the potential costs of downtime or total business failure. Develop a business case to optimally invest in risk mitigation.

Assessing your data and technology needs in the event of a failure in operations

- Determine the status of your existing disaster recovery plan. Do you have one and is it maintained? Have you tested the plan?

- Determine vulnerability of your organization’s technology infrastructure to natural disasters, including hurricanes, floods, fires, earthquakes, etc.

- Set clear recovery time objectives for each of your business/technology areas.

- Determine the need for off-site data storage and backup.

- Develop a technology plan that includes hardware, software, facilities and service vendors.
Secure clear understanding and commitment from vendors on your plan.

- Secure a backup vendor, if necessary, to perform that critical function if your primary vendor is impacted by a business failure

- Perform security risk assessments around specific threats where possible. Examples of data security include: virus protection, intrusion detection, hacker prevention, network events, component failures and systems crashes.

- Assess, if possible and per prior events, how quickly and accurately your business and technology were restored by existing staff. What were the lessons learned so they can be addressed in future planning?

- Determine the effectiveness of your data backup and recovery policies and procedures. Are the procedures fully documented and an appropriate staff member responsible for the maintenance of that documentation?

- Perform a data recovery test. Was the test successful?

- Prepare an incident plan for mitigating a security breach. Audit annually, as security threats can change.

Communicating your plan to employees and vendor partners

- Determine who needs to be contacted with critical information. Build distribution lists and maintain for accuracy.

- Develop a contact plan to reach employees: wireless, home, etc.

- Ensure employees know where to receive information and updates about whether they can return to work, or if they are to report to a different location (Internet, conference bridges, etc.).

- Ensure mission-critical employees know their role in the plan and have access from remote locations (i.e., home broadband, phone, VPN for security).

- Make sure the plan can be executed by alternate employees who are not necessarily the “expert” in cases where those employees cannot be reached.

- Determine the need for a designated recovery site for your people to resume work. Plan for communications, data connectivity, desktops and workspace at that site.

- If you require support from vendor partners, ensure they also have a documented plan that complements your needs. Review periodically to keep the plan current.

Coordinating with external organizations and helping your community

- Collaborate with your local government agency to share your plans and understanding of their capabilities in the event of a business-impacting catastrophe.

- Share your plan with your building management so they have a clear understanding of their role in safely securing the building and your employees.

- Share best practices with other business leaders in your community, chambers of commerce and business associations to improve community response efforts.

This Tuesday, I was excited to use my new Chase iPhone app, which lets me deposit checks using the camera interface. Much to my disappointment, when I tried to log on to the app it said Temporarily Unavailable. I then tried to access my account online, it too said Temporarily Unavailable. Ugh.

I immediately searched the Web for a reason to satisfy my curiosity. I read many comments on Twitter from frustrated customers and a blog post claiming the banking giant’s online systems were experiencing technical issues.

This got me thinking…I may only be somewhat inconvenienced by the outage, but how is this affecting the 16 million+ other customers trying to access accounts and continue with business as usual?

Downtime not only affects our business transactions and revenue-generating activities, but leads to customer dissatisfaction and reputation damage. Couple this with Chase’s remarks that the downtime was “scheduled maintenance,” and you have a PR nightmare on your hands.

One of the best ways to avoid downtime and ensure business continuity is securing your mission critical systems out of harms way and with a trusted colocation partner. Many data centers offer 100% uptime guarantees and are built for redundancy, high performance and security.

Chase may be back up and running as of 1 a.m. Wednesday, but the damage is done. I wasn’t burdened enough to take my business and run to another bank, but what about those customers that were extremely affected?

It’s of vital importance to have your company’s data and mission critical systems protected and accessible before, during and after a disruptive event. We have all seen the dollar amounts associated with downtime, but for a number of reasons many organizations continue to adopt the approach “we will be OK.”

According to the 2010 hurricane forecasts, this is not the year to assume you will be OK. We are already seeing activity with Hurricane Alex.

U.S. meteorologists at the National Hurricane Center released the 2010 Atlantic hurricane season forecast calling for a very active season with 18 named storms, including 10 hurricanes. The leading research team for hurricanes based at Colorado State University provided additional support of this forecast by stating of those 10 hurricanes, five were expected to develop into major hurricanes.

Whether you believe the probability of the research or not…the real question is if your organization is ready for a repeat of some of the past events to hit the Texas Gulf Coast?

Have you made the necessary plans to continue operations regardless of the severity of the disaster?

This season is going to be an active one, so make sure you have answered the following questions and backed up those answers with a written plan that has been exercised through real-time DR drills.

Are your assets protected and backed-up?

Be prepared. When an event occurs, you can rest easy knowing your job is not at stake because your company will not be able to access the applications or data it needs to continue operations. Or even worse, that data will be lost or destroyed.

Have you verified the effectiveness of your emergency response?

This takes practice. As a youth sports coach, I tell athletes that practice doesn’t make for perfect play, but rather “perfect practice” makes for perfect play. In the case of youth sports, millions of dollars is not at risk. When it comes to your organization, it is.

Have you mitigated your risks?

Not everyone has the financial resources to guarantee zero downtime, but you can minimize the risks your company is willing to take. One example is the location of your IT infrastructure. The costs are relatively the same to have your IT assets in a data center, a safe distance from the Gulf Coast, versus housing them onsite. Yet many organizations have not mitigated this risk.

Are you ready to continue doing “business as usual” when an event occurs?

Think business continuity, not disaster recovery. In most cases, a disastrous event is not long in duration, but the aftermath and clean-up can take days and even weeks. If prepared, your only concern will be for your own personal belongings and not whether your company can continue to access and use applications, which drive the business.

If you are part of an organization that deemed yourself lucky following events such as Katrina, Rita, Wilma, Ike or Allison, then hopefully you have heeded the warning and are completely prepared for a repeat. Based on this year’s forecast, a repeat seems more likely than ever. With the proper planning you and your organization can be one of the success stories.

With the latest technology improvements for data storage, tapes are no longer the ideal method for backing up. However, there will always be a use for them, and some companies still rely on tapes to store their mission critical data. If you have to use tapes as a method for backing up, follow these steps to ensure the process runs smooth.

Verification

After you have written to the tape, make sure you verify that the data was written correctly and not corrupted. Most companies – due to time constraint – will only do a quick verification through the backup software. This can lead to failures.

Make sure to do a full back up AND a full verification at least once a week. This will take longer, but will give you a better idea if the data was corrupted on the transfer to tape. And not to mention, it will let you know if the tape is bad to begin with.

Disaster Recovery

Why do you back up your data? Most assuredly it’s to prevent loss in the event of a natural disaster, to meet compliance requirements or to manage human error.

Tapes must be stored offsite. Some IT managers take them home or use a service that provides a fire-protected storage facility. You probably have to have a catalog system and rely on the service company to ship you the tapes when needed.

Storage

I have seen companies that store tapes in a room where on the other side of the wall is a large generator that produces electrical fields that can scramble data on the tapes. Store tapes in a stable environment. Keep them away from extreme temperatures, humidity or anything magnetized.

Rotation

Use a rotation method for your tapes. This means use more than one tape so you are not using the same one over and over.

Most companies use a five day rotation. Use a different tape for each day of the week, and then do a full backup on Saturday.

When the next week begins on Monday, use the Monday tape from last week. This will preserve the tape lifecycle. Plus, if you had a corruption, such as a virus you didn’t catch for a couple days, you can go back to the previous tape or back before the virus was first introduced. The worst is trying to restore from a tape that you backed up with the virus…OOPS!

Maintenance

Keep up with maintenance of tape drives regularly. Clean your tape drives so dust and particles don’t transfer to the tapes. This process is usually done every 12 to 18 months.

Tape drives will fail, so make sure you have a maintenance contract with the tape drive vendor. It’s easy to send a drive back and get a new one. However, the tapes that the drive was writing to may have been affected in the process, and that data could be lost forever.

If you’re using tapes in your storage and back up procedures, what methods are working for you?

Magnetic tapes have been used for more than 50 years to store data. Because tapes are inexpensive and mainly used for archiving and backing up, companies prefer this method by default. It may be a mature technology, but it definitely has its drawbacks. Here’s what everyone ought to know about the disadvantages of using tape backups.

Accessibility

Tape storage is accessed sequentially by scanning through the tape until the data you need is located. Think…rewinding and fast forwarding through an entire cassette tape to hear your favorite song. This is antiquated and slow compared to using disk drives. Also, tape storage adds infrastructure and management costs because you now need tape management software or special hardware to make this process faster.

Reliability

Each time you read or write to a tape, you take some of the lifecycle of that tape away. It’s like when we used to listen to cassette tapes. It sounded fantastic for the first few months, then it began to lose its quality. Sammy Hagar’s voice with Van Halen began to muffle over time. It just wasn’t Sammy!

Business Continuity

Tape drives must be kept handy to read and recover from tape storage. The disadvantage of this is now you have to maintain the drive and the tape and store them in a safe place. What happens if the technology becomes obsolete? Who is responsible for managing this and are you sure it will work in the event of a disaster?

You can test your drive and tape OR just pray when you need to restore. Wait! Testing reduces the lifecycle of the tape and leaves you with something other than the real Sammy. Not a very secure continuity plan, huh?

Cost

Most people would agree that tape storage as a media is cheaper than disk. However, I can’t count how many times I’ve heard…“ I have thousands of tapes to manage now when I only started with one hundred!”

Over time, companies obtain multiple copies of the same data. Tape requirements keep growing without end. So, you use more hardware and special software to try and alleviate this process. Hmmm…I think it would have been cheaper to invest in disks to start with.

*Note: If you’re a public company, Sarbanes-Oxley requires you to keep tapes for 7 years.

Process

How do tapes and drives work with your current processes? You probably restore to a hard drive on the application server (virtual or not). So, managing antiquated tape restoration on the latest server equipment can be daunting.

Also, the actual tape backup process takes a long time and will consume your network and slow down applications. Conducting backups during business hours is not a viable process.

No IT manager wants that call from his CEO asking, “What the (insert explicative here) is going on?” Companies have adopted the process of doing full backups on weekends or overnight when everyone has gone home. But what if you are a 24-hour shop or a global enterprise spanning multiple time zones?

I don’t mean to bash tapes, as there will always be a use for them. I believe the term is cold storage or archival. Use tapes as a method of backing up and storing data that is not critical.

It’s no secret that proper communication is vital to running successful business operations. Throw in a category 5 hurricane or other natural disaster and communications can come to a screeching halt. As Gulf Coast companies begin updating and testing their disaster recovery plans, it’s crucial to focus on staying in touch before, during and after the storm.

Here are five little known ways to keep lines of communication open.

Internet Telephony

Many businesses, both large and small, are employing VoIP phone systems to reduce infrastructure and communication costs. Users with Internet access and electricity can connect via voice-over IP phones from anywhere in the world. In the event of disaster, users can take their office phone and plug in to a laptop for instant communication capabilities.

Vonage phones are an affordable option to stay in touch with clients. All you need is an Internet connection and adapter for your portable phone. Calls will route from your business number to the phone and allow you to continue receiving normal communications.

SMS Messaging

For those of us that endured Hurricane Ike, we remember how hard it was to use our cell phones. Calls weren’t getting through, but text messages were a savior. Implement an SMS policy for employees to update management with their “hunker down” plans and to check in after the storm.

Social Media

Companies with Facebook pages or Twitter accounts can leverage social media to keep communications with employees and customers flowing.

• Use status updates to send messages to your team and let clients know that operations are up and running.
• Communicate with key personnel about DR procedures via direct Tweets and Facebook inboxes. These messages are kept private and will not show up in their main stream.
• Set up a private Twitter group for confidential messaging among a business unit.

Cloud-Based Email Services

Small and mid-size businesses should consider utilizing a hosted Exchange email service like Apptix or an online email security service like McAfee/MX Logic to manage their email communications. For a couple dollars per inbox per month, you can enjoy the benefits of spam and virus protection with added email continuity. When unplanned outages occur, services like these allow quick access via the Web and archive messages for more than a month.

Two-Way Radio

Never underestimate the power of a two-way radio. You don’t have to be a trucker or survivor man to understand that in a digital world, sometimes old school is best.

The secret to each of these ideas is developing a tested plan to ensure your employees are prepared. Arm them with the proper procedures, and make sure they have emergency numbers readily available.

Proper storage is essential to resuming operations after a disruptive event. Choose storage options that work for your business. After determining your RTO and RPO, the next step is investing in infrastructure objectives. How can you be sure to choose the right kind of storage?

Mission critical data loss is a key concern for IT departments, large and small. Lacking the appropriate hardware with functionality in place to ensure data protection can result in long-term negative impacts to more than just the bottom line. These impacts – poor market perception, reputation damage and diminished competitive edge – can cripple organizations.

The hard facts…

“More than 40% of all SMBs will go out of business if they can’t get to their data in the first 24 hours after a crisis.” – Gartner

“Some 43% of companies never resume business following a major fire. Another 35% are out of business within 3 years.” – U.S. National Fire Protection Agency

Define What to Recover

Most companies, due to the high cost of disaster recovery, will only secure mission critical data. However, separating proprietary data or compliance driven data from data that is less important can be daunting.

After determining what data needs to be recovered and how much storage is needed, focus on infrastructure.

“Small companies often spend more time planning their company picnics than for an event that could put them out of business.” — Katherine Heaviside, Epoch 5

Establish Data Storage

DASD (Direct Access Storage Device), tape storage and high-end storage array each have their advantages and disadvantages.

• DASD is directly attached to the server. It is inexpensive, but acts as a single point of failure. Also, without using special software, you can’t share the resources with other servers.

• Tape storage is the oldest and most common among backup strategies as it is portable to store offsite. However, tapes are hard to manage, grow exponentially and are not as reliable when you need to restore. Not to mention, if you don’t have some form of de-duplication, tapes can double every time you do a full backup. They are more commonly used for long-term or archival storage.

• High-End Storage Array – In my opinion, this is the most efficient way to data protect. Using a SAN (storage area network) or a NAS (networked attached storage) device from a leading manufacturer has no single point of failure built into the system. The data is replicated within the array and can be restored in an instant. Plus, the hard drives can be allocated as shared storage. With data management software built into the array, you can back up and manage data on the fly and send it to other servers in the event of a downed server.

High-end storage array is most efficient in the event of a site disaster because data can be replicated to another array at a secondary site. This could mean your company felt a small hiccup and your customers never know you had to restore. These arrays have a higher initial cost, but will give you the better return on investment.

Other things to consider when preparing your data and infrastructure are your data center needs. For large companies and organizations with many servers, colocation and managed hosting are options that allow for reduced energy costs, guaranteed availability of resources and connectivity, and ensured compliance and eco-efficiency requirements.

What options are you using for data storage, and why do they work best for your company/industry??

If money were no object, all large organizations would have fully developed hot sites, dual or triple-cored data centers and the latest technology enhancements to ensure instantaneous failovers are imperceptible to the end customer. The reality is that IT budgets are shrinking, and we are all faced with doing more with less. Understanding the basics of DR and BC planning arms IT directors and CTOs with the confidence that allows for a good night sleep.

How do you begin the process of planning the continuity of your business or a DR plan that is affordable and attainable? It all begins with understanding the parameters under which your business unit must operate in order to reduce the risk of unacceptable failure.

Step one begins with understanding your Recovery Time Objective and Recovery Point Objective. RTO is your maximum allowable or maximum tolerable outage. RPO describes the acceptable amount of data loss measured in time.

These concepts are key elements in the proper design of a BC/DR plan. More importantly, they define a parameter that ultimately drives cost. By understanding RTO and RPO and setting achievable and affordable limits, IT managers can write and test a plan that is appropriate for the business unit.

A small accounting firm can probably survive the loss of a days worth of data entry. Would it be a fun and happy experience to recreate all those key punches? No. However, a day’s worth of data entry is not an insurmountable task to reenter into a system. An RPO of 24 hours may be acceptable to an accounting firm

For any bank or financial institution, an RPO of 24 hours could mean many millions of dollars in lost transaction revenue. Therein lies the cost driver. A suitable BC plan for an accounting firm may involve an offsite backup every 24 hours, while a bank needs real-time transactional replication across multiple sites. The cost difference in these scenarios is astronomical.

Determine your RTO and RPO and don’t buy more business continuity than you need…and don’t buy less than you need either.

With hurricane season right around the corner, many Gulf Coast-based companies are revamping their disaster recovery procedures and testing their BC plans. The unfortunate truth is many of them will wait until the last minute and key elements will be missed. In a five part series, we discuss the essential pieces of an effective and affordable DR/BC plan – RPO/RTO, Communications, People, Data and Testing.

Recovery Time Objective & Recovery Point Objective

The first step to designing or reworking your DR plans is considering the amount of time and at which point recovery of operations is necessary for sustainability. RTO describes the total amount of time an organization can experience interrupted operations. RPO is the total amount of data loss tolerable in terms of time. Understanding these levels creates a foundation from which to design your plan. They play a significant role in determining where your people will go and how to communicate with them, from where to access your data, where to harbor your applications and storage area network, and how quickly these steps need to happen.

Communications

Restoring communication with key employees is a major step after a disaster. Whether you’re a large corporation or small organization, many options exist to reconnecting after an event. Internet-based social networking applications like Twitter and Facebook are not usually affected by Gulf Coast storms and may be a good way to send updates to displaced employees. Small organizations can implement text messaging check-in for employees. Coordinating these procedures in advance ensures constant communication before, during and after a storm.

People

Securing the safety of your people is crucial to not only continuing operations, but ensuring employees and their families are away from harm. If relocating key employees to an offsite hot-site is an option, have you considered bringing their families along and where they’ll stay? If you have employees working from home, how will you address connectivity and communications? People are the most important asset of any organization and should be properly cared for and considered in procedure planning.

Data & Applications

Proper storage, connectivity and power are essential to resuming operations after a disruptive event. For small organizations, managed hosting options ensure business continuance. Where is your hosting provider located? What kind of procedures do they have in place for disaster recovery?

For large corporations, evaluating whether to host your data center internally or outsource to a colocation provider is a major decision. If you plan to “hunker down,” where are your data and applications stored to ensure connectivity, power and uninterrupted data backup?

Testing

No plan is guaranteed without first testing and testing and testing. Are your people properly tested in communication and travel options? Is your data backup method guaranteed to deliver 100% uptime? Will you have uninterrupted connectivity; how can you be sure?

Seasoned IT professionals understand the significant elements of a successful BC plan and DR procedures. In course of this series, we’ll break down each of these elements and introduce new technologies and considerations. Stay tuned!

Check out Part 1: How Much Downtime Can Your Business Afford? (RPO and RTO)