How we built our contingency plan - Fourth Financial Corp - Disaster Recovery

Fourth Financial Corporation is a 13-bank holding company in Kansas with assets of $4.3 billion. Until recently, contingency planning received minimal attention in our organization. We had some plans in place, but then were dated and incomplete.

Several years ago our interest in disaster recovery was heightened as a result of Banking Circular 177--a Federal regulation requiring banks to address the impact of a loss of EDP services, and how that loss would be dealt with throughout the banking organization.

Fourth Financial's assessment of that impact revealed an unacceptable level of risk because of the very large numbers of online transactions processed each day. Each of those transactions represents some type of customer relationship that would ultimately be affected from the loss of the computer utility.

To meet the BC-177 requirements, and ease our own minds about these "what if" scenarios, Fourth Financial contracted with Comdisco Disaster Recovery Services in 1988 to maintain a back up "hot site" and do regular testing at their recovery facilities. Initially, we prepared a comprehensive disaster recovery plan, with Comdisco's assistance, for the computer services area. We test this plan three to four times per year because of the volatile nature of our computer environment. Today we have proven recovery of our critical bank applications at the recovery site within 12-14 hours of arrival.

However, given the integration of today's corporate business departments and the interdependence of business functions, a loss affecting one department may cause a ripple effect throughout the corporation, magnifying the financial impact of the disaster.

Most corporations readily recognize the need for a plan to protect the computer services function, but fail to address the need for a total corporate plan. Having only a computer services contingency plan is like having only a spare tire for the contingency of a flat tire. It is no good without the jack and lug wrench.

Recognizing this inherent exposure, Fourth Financial decided to fully protect its corporate assets by hiring Comdisco's Consulting Services division to help us develop contingency plans for our other corporate business units.

Business impact analysis

The first step in our corporate contingency planning process was a business impact analysis to determine what impact the loss of various units would have on the bottom line. We divided the company into three levels of impact--high, medium and low--to prioritize our plan development activity.

During this initial assessment, we discovered the critical nature of our voice communications. Even though we have a network of about 2500 data terminals and ATMs throughout the state, our voice capability was just as important to the overall operation of our banks as our data network.

As a result of the business impact analysis, we have met with our local carrier and are in the process of putting in a second switch at another location, splitting our incoming trunks between two switches. Should the main switch fail, the second will cut in with the critical stations, allowing continued business operations with minimal disruptions.

Another by-product of the business impact analysis has been that we have achieved a much beter understanding of our critical business functions--which has made the actual recovery planning more effective.

Recovery plan development

Once we had analyzed all of the corporate units' impact on the bottom line, we proceeded with developing recovery plans for the non-computer departments. By the time we were finished we had 46 separate corporate contingency plans for each of the business units, as well as 12 contingency plans for the various functional areas of computer services.

While there are hundreds of details that go into this type of planning, there are a couple of basic concepts which must be considered.

Fourth Financial, like most companies, has its EDP operations in close proximity to corporate headquarters. This means should we lose the EDP area in a disaster, we most likely would also lose the staff areas too. We had to plan for an end user recovery capability as well as for the computer area.

It's important to realize that by protecting only the EDP assets and not the end users, you have the problem only halfsolved. The other fact, which is surprising in these high-tech times, is that all the data needed to operate the corporation is not on the computer--much of it is still in the file cabinets. These are the types of issues that need to be incorporated into your contingency plans.

The details that go into contingency planning can be overwhelming. To help protect us from detail-overload, we made extensive use of a software tool called ComPAS (Comdisco Plan Automation Software) to develop our 58 contingency plans. ComPAS is a very flexible software system that allowed us to tailor fit the contingency plan to our own needs.

Developing contingency plans is a lot like publishing an encyclopedia. It's a massive effort, and by the time you're finished, the facts may have changed. The use of consulting services and a good software tool compress the development time to keep major changes and rewrites to a minimum.

Quality assurance

The third phase of our corporate contingency planning involves Quality Assurance, which is form of maintenance. As part of our plan we conduct a quarterly review of the computer services plan and a semi-annual review of the corporate plans. This ensures that the inevitable changes that occur in a corporation's operation are not overlooked. The use of a good software tool makes incorporating those changes into the finished product a relatively painless task.

Today, Fourth Financial's overall corporate contingency plan consists of 58 individual department and computer services plans, totalling about 3500 pages of customized documentation. The most common question is how long did it take to put all this together.

The corporate plans took slightly less than one year to complete, with the computer services plans taking another eight months. While we haven't had any actual disasters to date, we are fully prepared.

Fiscally speaking it's made a lot of sense too, when you consider that the cost of our entire recovery site, salaries, test expenses, and plan preparation and maintenance for an entire year amounts to about 42-minutes of losses after a 24-hour outage.

COPYRIGHT 1991 Nelson Publishing
COPYRIGHT 2004 Gale Group