IT Sense: Too Many Cooks

by Jon William Toigo
October 1, 2004

Just back from a consulting gig with a large Midwestern company, I got a first-hand look at a problem my mom used to refer to as “too many cooks.” I also saw what it takes to solve the problem: a true CIO.

The company wanted to build availability and recovery into the infrastructure it was fielding to support applications from a recent acquisition. They identified four, mission-critical software “stacks” that needed to be recovered within five minutes of any disaster and planned to use a second data center, situated between 50 and 80 miles away, comparably equipped to their primary data center, as a recovery site.

The two sites were interconnected via a WAN link from which a 2Gbps pipe had been allocated for their use in mirroring data asynchronously on an ongoing basis. The transaction rates presented by the applications appeared to justify the assertion that the allocated link bandwidth would be adequate for their data replication and failover strategies. Processing could be redirected to the alternative site and data would be accessible within the specified recovery time objective. In short, it was a wonderful world of wonderfulness.

Prior to my onsite visit, the CIO sent me a paper describing the strategy. After reading it, I called her to see about canceling the trip altogether. It was clear the development team had their ducks in a row, so why spend good money on a consultant site visit just to confirm what they already knew?

Despite my protests, they continued to press me to come to their site, regardless of my view of their strategy. It seemed the equipment vendor, whose consulting arm was being used by my client to assist with the transplanting of the acquired applications into the new portfolio, had written a document critical of the disaster recovery strategy that had raised red flags among the honchos at the company. Senior management was suddenly concerned its mission-critical apps were at risk, so the CIO and project director wanted me to visit their shop less to consult than to assuage management’s concerns.

I used every excuse I could think of to beg off the trip, but to no avail. I found myself on a flight, grinding my teeth all the way about my own failure to get out of it diplomatically.

Fortunately, when I arrived at the site, I found myself surrounded by a project team that was intelligent, friendly, and competent—from the CIO down to the junior staff. They had done their job well. In fact, better than well.

With today’s complex, n-tier client/server environment, it’s essential to design data protection and disaster recovery into application and storage architecture from the get-go. Unlike the bygone era of “secretary-friendly” disaster recovery planning for Big Iron environments, data protection can no longer be considered a “bolt-on” to an existing system or storage platform. Without some proactive planning during application design and platform selection, most companies confront a requirement to replace all equipment on a one-for-one basis with what they have installed in-house. Moreover, they must replicate infrastructure down to the IP address and hardware MAC address in order to resuscitate the failed n-tier app at the recovery site. To their credit, these guys had done their homework.