Consolidation & Virtualization Pay Off Big at First National of Nebraska

by Mary E. Shacklett
March 31, 2008

** Read this article online at http://www.mainframezone.com/it-management/consolidation-virtualization-pay-off-big-at-first-national-of-nebraska


With $20 billion in assets and more than 7,000 employees, First National of Nebraska (www.fnni.com) is one of the top-50 financial services holding companies in the U.S. Its headquarters is Omaha, NE, where it has been operating for the past 150 years. Today, First National serves more than 6.6 million customers in all 50 states, with more than 90 banking locations in Nebraska, Iowa, Illinois, Colorado, Kansas, South Dakota, and Texas. First National is the ninth largest merchant processor, a top-10 commercial card issuer, the eleventh largest U.S. issuer of bank credit cards, a top-20 electronic funds processor, the ninth largest U.S. agricultural lender with customers in 49 states, and one of the leading providers of project finance to the ethanol industry.

Being the CIO of such a diverse banking organization requires flexibility and attunement to diverse lines of business, each with its own specialized requirements for processing, security, business agility, and asset management.

“We have three distinct business areas that IT provides services to,” says Ken Kucera, First National’s CIO. “One segment of our banking market consists of small community banks that serve towns of 1,000 to 30,000 people. Each of these banks possesses its own operating charter and lending limits—but its technology is fully run from our 32-processor System z mainframe in the Omaha data center. At the other end of the banking spectrum, we have large metropolitan banks in cities such as Omaha and Dallas. The transactions and back-office activities of these banks also are processed through our System z. The third segment of our business is card and merchant processing. We provide national card processing and settlement services for credit and debit cards, and merchant processing and settlement services for major retailers such as L.L. Bean and The Gap. All this settlement activity goes through the mainframe.”

Kucera says that First National processes millions of transactions through its z9 mainframe every day. “We use proven System z software such as CICS and IMS for online processing, and make heavy use of VSAM for our batch processing,” he says.

Since First National’s z9 processes transactions for more than 20 separate banks, it uses a system of chargebacks for centralized service to these customer institutions. “We base our chargeback structure on the number of customer accounts and the number of transactions per institution,” says Kucera. “We then look at storage and CPU usage, and we come up with a weighted average cost, which is then proportionately assigned to each institution.”

In addition to its core transaction processing on z/OS, First National also has six Integrated Facilities for Linux (IFLs) on its z9 that run applications for customer service, branch systems, backroom operations, and Website traffic. “These systems formerly were run on separate servers, but we consolidated them on the System z because of its reliability and greater economy of operation,” says Kucera.


Building the Case for Server Consolidation to a Mainframe PlatformWhen Kucera became First National’s CIO in 2003, the first thing he did was to tour the data center.

“I saw 600 WinTel boxes, a Tandem computer, the zSeries mainframe and 40 Sun servers,” he says. “The immediate question I asked myself was, “Why can’t we put some of these boxes on the mainframe?’”

Kucera’s vision encompassed more than the simple consolidation of physical servers. He wanted to improve data center manageability, scalability, and reliability.

“Simply stated, the fewer touchpoints we had, the more we could marshal our resources and our quality of service for network monitoring, troubleshooting, and reliability,” says Kucera. “I knew upfront that the reliability we could achieve on the mainframe was virtually unbeatable. I also knew we would be able to guarantee the high level of failover redundancy required for banking operations by using virtual systems on a single mainframe platform, where it would be effortless to switch an application to another virtual machine partition in the event of a problem.”

Arvind Thapar, First National’s vice president of Infor-mation Technology, saw strength in the System z mainframe platform that would carry First National well into the future.

“We had all of our core legacy applications for transaction processing on the mainframe,” says Thapar, “And quite frankly, we have and continue to find the System z to be one of the best platforms out there. We have significant investments in System z applications, and with the R&D investments that IBM continues to make to the System z, such as new capabilities like zAAPs (System z Application Assist Processors), the mainframe isn’t going anywhere. Its robust security and the reliability of its architecture allow us to deliver high-quality transaction processing day-in and day-out. The system simply does not break.”


Securing Corporate SupportSecuring corporate support for a project that essentially was an application migration and decommissioning of servers—without yielding large, new applications for anxious end users—is always a risk for any CIO.

Kucera’s challenge was twofold:

“The first thing I did was to go to the CEO and tell him of the server dilemma and the advantages of server consolidation,” says Kucera. “Then, I asked him for funding for the migration project. . . . I told him about having a better environment for corporate security, reducing network management complexity along with the number of processors, and reducing the costs of running IT operations in the long term. He didn’t think the investment or the duration of the project were too large.”

Kucera also developed a plan to keep forging ahead with new application development and same-level, day-to-day support for production applications while the migration was going on.

“Our IT headcount is 494 people,” says Kucera. “Of this number, less than 25 persons were allocated to the server migration and virtualization project.” IT still continued to develop business applications, although not as many. This development effort included building out First National’s intranet applications and continuing its SOA (Service-Oriented Architecture) development. At the end of the 18-month server consolidation and virtualization project, First National had eliminated 572 separate physical servers. These servers were replaced with five IFLs on the System z and a sixth IFL for failover and backup. Network administration was simplified. Support staff on the server side went from 21 persons to eight.


Solving the Technology IssuesKucera recalls that it seemed relatively straightforward to migrate physical resources to the System z mainframe—but that it didn’t turn out to be quite that simple.

“Initially, we didn’t fully anticipate the number of interfaces between systems when it came to running applications and back-end processing,” he says. “Assuring that all these systems coordinated, and that all interfaces were accounted for in the new environment, took extra time.” As an early adopter of virtualization on the System z, First National also discovered that establishing one IFL on the z9 to handle the workload wasn’t going to work exactly as planned.

“We saw the one IFL really getting packed,” says Thapar. “It was a matter of trying out different approaches until we found one that worked. … We ended up with five IFLs on the System z to handle the workload.”

Since the early days of the project, First National has done significant system tuning. “We have learned so much,” says Thapar. “We initially discovered that Linux under z/VM was very finicky about how much memory gets allocated to each guest application. We now know how much memory to allocate, and how to condense Linux guests on the mainframe. We also have standardized some 400 Java procedures originally written for the Sun platform that we had to rewrite for SQL.”


Hitting a Snag With IT Staff Buy-InKucera is quick to say that any paradigm shift in IT system deployment can run into IT staff resistance.

“I would have to say that the most difficult part of this project was on the people [side] and not on the technology side,” he acknowledges. “As a CIO, you think about this constantly, because successful projects aren’t achieved unless there’s staff buy-in.”

In Kucera’s case, he was facing some pretty serious staff issues during the server consolidation project.

Several high-level members of his staff were fiercely loyal to the Solaris and SuSE Linux platforms they had been maintaining—and to the technical skills they had honed over the years in supporting these platforms. “They didn’t want to see the physical Sun and Linux servers eliminated and virtualized, so they became uncooperative, and it seemed like certain tasks in the project plan began to take forever to complete, with problems continuously arising to stop momentum,” recalls Thapar.

The situation finally came to a head, and Ken Kucera made the hard choice. “Changes with senior staff members who opposed the consolidation needed to be made so we could move the project forward,” he says. “Those senior staff members were given the opportunity to pursue opportunities elsewhere.” Morale issues in IT resolved themselves once the resistors were no longer there, and project tasks that had inched along for nearly 12 months were quickly resolved and completed during the next six months.

“These were good people with a long list of contributions, and it was difficult to have to part ways with them,” says Kucera. “But the server consolidation was very important to the company from the standpoint of security, manageability, reliability, and long-term investment. The senior people who left had a philosophical difference, and it made the most sense for them to pursue other options that were more in line with their views.”


Reaping the Business BenefitsFirst National has steadily seen the benefits of its server consolidation in these key areas:

The IT staff has undergone training to facilitate knowledge transfer to the new environment, and Kucera also has made a couple of strategic hires of top talent to give him additional System z expertise.

“It is difficult to find high-level System z people in the employment market,” says Kucera. “We had to hire a couple of top-gun experts for Linux on System z. They were hard to find, but eventually we got them.” Kucera combined these key hires with retained staff from the Sun Unix side to support the new virtual Linux environment on the System z. “This latter task wasn’t too difficult,” he says. “We have found that people who already know Unix can make a relatively easy transition to learning and supporting Linux.”

Kucera also says that the percentage of his staff available for work on new application development and other IT initiatives has increased since the server consolidation and virtualization project.

“The server consolidation has allowed us to ‘keep the lights on’ for less,” he says. “It has streamlined systems support and upgrades, and has allowed us to move more personnel from strictly maintenance functions. Before the consolidation, I would estimate that 60 percent of the IT staff was working on new applications and initiatives, and that 40 percent was committed to maintenance and to ongoing daily operations. Now, that percentage split has moved to 70 percent of IT personnel on new development and initiatives, and 30 percent for maintenance and daily operations.”

The improvement is meaningful for the work facing Kucera and his staff. More regulatory and compliance requirements are being directed to financial institutions that must be implemented on the IT side within very specific timeframes. Recent compliance measures have appeared in areas such as the Patriot Act, money laundering and the Office of Foreign Assets Control (OFAC).

Kucera still estimates that about 12 percent of his Intel server base remains in place for specialized applications requiring dedicated server support. However, wherever possible, First National IT strives to put new applications on its virtualized System z platform.

“You can always overcome the technology issues in a project like this and reap the benefits,” says Kucera. “The key is having the IT staff fully behind the project. We found that if you don’t have that commitment, the project will never get done.”


Concluding RemarksTaking on a server consolidation project is fraught with risk, especially when it requires upfront investment with very little return on new application development for the end business.

“A CIO is required to make a lot of tough decisions like this,” says Kucera. “Naturally, a server consolidation project presents its challenges because you’re going in to ask the CEO for more money, and all you’re doing in many people’s minds is changing the number of boxes in the data center. For a CIO, there’s always the temptation to just ‘stay under the radar’ and avoid decisions like this. I found myself taking a hard look at the dilemma and asking myself was I going to stay under the radar—or was I going to do something about a burgeoning data center that needed to be streamlined so it could work more effectively and produce greater benefits in the future?”

For Kucera and First National Bank, consolidation and virtualization paid off in immediate cost reductions—and has given the financial services organization technology and business agility to face a world of constant business, regulatory, and technology change. ME

This article had no comments at the time of this printout.