Storage & Data Management: Optimize Your Data Archiving Without Sacrificing Performance

by Jim O'Connor
December 1, 2008

Tier-one storage requirements put a significant amount of fiscal strain on organizations as a result of government-instituted compliance regulations. Naturally, companies seek to minimize their reliance on this costly storage media, but often find that the sacrifice in performance and data availability is too great to transition the data to tier-two storage. Sarbanes-Oxley, HIPAA, and the Securities and Exchange Commission not only require companies to ensure quick and reliable access to information, but also dictate the amount of time certain data must be kept. This is heavily contributing to the amount of storage capacity needed.

Tier-one storage is built to provide the highest levels of access, security, and availability. Widespread deployment throughout an enterprise is impractical, so it’s typically reserved for mission-critical business functions such as Online Transaction Processing (OLTP), and can handle transaction numbers into the hundreds-of-thousands per second. Many CIOs and data center managers struggle with the challenge of balancing tier-one storage costs against federally regulated archival requirements.

In industries such as banking and investment, every transaction, timestamp, email, etc. must be kept. This holds true for all publicly traded companies as well. While some data will be frequently accessed for a period of time, all the information is required to be archived and available for any future needs, whether it’s for fraud investigation or portfolio history.

Typically, institutions establish proprietary algorithms to determine when data can be rolled off tier-one and onto tape. Once the data is relegated to tape, queries require offline batch processes that are costly and time-consuming, so it’s important for these institutions to find a balance between the costs of maintaining high-performance, tier-one platters and offline data access.

Finding this balance is only half the battle, unfortunately. Once data has been successfully transitioned from top-tier storage, the question becomes, “What’s going to happen to this data over the long run?” On the short end of things, some compliance regulations require data to be held for only 10 years. On the other end of the spectrum (such as medical records and patient history in the healthcare field), archival data is required to be stored and maintained for a much greater period of time. Not all storage medium in existence today has that kind of longevity, so a technology transition and maintenance plan must be implemented and followed.

Today, I still talk to companies with tapes that are 30 years old. They’ve diligently stored and protected the tapes per HIPAA standards so they can review the data if necessary for fraud, medical malpractice, trends, symptoms, and exposure. But what are the chances of ever being able to read that data? Unfortunately, it’s very low. The estimated life expectancy of some of these tapes is eight years. When taking into account the current innovation cycle, it isn’t hard to understand why companies are struggling to maintain three to four generations of tape technology to satisfy potential reporting requirements.