November 2000

Lost in Email? 5 Archive Solutions

By Bruce Silver

• SIDEBAR: How Mail Archiving Works

Email today is more than just a communications medium. Corporate email systems like Microsoft Exchange have become long-term stores of critical business information. According to one market study, one third of the information used by employees of large companies resides in email.

The only problem is that the Exchange message store was never meant to be a permanent repository. It is a single database file on the Exchange Server, effectively limited in size by system performance and nightly backup time to 20 GB to 30 GB. The finite size of the message store translates to limits on the size of each user mailbox. A user's typical 45 MB allocation makes the mail folder at best a temporary repository - 27 days' worth at the average rate of 1.7 MB per user per day. To free up space, users either have to delete some messages or move them somewhere else - to an Exchange personal store on the desktop or a network file server. The Microsoft Outlook client supports autoarchiving old messages to personal folders, but every folder and subfolder must be set up individually, and unlike the Exchange store, these messages are not usually backed up.

The net effect is that critical information is routinely lost among the mountains of deleted messages. Companies face a tough choice: Either accept the high cost of administrator and user productivity loss when deleted information must be restored from backup tapes or re-created by the user, or accept the mail system performance and reliability degradation that occurs when the message store is allowed to grow without limits. A study by Creative Networks, Palo Alto, CA, found that restoring and re-creating deleted messages represents a significant cost to the organization, averaging $193 per user in large Exchange shops, more than the cost of all technical support and help desk combined.

In addition to the administrative and productivity costs, information lost from the mail system poses other problems. Companies are beginning to realize that email messages and attachments represent business records that must be retained and managed securely in order to comply with regulatory requirements, satisfy auditors and prevail in litigation. Increasingly, corporate legal departments are trying to impose corporate standards on email archiving and retention. In addition, email represents an increasingly large piece of the corporate "knowledge asset" that organizations now want to leverage using knowledge management tools for competitive advantage.

Out of these business needs have emerged a new class of solutions called "mail archives." With a mail archive, users can store their mail and file attachments for as long as they'd like on cost-effective media. Later, without administrator assistance, users can retrieve any message easily by browsing their original mail folders or by searching message properties and content. Users have ready access to their own information in their original Exchange folders with no manual effort for indexing or backup, and without degrading Exchange system performance or security. Archiving is done automatically on a scheduled basis, according to corporate policies that specify what should be saved and how long it should be retained. A mail archive is not the same as a backup system. Backup is designed for restoring the entire message store in the event of a disaster, not finding and retrieving a single message.

Bruce Silver Associates looked at four leading mail archiving products: three for Microsoft Exchange - Archive from Compaq, Houston; Enterprise Vault from KVS, Williamsville, NY; and Ixos-Exchange Archive from Ixos, San Mateo, CA - and for comparison, one for Lotus Domino, CommonStore from IBM, Armonk, NY. [Editor's Note: Imaging & Document Solutions separately reviewed EmailXtender - an archive solution for Exchange, Lotus and other servers - from OTG, Bethesda, MD; see "OTG EmailXtender: A Mail Archive Reborn"] While similar in their basic concept and operation, important differences exist in storage architecture, features, and suitability for record retention and knowledge management.

Product Information Table

Common Elements

All four products allow administrators to define archiving policies for users and groups. These are the rules that determine which items are selected for removal from the message store and transfer to the archive store. These rules also determine what information is left behind in the mail system.

Selection is generally based on an item's age and file size, sometimes in conjunction with the space remaining in the user's mailbox quota. In addition, all the products allow users to manually select items in their own mailboxes for archiving, or in some cases exclude from archiving.

On a periodic schedule, established by the policy, an archiving agent on the mail server combs each user's mailbox for items to be archived. Typically, the archiving agent then moves the selected items to a file server, leaving behind a small stub message in the mail system containing the message properties, such as Subject, Sender Name and Date Received, plus a link to the rest of the message. In some offerings, the file server is used as the long-term storage repository for the archived information, often in conjunction with a hierarchical storage manager and low-cost removable media. In others, the file server is merely an intermediate storage facility from which the data is transferred to a true archival store with its own index database and file system.

To the user, archived items still appear listed in mail folder views (usually with an icon looking like a CD), but the body and/or attachments have been physically removed from the mail server, saving significant storage space. Users can also find archived items by searching a content index engine - a capability vastly improved over ordinary Outlook Find or Advanced Find.

Users open an archived item from a folder view or search-results list, just like a regular message. A retrieval agent integrated with the mail client retrieves the archived portion of the message from the archive store, rebuilds the full item and displays it.

Users can reply to, forward or delete archived messages just like regular ones, although the four products differ in the details. Merely retrieving and viewing an archived item does not permanently restore it to the mail system, but users can manually select archived items and restore them.

Storage Architectures

Now let's look at the major differences among the four mail archives: storage architecture, search capabilities, and retention and knowledge management features. As mentioned earlier, the archiving agent for all four products is an NT service that extracts selected items from the mail system and writes them as files on a file server. As a departmental or midsize enterprise solution, Compaq, IBM and KVS allow the file server to act as the long-term archival store. For KVS, this is the only configuration offered. In contrast, Ixos considers the file server as only a short-term archival store.

Using a file server for archival storage is undoubtedly the simplest and least expensive implementation option, and it can be extended with hierarchical storage manager (HSM) technology to take advantage of low-cost mass storage media like CD or rewritable optical disk. The HSM manages space on fast magnetic disk by auto-migrating files used less recently to secondary storage while providing transparent, although slow, access to these objects through the file system. However, the file server approach lacks key features of an enterprise-class archival store built around a relational database management system on NT or Unix. For example, file servers can't maintain good performance with millions of items; document-level security and access control; advanced retention management; support of write-once, read-many disks optical jukeboxes; and transaction integrity.

Compaq, IBM and Ixos offer a true enterprise archive store for mail items, each leveraging a general-purpose mass storage subsystem that (with additional software) can also archive data and documents from SAP and selected other applications. The Compaq Archive features a programmable workflow that picks up archived Exchange items from the file server and routes them to specified storage devices that can be anywhere on the network. In addition to a stand-alone Exchange Archive, Ixos supports a shared SAP/Exchange configuration, with its Universal Archive with Alternative Access, or integration of archived Exchange items with revisable document content in its new eCon Portal.

IBM relies on ContentManager as its archival store for Domino. This is the same flexible enterprise repository IBM employs for document images and Web content. Ixos and IBM provide document-level security and access control; with Compaq and KVS, access control is at the mailbox or folder level.

Search Capabilities

Compaq and KVS use AltaVista for indexed searching based on standard message properties or text content. While Compaq restricts searching to mail items from the user's own mailbox, KVS allows searching for files archived from public folders, indexing attachment content, and authorizing users to search across multiple mailboxes or Exchange systems. Ixos uses a Verity engine for content searching in combination with Universal Archive database queries using any message attributes, including custom forms and properties, anticipating a much broader application of email for structured business communications and e-commerce.

IBM CommonStore only allows searching against document properties, not text content, but as with Ixos these properties are fully customizable. Also, in conjunction with IBM's Enterprise Information Portal, users can make federated queries across ContentManager and other document repositories in the enterprise.

Retention and KM

Retention and knowledge management extend the purpose of mail archiving from personal productivity to meeting strategic goals of the organization as a whole. IXOS places the biggest focus on retention management, and it is the only vendor of the four to view record retention as the key business driver of mail archiving. Ixos allows each item to be selected for archiving based on a complex query and assigned to a "logical archive" independent of the original Exchange topology. Each logical archive has its own specified retention period and storage media and removes archived items at the end of the retention period. Ixos also provides a full archiving audit trail, which is stored on write-once optical media (WORM). The Compaq Archive stores archived items on WORM and does not allow users to delete them. KVS, with its file server storage, does support retention management through integration with Exchange journaling and a retention category attribute for archived items.

Of these four vendors, IBM places the most emphasis on knowledge management, reflecting the lead Domino has built over Exchange in that area. CommonStore is supported by IBM's Enterprise Information Portal initiative, which provides federated search and retrieval across disparate information repositories in the enterprise. Ixos provides a measure of knowledge management through its eCon Portal configuration. Compaq and KVS simply leverage Microsoft's current and emerging knowledge management capabilities.

Conclusions

In general, Compaq and KVS emphasize solving the productivity problem of restoring deleted messages, while Ixos and IBM place more emphasis on retention and knowledge management. KVS provides a feature-rich, file-server-based archive with basic retention management capabilities. Its strength is its indexed search and retrieval based on file attachment content and attributes in addition to message properties.

Compaq does not archive calendars and public folders, but it offers both a file server and scalable archive store that can be used for SAP data as well. Its strengths are scalability and its position as the world's leading integrator of Exchange systems.

Ixos does not offer a file server version, but it does support a range of storage alternatives depending on whether the archive is Exchange only, extensible to SAP data or part of a knowledge management portal. Its strengths are scalability, flexible retention management and security, and its position as the world's leader in SAP archiving.

IBM supports Domino, not Exchange, in both an entry level file server version, using Tivoli Storage Manager, and a true enterprise archive store. Its strengths are its high degree of programmability, scalability from NT to mainframe, and integration with IBM/Lotus' broader content management and knowledge management initiatives.

How Mail Archiving Works

1. The Mail Archiving Agent periodically selects items from the Exchange Message Store, as determined by archiving policy, and writes them as files on an NT file server.

2. The message body and attachments are removed from the message store and replaced by a small stub that allows the items to appear in Outlook views and searches.

3. In some archives, the NT file server (with HSM) is the archival store. In enterprise archives, the file server is only a temporary cache before transfer to a true archival store, which has an integrated database, optical device support, and advanced workflow and retention management. The archival store may have interfaces to additional sources of archive content, such as SAP.

4. Archived items may be queried using a search agent extension to the Outlook client.

5. When an archived item is opened from Outlook, a retrieve agent extension fetches the content from the archival store, rebuilds the message, and calls on Outlook to display it.

Bruce Silver is President of Bruce Silver Associates, Aptos, CA, 831-685-8803, brsilver@earthlink.net. Reports available on www.brsilver.com.