Megabytes and Gigabytes and Terabytes, Oh My
by Chris McTiernan

Every Net market or B2B marketing initiative is unique in its methods, goals, and aims, yet they all share one fundamental characteristic: They produce data, data about every transaction, every event, and every user with whom they come into contact. Most of this data will be stored in logs that no one will ever look at. Most of it is never used.

I have, over that past few years, become very interested in this data, in the reasons we collect it, and in what we can possibly hope to do with it all. I keep asking myself, "What good are terabytes of data when all they do is fuel the memory and storage markets?"

The collective "we" (you and I and everyone engaged in e-business) are collecting quite a lot of data and we don't seem to be doing anything meaningful with most of what's produced. That's not news, though. Ever since we figured out how to write, we have developed a certain affinity for recording things. And, once ones and zeros came along, things started to get out of hand.

It's not that the data we are gathering is not valuable. To the contrary, the Internet is by far the richest source of data about customers, suppliers, and partners that we could possibly conceive of having. It offers almost limitless value. One of the key challenges is the extreme difficulty of determining the relevancy of information from the terabytes of data we collect and store. It has also been difficult to determine meaningful correlations among the data itself.

The old analogy of drinking from a fire hose could not be more applicable. We want water but not quite that much and not with that much force. What we want is timely and relevant information so we can make better-informed decisions.

Making Sense of It All

The good news is that smart people have been trying to figure out how to categorize and utilize information for a long, long time. The Library of Congress and the Dewey Decimal System do not have aims fundamentally different from those of data-warehousing solutions from Oracle or analytical tools and applications from Cognos.

More good news: There are a series of converging forces that are making the intelligent use of this data a reality.

1. We have the production of data itself. We are coming to understand the value of data and are beginning to create systems that will help capture valuable and relevant information. We have web data and customer service data and financial data and media data and operational data. It can be overwhelming to consider, yes, but we at least are developing discipline to amass this data so it's usable.
   
   
2. We have tools powerful enough to assist us in collecting, managing, and understanding data. These tools can assist us in determining the correlations among disparate data sets such as web sales and retail sales, or web sales and cost of customer service. While different in architecture and application, tools to date from such companies as Personify or NetGenesis have impressed me the most.
   
   
3. We are beginning to understand how to empower the enterprise to make more intelligent decisions. This will make our marketing efforts more successful, our markets more attractive and efficient, and the service we deliver more relevant and informed.

Obviously, these steps are not the end of the game. Using data to create competitive advantages will be on an ongoing challenge for all of us. It is a necessary undertaking, though. It is vital to the success of each initiative planned and, arguably, the foundation of all future success the more we come to rely on e-business.

Chris began his Internet career at BigBook, the leading online Yellow Pages at the time, managing advertising services. He later worked with CNET, helping to build the online publisher's advertising services department, serving on the development team for Snap! (now NBCi), and assisting in implementing the company's first e-commerce partnerships.

(c) 2000 ClickZ Corporation.  All Rights Reserved.