Even with hardware getting cheaper and analytic software becoming more prevalent, there comes a point of diminishing returns when your returns are being measured. If your Web site is generating 2 gigabytes of server and transaction logs every day and you need machines twice the size of the servers to keep up with the analysis of that data, at some point you have to throw up your hands and say, "Either give me another number cruncher or stop asking for new reports!"

Motorola ran a spreadsheet last year that resulted in a cost of $120 per person to analyze sufficient detail to satisfy upper management. That might have been fine if they were looking at prospects considering the purchase of satellite and broadcast network systems. Unfortunately, this was the mobile phone division. "It was obvious that the economics were not aligned with our business," one spokesperson told me.

The problem is one of desire versus ability. If I promised to tell you what percentage of products get put in the shopping cart but never purchased and which ones they are, you'd be thrilled-until I told you it was going to cost a quarter of a million dollars. Each month. Sampling is the solution of choice for a great many large Web sites. Take a picture every now and then rather than making a movie all day long. Track a handful of people as they wander through the site, rather than track each and every individual. This is an area of keen frustration for marketers and analysts alike. The promise of the Web was to get to that one-to-one relationship-to be able to keep tabs on people on an individual basis.

So how do you make the determination of which data is worth saving and which calculations are worth running? Kris Carpenter, vice president of Excite Products and Services at Excite, got creative. When dealing with a Web site that attracts so many people every day (15 million per month, according to Nielsen//NetRatings), creativity is a must.

"Well, what we've actually done," Kris explained, "is sort of twofold. We have the core data that gets archived and then often stored, not necessarily in a live database but in various archived formats that we can go back in and do time-based analysis against, if it makes sense. For specific products, where we have additional data products, we just need to understand some fundamental trends.

"For example, with the search engine, you don't want to know every query that's ever been performed, but to understand the top thousand queries. How is that changing in terms of frequency? What types of categories are consumers looking for most frequently? So that you can do the best job possible of delivering the right body of information in that database. "So, that kind of information we end up archiving on a by-product basis because we also correlate it back to things like advertising inventory or consumer demand for specific subject."

I picture what used to be called the smoke-filled room, with executive types sitting around, scratching their heads, saying, "You know, folks, it would be really interesting to find out if ..." and dream up some correlation between subject matter, frequency, recency, and so on. At the other end of the table, the IT representative says, "Yeah, but that would cost us $2 million to run the data, and what's the value of that kind of a calculation?"

Excite runs through an ROI business requirement analysis each time a new Web initiative is proposed. Kris describes it this way: "Every quarter there's a review of specific research we create, partnering with a third party like a Harris or a Jupiter or a Nielsen. We have specific budget allocated for those things. Every project is then evaluated based on the ROI. The same thing occurs for internal reporting. We make a case that says we need to have this type of data warehousing in place, we need to have everybody in the business be able to access and look at trends, and understand them."

One of the things that has come out of that approach to investment analysis is something Excite calls "cow pathing," their term for clickstream analysis.

If you find yourself dealing with a ragtag group of big-band leaders who are marching to the beats of conflicting goals, do not despair. There is a tune you can play that will be music to their ears no matter what their personal predilections. Even if they cannot fathom the Web and it's potential enlightenment, they will understand the words "Economies of Scale."