Benchmark is an after-the-fact snapshot metric whose utility is in the comparison with other snapshots. A downstream metric might have some utility for benchmarking a process against other instances of itself, but in order to be useful to another process in another organization, a thorough normalization must take place, making sure the process and general context is similar between the two cases.

Here's where we run into difficulty. Benchmarking as commonly applied is the process of comparing many companies, identifying the best in class, and then presumably comparing your organization with the purpose of improvement.

Benchmarking assumes that there is a well understood set of characteristics that are being benchmarked and that there is a meaningful sample size.

For benchmarking, Computerworld offers a very straightforward definition: "Benchmarking is a measurement tool used to gauge a company's operating performance against that of competitors to identify best practices and make improvements. Examples within corporate IT include measuring the costs of supporting a data center or network infrastructure." But then they address e-commerce benchmarking as a separate entity: "E-commerce benchmarking is used to help companies compare the costs and performance of their online operations with those of other companies."

In the Computerworld article defining e-commerce benchmarking, Mark Czarnecki, president of The Benchmarking Network, Inc., says, "All the concepts and principles of e-commerce are still not set," making benchmarks very difficult:

Normally, there's a set of "anchors" that companies can rely on when they do benchmarking—a defined measure of performance, such as transactions per second or monthly sales, Czarnecki says. "Defining it means there's some kind of stable business process. [With e-commerce], there's not a stable business process," he adds. The top priority for electronic businesses is turning a profit, says Jim Sample, a consultant at Reston, Va.-based benchmarking service provider Compass America Inc. "The drive toward profitability then trickles back to having to put in some traditional measurement system," he says.

Benchmarking Specifics

Comparing two advertising venues is extremely difficult due to the vagaries of the language used by different venues. Kris Carpenter from Excite wishes we could come up with some hard-and-fast, non-vendor-oriented, truly consortium-level definitions: "Lycos Terra has a very effective way to fold their U.S. audience into their international audience. So every company reports it differently. For some it's distinct, for some it's rolled-up, and so you never have a good means of seeing apples to apples. That's one of the advantages that companies have in putting the best light on everything, but from a research perspective it does make it very difficult to know exactly what should be communicated." In Sweden, Mats Rene manages Ericsson.com, the international telecommunications giant. He wants to know how his site and his processes for managing it measure up: "Sometimes you feel a bit alone, especially when you get into the area of content management, which is extremely complex. I don't really think that anyone can understand how complex it is until you are in the middle of it. I mean it's so much work and it's expensive too. To get some benchmarks in that perspective would be really interesting."

The picture at Kodak is the same. Terry Lund wants to know how some features on his site compare to competitors of all stripes: "There were some 137 startup dot coms trying to do this [new photographic feature] as well. And the perception from senior management was they were going to eat our lunch. So we started reporting using Nielsen//NetRatings data because it's a set of data that covers everybody and is independently derived. In other words, we can't really use the data we collect with our Accrue software because we don't have any Accrue data for anybody else."

Terry finds it very frustrating, and he's not alone. "There's not a particularly strong correlation between what Nielsen measures and what we measure ourselves with Accrue," he says. "The absolute numbers are off by a big factor." Apples to apples indeed.

In the advertising marketplace, there are the Web sites that sell ad space, there are the brokers that serve the ads, there are the Webmasters who receive the clickthroughs, and there are the NetRatings types that report on who's getting the most traffic. Talk to anybody running a Web site of any size and you hear the same lament. None of those four numbers match in any way. There's not even a formula to relate them to each other. They're not normally within 20 percent, or a multiple of 3. Those numbers are consistently all over the map.

Why the frustration? Everybody's got a different methodology. Nobody's got a standard definition. What we're measuring is still misunderstood. Everybody is delivering numbers that will allow them to bill as much as possible, instead of looking for accuracy or coming to some approved agreement.