• by editor , David Baker , April 11, 2005

When a new e-mail marketing trends report or click-to-purchase conversion rate chart comes out from the likes of Doubleclick, Jupiter, or eMarketer, people flock to these stats convinced that they are an exclusive tool to help improve their program, or at least some perception of their program.

Like most of you, I've spent some time and money on e-mail benchmarking. And also like most of you, I've come to realize its not always the best way to measure your success. While charts and reports are unfortunately a necessary part of this business, they may not be the best guide to help gauge the success and growth of an e-mail program.

Why? Well first, these statistics are usually aggregate by category. There is not a clear distinction between companies or brands within vertical and subsequent categories. Then, there's the accuracy of the source of list, method of permission, type of communication, intent, and the fact that the overall sophistication of e-mail marketing by sector is so scattered. It's very rare that all of these factors could line up to provide an accurate reflection of your program let alone a benchmark on which to measure your success.

Next, while statistics might be considered impersonal, they are in fact very personal to you and what it means to your business. Most of the time, there is little context as to what other's benchmarks mean, how they were derived, and if they are unbiased. I often wondered, but could never prove, if the analysts used biased data for the benefit of the report or industry in general, which could be argued with all the differences in how deliverability is being reported today.

So, now that we've put the industry charts aside, how will you survive? When factoring the statistical validity of your program or the value of benchmarking, here are a few guidelines to keep in mind:

Isolate benchmarking to lists, divisions, brands, or product lines and be careful about cross comparisons.

Take the tenure of your lists into consideration when factoring or comparing benchmarks as conditioning can have a big effect.

Consider the type of relationship your customers can have through e-mail and their propensity to respond by type of e-mail (notification, promotion, retention, affiliate etc.)

Gain a clear understanding of what the stats mean to your business and only worry about those that bring you 'learnings' that drive business value.

Consider the technology you are using and the potential limitations in the disparate systems, including that of third parties or ESPs. Some can be more detailed in reporting than others and they often report differently.

Take into account cyclical market trends and how they can impact the program at a more granular audience level.

While benchmarking will always exist and be seen as interesting and useful in this business, your success is not dependent on a third-party method of reporting statistics. Your success is dependent upon your ability to improve your own benchmarks and derive meaning from those learnings.