• by editor , David Baker , Roshen Mathew , April 24, 2006

Sometimes we get so drilled down into our work that we don't create fresh ideas and thoughts. So I asked Roshen Mathew, one of the senior e-mail strategists at Agency.com, to author a column on e-mail testing. He has an interesting view--he takes things back to the basics, providing a great refresher about e-mail and the value of testing. Here's Roshen:

Test and learn, test and learn. As marketers, we've all heard or given that advice over and over again. However, lately these sensible words have lost their pull with our clients, and in our own organizations. Why? Quite simply, it's the cost/benefit assessment that concerns most marketing teams. They ask themselves: "What truly actionable insights will we reap?" "How much time will these insights take to put in place?" And finally, "it's going to cost how much to conduct this test?!"

The reality is that all channels are not equal when it comes to test and learns. Testing calls to action, messaging, and creative approaches in online media, and offline direct mail can be an expensive and time-consuming venture.

But today's e-mail channel is thoroughly trackable and can be tied in with your user database and market map segments, making it a quick and effective channel for testing.

When a consumer opens an e-mail, that action--the "open"--is logged, and he or she is presumed to be focused on the contents of the e-mail. You can now consider this consumer a test subject. You can determine the effectiveness of your test elements by examining your click-to-open rates. Count the unique e-mail click interactions associated with your test elements or primary call to action, and divide by the number of unique opens. You then know, for instance, that of the test subjects who definitely viewed "Test e-mail A," 35 percent of them clicked through, as opposed to "Test e-mail B," with 48 percent. Simple, quick, cheap and effective.

Compare this e-mail scenario to offline direct mail. For a start, you don't know whether a specific consumer has opened the mailer. The only way to find out is to conduct a telephone survey or wait an extended period of time for redemption data to come in, then analyze it.

With online media the results get back to you much more quickly, but the costs can skyrocket, depending on what sites you want to test. Also, your test elements aren't necessarily the only significant piece of content vying for the consumers' attention. You have to compete with other ads, as well as the very content of the Web site--the real reason the consumer came to that specific page. For example, if you want to test messaging using a simple banner ad, but portions of your test impressions have to compete against another company's animated advertisement, are you really setting up a fair test? After all, you have no control of what Web sites run alongside your ads (without paying a little extra, of course).

Whether you're testing a message, a call to action, hero shot versus product-only shot and so on, you can ascertain these learnings through e-mail, then roll out the winning elements into all of your media channels: landing pages, banner ad development, direct mail components, etc.

It's important to note that there are other forms of testing that might make more sense with some channels. For example, brand awareness studies are better executed with online media campaigns in comparison to e-mail. But even with brand awareness studies, costs continue to grow in the online media channel because today most Web sites are charging for the control group buy--something relatively new in the space.

Now there's no doubt that if you are committed to any channel, you should test and gather what information you can within that specific channel. But think twice about committing a heavy spend in a channel with untested components, when e-mail is readily available for basic learnings. E-mail testing can make your other channel spends more effective.