Web Optimization for DR: The Test Plan

by Oded Noy on Feb 17, 2015 3:00:00 AM Digital Marketing

Web_Opto_Image1-223060-editedThis post is the fifth and final in a series highlighting the concepts, best practices, and techniques for DR companies to optimize their Web conversion path.

In the previous posts, I discussed the fundamentals of Web optimization for DR. I covered the basics, compared the Web testing strategies with telesales scripts. Next, I discussed optional visitor flows from basic to more advanced, and then mobile. This month, it’s time to put the pieces of the puzzle together.

In order to optimize, a test plan is required. In previous posts, I addressed the portions of each page and how they are tested separately. A test plan is intended to take into account the various marketing needs and how they fit into tests that can provide statistically significant reports that move the needle.

The end result of a Web optimization strategy is better performance. This means actionable results that yield top-line and bottom-line value to the business. To do that, it is critical that the plan has enough variability, but be balanced with how much traffic is required to get timely statistically significant results.

Here are some considerations for building a Web optimization test plan for DR:

1. Schedule of major media outlets, which may include:
  •    When is a big TV or radio spend likely to start
  •    There is a difference between long-form commercial and a drive-to-Web short-form TV campaign
2. Seasonality of the product itself
  •  Tests are most effective when comparing apples-to-apples
3. Variability of the audience segments for the media campaigns
  •  As I’ve covered several times in this series, a display prospecting campaign visitor is not the same as a branded search visitor; and therefore, lessons from one audience do not always apply to all
  •  Affiliate traffic, for example, can spike greatly from day to day and—depending on the source—have very different results. When preparing a plan, care should be taken for both how such media is trafficked and how the test reports are read.
4. Change in offer
  • When launching a BOGO offer (Buy-one-Get-One free) or a significant discount offer, all prior test results need to be reset, as prior learnings may not apply. This new offer will likely drive different user behavior; a new offer versus a previous one is not apples to apples testing.
5. Change in the marketing of the product
  • Example: introduction of a new celebrity spokesperson
6. New competition in the marketplace; examples include: knockoffs and similar products from other brands
  • New e-commerce, including: outlets, affiliate SEO and SEO buyers, etc. If a retail partner is doing a big promotion that includes your products this will affect the test.
7. Change in market conditions
  • Consumer confidence, status of employment, and general spending moods. Do not assume that if it worked last year, it will work this year.
  • Changes in the tax code.
8. Last but not least…device type
As indicated in last month’s mobile blog post, do not rely on responsive design to determine the test between desktop and mobile devices. As you will see below, it is typical that cell-phone tests will have a smaller amount of variants than desktop—but that is dependent on where your traffic is generated and where the performance improvement is more likely.

An example of a plan (a.k.a. seven-layer cake) might look like this:

As a reminder:
  • LP = landing page test
  • SAS = Select-A-System (think McDonald's Happy Meal) Note that in this flow, the SAS options simultaneously exist for different funnels.

Notice that the seven-layer cake provides the opportunity to test not only the creative treatment of the landing pages, but also the effectiveness of offer configuration, upsells, etc.

To break it down further:

1. Always—and I mean always—have a control for the test plan



2. Have several alternate Landing Page treatments (detailed earlier in this series)



3. Select alternate paths, deeper in the funnel:



And when put all together...here is a possible test plan:


At this stage, here are a few things to consider:
  • As a rule of thumb for statistical significance – 100 orders (or leads) is when a test is likely to have statistical confidence
  • Never treat all traffic the same – in this case, it may be that only some campaigns will drive customers deeper in the funnel
  • Testing for smartphones can be a simpler plan if needed:


In order to complete such a plan, you’ll need creative assets.

Here is an example for test-plan implementation based on the seven-layer cake over time:


If, at this point of the article, you think to yourself, “Is all the work worth it?” consider the following: Typical improvement of 15 percent+ in conversion rate plus increased AOV is within reach! And these tests can also unlock 30-50 percent increases, depending on what you are starting with...

If that possibility is worthwhile for your business, then this is the roadmap to get there. The amount of testing is typically in proportion to the amount to traffic flowing to the website tested and the amount of revenue that site generates.

This blog post is intended to frame the discussion about the entire optimization process, focused on the design of the test plan.

Lastly, here is the distilled (tried and true) Web Optimization Must Have List: (Note: I hope that after reading this series these tips are self-explanatory)
  1. Have a Control for every test.
  2. All campaigns and offers are different; split the traffic and the tests as needed.
  3. Run targeted offers for targeted campaigns to increase efficiency.
  4. Test, learn, test, learn, test…which means discipline to plan tests and the patience to wait for statistically significant results.

That’s it for this series. I hope that you find these concepts helpful in a practical way. For simpler reference, ERA will be releasing an eBook based on this series. As always, please don't be shy and leave a comment so we can start a dialog.

Image courtesy of David Castillo Dominici/FreeDigitalPhotos.net

Oded Noy is cofounder and CTO TargetClose.com.

Oded Noy's blog
Get a2bFilfillment's FREE Ultimate Guide to Fulfillment e-Book
Subscribe for tips on how to grow your direct response marketing business!
Subscribe Now!

Follow Us

New Call-to-action

Editorial Disclaimer

The statements, opinions, and advertisements expressed on the ERA Blog and other online entities owned by the Electronic Retailing Association are those of individual authors and companies and do not necessarily reflect the views of the Electronic Retailing Association.