SMX East 2011: Best Practices for Paid Search Testing

Sid Shah will start us off. Public service announcement: Sex ed is wrong. Babies are delivered by storks. Let’s prove this with data:

The cross section analysis of population of stork and babies shows hat they follow. Is he mad? In the biz world, biz make the wrong conclusions based on inaccurate analysis all the time. He’s going to talk about measuring, testing and analyzing in a way that will

Goals:

• Formal testing methodology.
• Mathematical intuition behind testing and analysis.
• Explain how to do experiments without spending time and money (just math).

Analyzing results is most often done wrong.

Generate Hypothesis

A good hypothesis is about specificity: adding “official site” to ad copy improves ROI.

Better: adding “official site” to ad copy of branded terms in search campaign improves ROI.

Best: adding “official site” to ad copy of branded terms in search campaign of my auto business improves ROI.

Create Experiment

Hypothesis, performance metrics, external factors, testing parameters, constraints like HiPPOs.

The highest paid person’s opinion can be a constraint you need to work around. It’s best to have as much data as possible and hope their reasonable.

Analyze Results

It’s nice to hope your results will be nice and clean and clear. But real data is more difficult to read, and it can be hard to say.

He’s showing graphs that show that the average, or mean, isn’t always the data that matters. The distribution may be very revealing for success or failure as well.

Hypothesis testing: with the collected data, can I tel with statistical confidence if the pre and post results are from statistically different populations? P value shows the probability that the data is right.

Models and Simulations

They’re quick, cheap, forces a review of assumptions and answers “what if” quesitons almost instantly.

One can simulate the performance of a search campaign using sophisticated mathematics. Simulations can be used as a planning tool in any area where it’s expensive to run an experiment.

Final Tips

• Be data-driven
• Compute with care
• Simulate experiments
• Constantly re-assess your assumptions
• Experiment as often as possible

Shannon Anderson is next. Keep in mind:

• Build best practices, then rebuild them.
• Don’t test for the sake of testing – have a plan and purpose.
• Remember, it’s marketing! Speak to the end user.

Her testing process:

• Purpose of test, testing variables, timing, launch, analysis, use the data, repeat
• Establish a budget for the test up front and be okay with the fact you may not get any revenue out of the test and may lose the money with no results. You have to let the test run all the way through.

Example: ABC Travel

Purpose of test: determine best practices to launch during peak booking season
Testing variables: purchase cycle, call to action, headline, targeting, day parting

Headline testing:

• Using “official site” in the headline
• Using registered trademark symbol

They found a true improvement with the trademark symbol.

Call to action testing:

• $500 off urgency with offer end date vs. “Limited time!”

Saw worse results with end date in the test. It may have been too aggressive to have the end date.

Targeting testing:

• Used “escape the cold” language in copy of tests in Chicago

Day parting:

• Conversions happened more in evenings

Click Assist Report helped understand the data, especially keywords that aren’t meant to convert, but help to assist conversions further down the line.

Chris Goward will talk about how to get max conversions once traffic is on your site. These methods work for all industries in all countries.

6 conversion factors that influence LIFT (landing page influence function for tests):

• Value proposition
• Relevance
• Clarity
• Anxiety
• Distraction
• Urgency

When hypothesizing for tests, turn problems into hypotheses. Make sure you’re addressing problems with your tests, not “what color button?”

Great hypothesis:

1. Specific
2.  Testable
3. Solves conversion problems
4. Builds on marketing insights

Funnel Experiment Map: a document that outlines the analysis, hypotheses, variations, the test plan before going to the design department.

Test plan includes:

• Test structure: A/B/n, MVT, alternative flow
• Traffic sources and volume estimates
• Calculation of number of variations
• Hypothesis
• Wire frames

Graphic design and copy: Make the wire frame look like the control and reflect the strategy of the test.

Final tips:

• Do not use the before and after method. Use a controlled testing method.
• Do not start with segmentation, targeting or multivariate. Start with A/B testing.
• Do use a controlled testing tool with statistical significance and 95% confidence.

Virginia Nussey is the director of content marketing at MobileMonkey. Prior to joining this startup in 2018, Virginia was the operations and content manager at Bruce Clay Inc., having joined the company in 2008 as a writer and blogger.

See Virginia's author page for links to connect on social media.