Lee Willis

How to Split Test WordPress Themes

| 119 Comments

I recently re-worked a site for a client. He was moving over from Joomla to WordPress, so I started from scratch with his theme design. They liked the new theme I came up with – however I’d put the menu and other navigation (Links to recent articles, etc) in a right hand sidebar, and they weren’t too sure about that.

“Doesn’t everyone have the menu on the left?”

Now I’m no usability expert, and while I figured they were probably right I thought we’d run a little experiment. I trawled around a bit, and I found a nice little post by David Dellanave. While I was OK implementing David’s solution I though it was probably a little complicated for some of my other clients, especially when it came to running 3 Google analytics accounts – so I did some digging.

The result is the SES Split Testing plugin.

To use the plugin you’ll need to be using Google Analytics tracking on your WordPress blog already (If you’re not, then I’d recommend the “Google Analytics for WordPress” plugin), then :

  • Create two (or more) versions of your theme
  • Select them in the plugin settings
  • Define custom segments in Google Analytics
  • Sit back and watch the results roll in

Create theme variations

In most cases, you’ll want to test two versions of 1 theme with some slight changes. Let’s assume your theme is in the folder wp-content/themes/mytheme. Take a copy of your theme and put it in fresh folder, e.g. wp-content/themes/mytheme_2. Open up the style.css file in the second folder, and change the Theme Name, e.g.

/*
Theme Name: MyTheme
*/

to

/*
Theme Name: MyTheme (v2)
*/

Set up plugin settings

Wordpress Theme Split TestingNow you can drop the plugin into your plugin directory, activate it in the WordPress backend, and then go to the settings page (Settings->SES Theme Split Test). The plugin will list all of your available themes, and you can choose which ones you want to be part of the test.

Once you’ve saved the settings, the split testing will kick in. Visitors will be randomly assigned one of the chosen themes, and they’ll receive that theme any time they visit in the next 30 days. Not only that, but their visit will be tagged with a special variable that you can pick up on in Google Analytics.

Set up Custom Analytics Segments

Google Analytics lets you define custom segments (Ways of grouping customers). In this example we want to group customers according to which theme they were shown and then we can measure our chosen metric (Purchase / sign-up / page views – whatever) for the two groups and compare them. To get started, you probably want to wait 24 hours so that Google starts to receive the tagging data.

Setting up custom segmentsLog into your analytics account, and click on the “All Visits” dropdown (Top-right, just above the date selector), and then choose “Create a new advanced segment”. Then, you want to expand the “Visitors” dropdown under “Dimensions”

Custom Segment Build

Scroll down until you find “User Defined Value” and drag it onto the right hand box that says “dimension or metric”. If Analytics has got your tagging data (See – I told you to leave it 24 hours!) then you’ll get to choose from the values being passed, e.g.

Analytics Custom Segments

Create a segment for each of your themes, and give them a meaningful name, then you’ll be able to select them as custom segments in your reports.

Segmented Analytics Reports

Now it’s just time to wait, see what happens, then decide. Once you’ve got a winner then just set that as your blog’s standard theme in the normal way, and either deactivate the plugin, or unselect the themes in the settings and users will see the blog default.

119 Comments

  1. Do you know if the same issues exist with Quick Cache?

  2. Thanks for the plugin!
    Does it swap the themes for search engines too?

    • Search engines will get the same treatments as normal users. Whether they’ll retain a particular theme for a particular set of “visits” is up to the crawler in question and whether it acknowledges and re-presents the relevant cookies.

      • Hi Lee,
        your plugin looks nice, but for SEO I highly recommend to filter search engines to always get the default template (think this is the intention Anton asks about this, too.)

        Depending on how big your changes between the themes are, the sourcecode and content position could differ strongly. This could have an impact on your search engine ranking cause search eingines reappraise your content relevance when it is changed (even if it is just the content position).

        If you run a test with a few different themes varieties and the crawler (as it will not be the same every crawl) always sees another mixed sourcecode, this isn’t a good idea and can induce problems.

        As the SE bot isn’t relevant for the a/b testing, he should always see the default template to avoid SEO problems.

        Can you implement this?

        Best Regards

        • I don’t have any idea how to reliably identify SE bots, any ideas?

          • Did you recieve my last comment? Think it’s gone into the spam filter, cause it has three links as reference.

            Not shure if there was another problem, cause there is no message for waiting moderation.

          • Hi Lee,

            I think there went something wrong with my last comment so I post it again.

            I am no programmer but I think you can filter SE bots by user agent (at least the SE bots from the main SE like Google, Yahoo, Bing):
            support.google. com/webmasters/bin/answer.py?hl=en&answer=1061943
            useragentstring. com/pages/useragentstring.php

            Perhaps have a look in the sorucecode of phpabtest, which claims to filter SE Bots:
            phpabtest. com

            See also
            “It also filters out all bots and shows them your control page so that it doesn’t affect your SEO efforts.”
            visualwebsiteoptimizer. com/split-testing-blog/server-side-ab-testing-using-phpab-what-is-it-and-why-should-you-care/

            Hope this helps. What do you think?

        • The possibility of split testing having a negative impact on search engine ranking made me a think twice about implementing it using the technique in this article. But after some research I came across an article that indicates that Google is fine with split testing, and that you should not treat the Googlebot crawler any differently than your normal visitors, i.e. it’s perfectly fine to send the crawler to different versions of your site each time it crawls your site.

          Here’s the article I mentioned: http://blog.convert.com/matt-cutts-split-testing-is-perfectly-fine-by-google.html

  3. Hi Lee :)

    I have also created a plugin that allows you to A/B and split test your WordPress site’s content (posts, pages, and widgets), stylesheets, javascripts, and themes.

    If you want to check it out and give me your comments?
    http://lassebunk.dk/plugins/abtest/

    Thanks in advance,
    /Lasse

  4. hi lee…

    It’s been several days now… still i don’t see that tagging data on the analytics segment..

    In your screenshot you got there mytheme and mytheme_2.
    Is that the name of the folder of the theme? Or is that what you place on the stylesheet code?

    If it is the folder name of the theme… I have space on mine… ex: “My Theme Ver2″ and “My Theme Ver1″.. is this okay?

  5. Hi,

    I have got the tagging data from Analytics and so it still doesn’t recognize it yet. 2-3 days ago it was installed.

    Can you help Lee?

    Thanks,
    Amro
    http://www.speedmatters.nl is the website in question

  6. What mechanism does this use to determine which template is assigned to which user? I’m trying to implement this on a test site and I want a way to ensure that it is indeed showing both but every time I visit the test site, clear my cookies/cache/etc, I am still shown the same theme.

  7. Hi Lee,

    I dont use any plugin to integrate Google Analytics.

    Would you be kind enough to provide instructions of exactly what I need to do to get it working with the latest analytics code?

    I can manually edit/place the analytics code as required

  8. I was looking for an A/b splittest plugin. Will give this one a try on one of my websites.

    Thanks for the plugin and this great tutorial.

  9. Does this use the standard WP template engine? I’m going to be split testing Roots child themes, and didn’t know if would support that.

  10. I am having the same problem, as far as I can tell the plugin is not functioning at all. No extra tracking info shows up anywhere that I can find and only one theme ever shows. ?wp_splittest_force=themefolder does nothing. In the settings menu several 4 themes are selected even when I only select 2 and save.

  11. Still not working for me. I only get the first theme I select, any others never get shown.

  12. I’m having a Google Analytics issue (and the instructions here are for the old version of Google Analytics, making the setup a bit tricky).

    But when I check my report under Audience –> Custom and check my stats they seem low. In the past month it is showing both of my themes in my test have 133 visits to the site out of 1,888. Do you have any advice as to why this might be? Shouldn’t this be 100%?

    • Hi Patrick, not off the top of my head – no. I’d recommend testing with Chrome’s ingognito mode a few times and check you get different themes, and google analytics code that looks correct.

      Unfortunately I can’t provide support for my free plugins unless it’s on a paid for basis. If it’s important to you, feel free to drop me a note on the contact page and we’ll see what we can work out.

Leave a Reply

Required fields are marked *.