How to Split Test WordPress Themes
17 Oct
I recently re-worked a site for a client. He was moving over from Joomla to WordPress, so I started from scratch with his theme design. They liked the new theme I came up with – however I’d put the menu and other navigation (Links to recent articles, etc) in a right hand sidebar, and they weren’t too sure about that.
“Doesn’t everyone have the menu on the left?”
Now I’m no usability expert, and while I figured they were probably right I thought we’d run a little experiment. I trawled around a bit, and I found a nice little post by David Dellanave. While I was OK implementing David’s solution I though it was probably a little complicated for some of my other clients, especially when it came to running 3 Google analytics accounts – so I did some digging.
The result is the SES Split Testing plugin.
To use the plugin you’ll need to be using Google Analytics tracking on your WordPress blog already (If you’re not, then I’d recommend the “Google Analytics for WordPress” plugin), then :
- Create two (or more) versions of your theme
- Select them in the plugin settings
- Define custom segments in Google Analytics
- Sit back and watch the results roll in
Create theme variations
In most cases, you’ll want to test two versions of 1 theme with some slight changes. Let’s assume your theme is in the folder wp-content/themes/mytheme. Take a copy of your theme and put it in fresh folder, e.g. wp-content/themes/mytheme_2. Open up the style.css file in the second folder, and change the Theme Name, e.g.
/* Theme Name: MyTheme */
to
/* Theme Name: MyTheme (v2) */
Set up plugin settings
Now you can drop the plugin into your plugin directory, activate it in the WordPress backend, and then go to the settings page (Settings->SES Theme Split Test). The plugin will list all of your available themes, and you can choose which ones you want to be part of the test.
Once you’ve saved the settings, the split testing will kick in. Visitors will be randomly assigned one of the chosen themes, and they’ll receive that theme any time they visit in the next 30 days. Not only that, but their visit will be tagged with a special variable that you can pick up on in Google Analytics.
Set up Custom Analytics Segments
Google Analytics lets you define custom segments (Ways of grouping customers). In this example we want to group customers according to which theme they were shown and then we can measure our chosen metric (Purchase / sign-up / page views – whatever) for the two groups and compare them. To get started, you probably want to wait 24 hours so that Google starts to receive the tagging data.
Log into your analytics account, and click on the “All Visits” dropdown (Top-right, just above the date selector), and then choose “Create a new advanced segment”. Then, you want to expand the “Visitors” dropdown under “Dimensions”
Scroll down until you find “User Defined Value” and drag it onto the right hand box that says “dimension or metric”. If Analytics has got your tagging data (See – I told you to leave it 24 hours!) then you’ll get to choose from the values being passed, e.g.
Create a segment for each of your themes, and give them a meaningful name, then you’ll be able to select them as custom segments in your reports.
Now it’s just time to wait, see what happens, then decide. Once you’ve got a winner then just set that as your blog’s standard theme in the normal way, and either deactivate the plugin, or unselect the themes in the settings and users will see the blog default.
It’s so funny I stumbled across this as this has been on my mind alot recently. I think people were used to Left Hand Navigation but Right Hand Navigation has become more mainstream with the explosion of blogs . . . it will be interesting to see the results. I think something to take into account is the site’s user demographic. For me I have an older customer base so I still with left hand nav. Another thing to tak into account is the site a full on blog or is he using WordPress as a CMS – for CMS I’d say Left is best.
Nice Post – I just blogged about it.
Anyways, whats your opinion now ? Menu on the right or menu on the left ?
Thanks – glad you like it.
All evidence suggests that having the menu on the left makes people “engage” more with your site (Higher pages per visit). Not measured goal conversion – but I’d love to hear anyone else’s findings!
Hi Lee,
I plan to use your plugin to do some A/B testing in my own site, this is what I’ve been looking for for a long time now.
One question: does it play well with caching plugins such as WP-Supercache? Or do we need to disable it?
Hi Luciano,
I don’t *know* the answer, but I’m 99% sure that you’ll need to turn WP-SuperCache off to run a theme split test – otherwise everyone will see the same HTML (I think). I’ve had a quick read up on WP-Supercache and there doesn’t appear to be a way of cacheing multiple different versions of a page, and serving the right one based on e.g. a Cookie value.
So for now – I think you’ll need to turn it off.
Thanks Lee! I’ll let you know how the testing goes!
Hi Lee,
Im excited to check out your plugin but it does not appear to be compatible with the new version Google Analyticator which is the most popular one right now .. What do you think ?
Also I tested it out briefly on three browsers and got version A three times .. Maybe as an improvment for a future release there might be a way to force version B ir you are an admin..
That said I am very excited about testing this out because if you have a lot of natural SEO traffic to play with then its so much more effective to test themes versus individual pages
thanks!
Hey there – I think you’re right. Google Analyticator looks to have moved over to the new asynchronous GA code already (Unlike Yoast’s plugin which I normally use).
I’ll take a look at what’s required to make it work with this – I think the complicated bit will be working out which style is in use.
Hi Lee, So I installed the yoast plugin .and looks like in the near future he will be moving to the other syntax — async or rather will give people the option to choose either ( that could be complicated phew ) anyways .. I manually pasted the analytics code in my footer .. I will update you about my findings
Are you planning to create a plugin that split tests changes in the copy and not just between themes? Would love this!
Hi – no I don’t have any plans to do that at this point. Maybe check out these articles / plugins:
http://wordpress.org/extend/plugins/google-website-optimizer-for-wordpress/
or
http://carsonified.com/blog/business/how-to-do-ab-testing-in-wordpress/
Hope that helps.
Greetings Lee, and I suppose whoever else is interested … not sure if you’ve figured it out on your end, but I just added an update to the ‘output_themesetvar’ function in the SES Split Testing Plugin to choose between the current or the new asynchronous tracking functionality depending on which one exists on your website.
So it should work with both Google Analytics tracking code types.
====
function output_themesetvar() {
if ($this->splittest != “”) {
echo ‘if (pageTracker){pageTracker._setVar(“‘;
echo htmlentities($this->splittest);
echo ‘”);} else if (_gaq){_gaq.push(["_setVar", "';
echo htmlentities($this->splittest);
echo '"]);}’;
}
}
=====
Haven’t tested it though. Will see how it works over the next couple of days since I just installed Google Analyticator.
I’ve had a couple of emails with Moyo and he’s now suggesting:
if ($this->splittest != “”) {
echo ‘<script type=”text/javascript”>try {pageTracker._setVar(“‘;
echo htmlentities($this->splittest);
echo ‘”);} catch(err) {_gaq.push(["_setVar", "';
echo htmlentities($this->splittest);
echo '"]);}</script>’;
}
We think this will work. If anyone can confirm then I’ll get it pushed into the plugin proper …
The syntax looks right to me ..
The force pageview syntax is like this
_gaq.push(['_trackPageview', URL]);
So I bet it will work
Yeah – we’re pretty sure it’s working. I’m going to push it out as an update when I get SVN access (This evening probably).
I’ve just pushed out 0.3 which should support both forms of tracking.
It’ll be on wordpress.org within 15 minutes or so – thanks for the patch.
Great to hear — and no problem. It’s a solid plugin, glad I could help.
This is a FANTASTIC plugin. Thank you so much.
The way I use it to track improvement is by using an ad server (openx, self hosted, free) and having my own “ads” that are simply call to action text after articles for people to move down the funnel. I simply create identical ads for each theme… and just track CTRs and conversions (openx does that). Then based on theme, I track additional stuff like putting in graphics and stuff, and they do have different impact based on theme. Due to the simple metric (CTR, and endgoal conversion based on theme(ad)) I prefer openx, because you can easily run some simple additional split tests with it.
What’s great about this plugin is the speed of being able to test a theme. Simply copy the directory, rename the plugin, do the changes, choose the SES settings and it’s running.
Works out great. All in all, an amazing time saver and headache reducer. Thanks for saving dozens of neurons!!
Hi! You did a great job developping thsi plugin so thank you very much.
However i had some problems with it, stats from my GA account:
* All Visits : * 163 Visits
* Split B : * 24 Visits
* Split A : * 20 Visits
Those numbers didn’t add up and now i know why:
You need to put setVar before you call trackPageview, and my theme and plugins where constructed in a way that setVar was after trackPageview and it was counted by GA only when someone clicked another page on my website.
I dealed with this issue manually, but today i just changed 1 line in your plugin code and it works now. I changed this line of code:
add_action(‘wp_footer’,array(&$this, ‘output_themesetvar’),99);
to this:
add_action(‘wp_footer’,array(&$this, ‘output_themesetvar’),1);
And now this plugin code appears before all of my Analytics codes in the footer.
Best regards
Mario
Hi – I’m actually surprised things work at all with your changes.
Ideally, we should set the variable after the tracking object has been created, but before trackPageView is called. To do this properly, we need the right hooks in the main analytics plugin.
I’ve contacted Joost de Valk about his Google Analytics for WordPress plugin, but heard nothing back. I’ve also contacted Ronald Heft who has said he’ll add it – but nothing’s appeared yet. If you have suggestions about other analytics plugins I’m happy to contact the authors involved.
You are right, it doesn’t work with this changes, i made so many changes that i was mistaken that it works. Now i’m trying other modifications and i’ll let you know when i get some results. When i use your plugin without any modifications i get those strange numbers that i wrote before. I think it’s because of the “pageTracker._trackPageview();” code which is executed before setting custom variables and those are recorded only when a user is reloading the page or visiting another page on my website. This could explain the weird bounce rates:
* All Visits :
* 85.13%
Bounce Rate
* Split B :
* 18.52%
* Split A :
* 17.39%
You’re absolutely right. I know exactly what the problem is – but it’s difficult to fix without support from the analytics plugin authors.
Which analytics plugin are you using?
I don’t use any analytics plugin, in my theme ( carrington theme – great for developers ) i have a place where i can put my analytics content that goes to the footer. I got it to work by deleting “pageTracker._trackPageview();” from GA given code and pasted it in your plugin into the “output_themesetvar” function:
function output_themesetvar() {
if ($this->splittest != “”) : ?>
try {
pageTracker._setCustomVar(1, “variation”,”splittest); ?>”, 3);
pageTracker._trackPageview();
} catch (err2) {}
<?php
endif;
}
and it works like a charm. I changed the _setVar fn to _setCustomVar, because the first is deprecated but it should be working both ways.
I think with the new asynchronous GA code ( _gaq.push(['_setAccount', 'UA-XXXXX-X']); ) it should be working fine without any modification but i haven't tested it yet.
I’ve known about how important testing is, and have seen/used the website optimizer plugins for pages, but have been wondering how to A/B test a whole theme. So this looks absolutely awesome.
Did the issues w/ the analytics get worked out? A few comments up someone mentioned that the number of visitors sent to theme A and the number sent to theme B did not add up to the total visitors. Also, which analytics plugin does this one play well with?
Thanks for the great work.
Hi – thanks for the comments. You’re right, that A+B doesn’t equal total right no – but we know why! The difference is users you bounce straight out of your site. Unfortunately none of the Analytics plugins I’ve contacted have been responsive enough to add the right hooks for us to drop into yet (Joost de valk (GA for WordPress) hasn’t replied, and Ronald Heft (Google Analyticator) has said he’ll add it in the next release, but I don’t have visibility of when that’ll be.
If you’re using something else I’ll gladly work with the authors to get the relevant stuff added – or if you’re hand-coding your analytics JS in your theme I can explain how to get it working.
Great news. The split testing is very used right now, all want to optimize the contnet, the sales pages, everything. While the sales pages are easy, you can create 2 similar pages, i have an website where i want to test adsense placement where this plugin rocks.
Great post Lee!
Question for you. Rather than A/B test two different themes, I want to just test the menu navigation wording.
For example Test A would have “Our Services” as one of the nav menu items, and Test B would have “Consulting Services” (just an example). Can you think of an easy way to do it without having to do two separate theme installs?
Thanks!
Hi Matt,
Thanks for your comment. This is actually pretty easy to do.
Set your WordPress theme up so that it has two sidebars (We’ll call them sidebar-a and sidebar-b for convenience). So – your theme will include 2 calls to register_sidebars – one for sidebar-a and the other for sidebar-b (Plus whatever other sidebars you have).
Set up the two widgets you need, and publish one to sidebar-a and one to sidebar-b.
Now set up your theme so that only one of the sidebar A is actually displayed. Copy the theme, and amend the copy so that only sidebar B is displayed.
Run a split test with the two themes. Job done
Isn’t it possible to test different copy versions with this plugin in this same way?
Thanks!
I’m afraid not – this plugin is just for testing themes…
Great Lee! Thanks for the quick reply. I will try it out.
Will this allow me to see the results in Google Website Optimizer, or just in GA itself?
GA only. This is nothing to do with Google Website Optimizer.
Hey man is this compatible with WordPress 3.0?
Hi Brad,
I haven’t formally tested it – but I can’t see any reason why not. If you have problems, let me know and I’ll fix them up – but no complaints so far!
Hi Lee,
Thanks for creating such a fantastic plugin. I’ve finally got round to using having come across it a month or so ago.
However, I’m trying to test a child theme and unfortunately this seems to stop the plugin working properly.
When I select the child theme and another theme (eg classic) on the plugin options page, then after I submit, I have 3 boxes selected – as well as the child and classic theme the parent theme gets selected. Ditto if I only select the parent theme and another theme.
Then the split tests only seems to test the parent and the classic theme. The child theme doesn’t seem to appear through random chance (though I haven’t tested this extensively due to the following issue).
And when I try to view the child theme (using the wp_splittest_force parameter) only a small part of the page appears properly (I’m guessing that it can’t access the parent theme files).
This is a shame as I don’t want to test the parent theme, just the child vs my current theme. I’m going to try creating a theme which is a combination of the parent and child and doesn’t inherit anything. I’m sure this will work with your plugin. However if you have any alternative suggestions I’ll be v happy to try them!
A great plugin, and if you can fix this issue it will be greater still!
Cheers,
Mike
Small note: will produce an error if you short_open_tag = Off in your php.ini
So change two times “<?" to "<?php"
Cheers!
Hi Johannes,
I’ve just pushed out an updated version (1.1) that fixes this. It should be up on WordPress.org within 30 minutes. Thanks for the feedback.
…and in v6.1.1 of Google Analyticator it sais: “Adds two hooks, google_analyticator_extra_js_before and google_analyticator_extra_js_after, enabling other WordPress plugins to insert additional tracking code.”
Is this the hook you have been waiting for?! =) (strange numbers issue)
Yes, although I haven’t had chance to re-work the plugin to take advantage of it just yet!
Thanks for the fast reply! Even without those exact numbers I’m excited to test the plugin tomorrow!