If you’re working to improve your site search it’s sometimes difficult to know what’s working and what’s not. You do a bunch of stuff — modify settings, change the user experience — and measure the outcome. If search success is better, then you pat yourself on the back. Your search improvement efforts worked! But did they? Search is really dynamic. If nothing else changes, you know that the content changes constantly. So was it a content change that improved your search success or something you did to the engine? Fortunately, you can take a lesson from marketing to assess the effect of changes. You can A/B test your search engine.
A/B Testing Site Search
A/B testing is a pretty simple concept. You present to a target audience one of two alternate experiences. You measure each experience for the desired outcome. The outcome that scores better then becomes the default experience going forward. You experience this every day when you click on a web advertisement. The landing page you see is either one of two that they’re testing or the one that “won” the competition. Promotional emails you see in your inbox are tested in the same manner.
A/B testing site search is the same principal. When a searcher enters a keyword they can be directed to one of two search engines. One engine is the base production engine. The second engine is the one with your improvements — different algorithm tuning, a different results page, different whatever. You measure the search success of one against the other. If things get better on the second engine you can then push those changes to the base engine. Search success improves. Your customers find what they need. You do more business.
Five Search Improvement Tips
So that sounds pretty easy. Let’s talk about a couple of things you can do to make sure that your A/B test is the most effective it can be.
- Measure outcomes – Measurements are at the heart of A/B testing. Before you start, make sure you have the ability to measure search success. It’s the only metric that matters to your visitors and it should be your north star in measuring search improvement.
- Be patient – You don’t need a ton of data to know which version is better but you do need to be thoughtful about how much data you need. In theory, all you need to do is calculate a sample size and measure the sample. The reality of search is more complex. If you have high keyword density — a few keywords make up a majority of your searches — this approach might work. But most companies have the opposite problem, in some cases, hundreds or thousands of keywords account for the majority of the volume. So you want to let your test run for a week or two at a minimum.
- Prepare your team – Executing an A/B testing program does require some discipline. As mentioned above, you need good measurements. But you also need a good process. Essentially you’re running a scientific method experiment. So you’ll want to make some guesses about which factors need to change, make a prediction on the desired outcome (e.g. success goes up), run the experiment, measure the outcomes of the experiment and then either accept the changes into production, modify the experiment and rerun, or reject the changes and come up with another experiment.
- Focus on the keywords – One of the things that you should look at in your experiment results is the keyword distribution between the A and B engines — especially for your high volume keywords. If there’s something funky going on there then you may want to change your routing so that it distrubutes between the A/B engines at the keyword level.
- Develop a relationship with your search vendor – Your search vendor can be helpful as you build out your search improvement capabilities. Sometimes your A/B tests may turn up results that don’t make sense — you’ve done something obvious that should improve things yet things don’t improve. Your search vendor can provide insight into the particulars on how a feature or function performs in the wild. The expertise can dramatically shorten your learning curve.
Learn More: Free Webinar
Mike Moran and I are going to talk about A/B Testing Site Search on the Biznology Webinar. Click through to watch the recording — you’ll get lively conversation, rambling commentary on the state of tech, and a few more tips about making your site search better.