skip to Main Content
Example diagram showing how to a/b test site search

Five Effective Strategies for A/B Testing Site Search

Recently, SoloSegment CEO Steve Zakur participated in a webinar on Biznology and shared five strategies for how companies can improve onsite search using A/B testing. Using these methods you can test new settings against a portion of your live traffic without risking tanking things altogether. Once the new settings have proven themselves, you can deploy them to the primary search engine.

Most companies don’t measure site search well. They rely upon out of the box measurements that measure activity instead of outcomes. Nowhere is this problem acuter than when you move from one search engine to another.

  • How do you exceed customer expectations if you don’t have an effective baseline?
  • How do you know you’ve done the migration well if you don’t measure the gap before and after?
  • How do you ensure improvements in content and algorithms are effective?

A/B testing of the search engines is the answer. Check out the webinar or you can read the transcript below.

Site search often represents the first personalized experience for customers on your website and is key to improving your website’s engagement, lead contributions, and conversions. If you’d like to learn more about how to do this for your company, don’t miss our new ebook, “The 6 Personalization Truths Every B2B Marketer Needs to Know Right Now.”

Five Effective Strategies for A/B Testing Site Search Transcript

Mike Moran: Hi everyone and welcome to today’s Biznology digital marketing webinar. I’m Mike Moran, founder of Biznology and a Senior strategist at Converseon, Revealed Context, and SoloSegment. I’m the co-author of “Search Engine Marketing Inc.” and “Outside-In Marketing” and the sole author of “Do It Wrong Quickly.” I’m a veteran of IBM, retiring in 2008 as a distinguished engineer.

Mike Moran: Today you’ll be hearing from Steve Zakur who will present “Five Effective Strategies for A/B Testing Site Search.” But before we start, we need to recognize our sponsors: Gerris Corp, a full service digital strategy firm that reaches deeper into the conversation than any other agency anywhere. MountainTop Data, a B to B marketing intelligence company providing marketing lists as well as data cleaning, data pending, and data maintenance services. And SoloSegment: revenue is trapped in your site search. SoloSegment’s Site Search Inspector can help you set that revenue free.

Mike Moran: As we wait for more attendees to join, let me review the format of our webinar. Our Biznology webinars last just 30 minutes. You can easily fit them into your busy schedule. We record each webinar and we’ll email you that link later this week. During our presentation, you can use big marker to chat with your fellow attendees. Just type in the chat tab. There you will see tweets about our webinar. Feel free to tweet with the hashtag #biznowebinar. If you have a question to ask our speaker, click the Q and A tab. I’ll select a few questions at the end of the webinar and I’ll pose them to Steve. Lastly, in the handouts tab, you can find a PDF of today’s presentation. Thanks to all of you for spending 30 minutes with us. We know how valuable your time is so let’s introduce today’s speaker.

Mike Moran: Steve Zakur is a global technology executive who has spent the last decade solving complex business problems and growing technology businesses. Today he’s the CEO of SoloSegment, where he focuses on improving customer journeys that go through site search and applying machine learning to improve marketing conversion rates. So if you’ve ever struggled with your site search, this is the webinar for you. Steve, take it away.

Steve Zakur: Thanks very much Mike. So today we’re going to cover A/B search testing strategies. And before that I wanted to create some context for search in general and site search in particular. And we’ll start with how site search fits into your overall search portfolio and then more importantly what the business value of that is to the business.

Steve Zakur: At the end of the day, everything you do here, everything you do to improve site search has to deliver value to the business and to your customers. Most companies think of site search as part of our portfolio. As a matter of fact, we were talking with a client yesterday who has one executive responsible for organic, for paid search, and for site search. In other companies, these things are split with marketing generally owning organic and paid and the IT teams own site search.

Steve Zakur: But regardless of how the management structure works within an organization, it really is important to bring these three things together if for no other reason than a very complex term called co-optimization. As you seek to improve the site search experience, (that is to say the experience for folks who are on your website), you’ve already captured them as prospects or as customers. As you improve their site search experience, all that effort yields benefit into organic and, to a lesser extent, paid search.

Steve Zakur: So looking at site search as part of this overall search portfolio is critically important because of the interdependencies and the inter benefits of doing so. Even more importantly is talking about the business aspect. And so sometimes the technology teams talk about improving search in terms of infrastructure, in terms of performance. But at the end of the day really talking about them in terms of the business understanding of the values is going to allow you to make compelling business cases for improving site search for your customers.

Steve Zakur: So seeing things as simple as if someone’s coming from a marketing campaign, marketing wants them to become a marketing qualified lead and of course sales wants them to ultimately become a sale. But talking about in terms of improving site search, improving that portion of the visitor or buyers journey that goes through site search yields marketing qualified leads. And no matter what the case, we could talk about support use cases. You probably have many other use cases within your organization. There’s always a business value statement that can be attributed to that.

Steve Zakur: Because those business teams have business objectives that are quantifiable. And of course improving all of that improves the customer experience, hopefully improving customer satisfaction and net promoter score values. Most of you who are working on site search probably already know this, but I think it’s worth covering just to make sure we have a common language for this presentation. When you look to improve site search results, there are two approaches that you take.

Steve Zakur: For your popular keywords for those terms that are generally your top 100, top 200 terms and have a lot or a high percentage of the volume overall of your business, you’re going to mostly focus on the content because having the right content curating for those top terms is critically important. One, there are a lot of searches for those, but two, it’s usually very obvious what the right content is. So you’re going to spend a lot of time handcrafting the content that appears in those searches and really working with the content owners to make sure that happens.

Steve Zakur: What we’re going to talk about today is really that long tail problem, which is what we refer to again as the unusual keywords. It’s generally those keywords that get fewer searches, but there’s a lot more of them. And of course you could never hand curate the response for your bottom 10,000 keywords. So you’re going to take programmatic steps to improve those search results. Most of that’s focused on the tuning of your search engine, but it can also be additional indexing of your content or working with content management systems or databases that contain some of the content to improve the data that’s drawn by your search engine.

Steve Zakur: There really is a long tail approach to the conversation today. So let’s start with, “What is A/B testing?” For those of you familiar with marketing, when you see a marketing campaign or a marketing campaign landing page, often that campaign landing page has been tested using two versions. And the version that gets the best performance (rightly so the most conversions or the most downloads or whatever the marketing campaign objective is), that one wins and then that winning campaign landing page is promoted for everybody to see.

Steve Zakur: A/B testing and search works in much the same way. So it starts with an A engine and a B engine. And so the goal here is at the start, that you’ve created two identical search engines. Both running the same configurations, both indexed with the same content. And in front of those two search engines, you built a query distributor and this is a very simple piece of technology that just takes whatever inputs coming from the search box and for a particular searcher, routes them to the A engine or the B engine.

Steve Zakur: That searcher will remain in either the A engine or the B engine for the remainder of their search experience. You can then allocate those searches based upon whatever algorithm (fancy way of saying percentage) that you choose, but we usually recommend folks start out with 90/10. That is to say 10% of searchers go to your B engine (which is your test engine and is the place where you’re going to test your new configurations, your indexing, or your data) to determine whether or not it’s improving the experience overall.

Steve Zakur: And that really is the goal, right?, to have a little sandbox in the B engine where you can test your hypotheses. If they work, then you can promote those changes to the A engine. If they don’t work, you keep them in the B engine, revise your hypothesis, make some other changes, and go ahead a run a new test.

Steve Zakur: Now you can also use an A/B test in another use case, which is when you’re changing search engines. There are a lot of reasons why folks might go from one engine to a new engine. Usually it’s chasing new features and function. Sometimes an old engine is going out of support and sunsetting so they have to move. We’ve seen that recently with some of the changes that Google has made. But in a like sense, what you want to do is you probably have pretty good performance in your old engine. So when you bring the new engine up, you want to have at least that good a performance in your new engine. And the challenge there, of course, is your old engine has probably had years of tuning and indexing and all the other stuff that makes good search results.

Steve Zakur: So you have to replicate that in your new configuration and new search engine. So in the same sense, you put your old and you new engines behind the query distributor, you begin to route some of your traffic to the new engine and you just look to improve that new engines performance over time in terms of success so that when it’s gotten to a point where it’s performing at least as good as the old engine, you could swap it out and then continue to tune the new engine.

Steve Zakur: And of course that new engine then becomes your A engine. You can stand up a whole separate identical instance of your search engine as your new B engine and then you can run the same testing and production that are referenced earlier. So there are lots of different ways you can use A/B testing, but the goal really is to be able to test in the sandbox outside of your production engine so that you don’t make your production engine worse without having some real live data about what’s going on in the business.

Steve Zakur: So let’s talk about tips about how to do this. And by the way, a lot of these tips really do apply to search improvement in general even if you’re not using A/B engine setup. We, of course, recommend that setup because it allows you to do all this testing without, again, meddling with the large proportion of your searchers, but again many of these tips will apply to any of the work that you’re doing in your search improvement efforts.

Steve Zakur: So tip number one. Measure. This seems really obvious, but it is startling how many companies don’t actively measure site search performance. Only 15% of companies actually have people dedicated to this work. A little less than half don’t measure it at all. They really just have their site search on autopilot. So paying attention and measuring in quantifiable ways is really important because what you measure is what you get. If you’re ignoring them (your results and your searchers), your customers or your prospects’ experience is going to suffer for it.

Steve Zakur: So what do you measure? Success at the end of the day is what your customers care about. Success for them in kind of layman’s terms is that they’re finding the content that they need to achieve their objective. So if they’re coming in to download a white paper, they’re able to find that white paper and download it. If they’re coming in for support, they’re able to find the document that answers the support question. If they’re coming to buy something, they’re actually able to ultimately put that in a cart and check out.

Steve Zakur: So success, for the customer, is kind of conceptual. But as search professionals, we can actually begin to quantify that. And it starts with this simple formula, which is success starts with they see a results page populated with likely results. That one of those results is presented in a compelling fashion that causes them to think or believe that is the answer to their question. And finally that it ultimately does lead them to the answer so that the result they select is the answer.

Steve Zakur: But this can be a little tricky to measure in reality. There are lots of reasons why. The biggest reason is the volume of data, right? If you thought about instrumenting from an analytics perspective, all the results that were likely for a specific keyword, you would be instrumenting tons of content. Unlike a marketing campaign where they might only have 10’s or 100’s of pages and paths to instrument. From a search perspective, instrumenting every path that someone takes through search would be tremendously difficult to do, virtually impossible.

Steve Zakur: And so we have to take an alternative view of search metrics and that is to say not looking for what is happening, but looking for what’s not happening. We look for the opposite of the behaviors that we desire because they’re a lot easier to measure. For example, if someone types in a search query and gets no result, it’s very easy to measure no results. If somebody sees a search engine results page with a bunch of results on it and none of them meet their needs, it’s easy to measure. They didn’t click on anything.

Steve Zakur: And then finally while it’s difficult to measure what happens downstream of the search engine results page, we can measure what happens on the search engine results page if they don’t find what they’re looking for. And that behavior, I think, we’re all familiar with. This happens to me all the time when I’m in Google looking for something, I’ll click on a link, I’ll go see the page, I’ll say, “Yeah. It’s not what I wanted.” I’ll bounce back. And so that behavior, clicking on a link, seeing something, knowing immediately that it’s not the result, and bouncing back is something we call a pogostick.

Steve Zakur: But regardless, this is what you measure when you’re looking to determine if something was successful. Our system does this very well. There are a variety of other analytic systems in the marketplace that you might be using to measure your marketing analytics that may give you some of this. So depending on what your measurement system is, you may get all or none of this. We’ve seen measurement systems that don’t do any of this well. But again, the goal is no matter what your measurement system is, make sure you’re measuring because, at the end of the day, that’s the most critical thing that you can do because you can’t do A/B testing unless you have some outcome that is actually measurable to tell you whether you’re doing better or you’re doing worse.

Steve Zakur: Let’s talk about tip number two. Prepare Your Team. Site search success improvement is team sport. You can’t do it alone. If you’re the owner of search, you can’t do it alone. If you’re one of the functional teams, you can’t do it alone. Again, it really requires a team. And it requires a cross-functional team. If for no other reason than really understanding the keywords and the search terms is very, very hard and we’ll talk about that in a moment when we talk about keywords.

Steve Zakur: At the end of the day, you’re going to need subject matter experts that can tell you and interpret what you’re seeing. More importantly, once they interpret the keywords for you and tell you what they’re seeing, they also probably own the content or have access to the content or the databases or the CMS or whatever it is that’s managing the content. So they can help you improve. Creating that cross-functional team is tremendously important and having those voices at the table.

Steve Zakur: It also helps you with your business value case, because if you’ve got them at the table and they understand their role in improving search success, it’ll help you demonstrate the value to the business that site search brings. And of course, that team has to have a management system that’s fairly disciplined. The scientific method at the end of the day is the method that we all use in accessing and improving our performance problems. You start with a guess, right, a hypothesis. “Gee, we’ve gotta change this setting in the search engine or index this content or make these changes in this content management system.”

Steve Zakur: Whatever that is, you come up with some hypothesis about why performance is low for a certain set of keywords. And then you run your experiment. You change your setting, you index your content, you do whatever it is that you think is going to yield the improved performance and you do all of that in your B engine (your test engine). Then you observe what happens in your test engine. And if things get better, great. You promote that to production. If things don’t get better, you revise or hypothesis and change. If things get a lot worse, and we’ve seen that happen, you breathe a sigh of relief that you have an A/B test engine, because none of that bad stuff got into your A engine. It only lived on your B engine and only affected a small subset of your users.

Steve Zakur: So again, making sure you have the right team to help you understand and interpret search results is critically important. As with most things in life, patience is a virtue in measuring and improving site search. For those of you who have worked on organic search, try to get your search rankings for keywords up in Google. That is a pure game of patience because you’re competing for keywords and the algorithm is mysterious to you, so it takes a long time for things to change there.

Steve Zakur: The same generally can be said for site search. Now while you do have better visibility into the algorithm, you can make the changes there. You have much more control over the content and generally, you’re only competing against yourself. By the way, sometimes business units do compete against one another–the marketing folks will, in one product, want a term while the marketing folks at another product team will want the same term.

Steve Zakur: Generally the competition’s a little more friendly. But at the end of the day, moving the needle on search improvement can take time. And you need a fair amount of data in your data set. If you’re getting half a million searches a month, like one of our clients, then great. You’ve probably got enough data in a couple of days to tell you whether your change is working or not. But if you’re only getting 10,000 searches a month, you might wan to let your experiment run for a bit longer. And you really want to spend some time interpreting those results.

Steve Zakur: Really look at and ask yourself why something’s happening. Because it really is in that understanding that allows you not only to improve the thing you’re testing today but to learn how to improve other things as well. We have one client that we’re working with and they have lots of alphanumeric searches because they have standards and regulations to go after. And that part of the work that we’re doing with them is really understanding what’s going on with some of these keyword sets, and it’s actually providing insights into other keyword sets that dramatically help us help them and them help themselves in improving search results.

Steve Zakur: Tip four. It’s the most obvious tip today, but it really is Focusing on the keywords. First, as I talked about earlier when I was talking about the team, you really do want to understand what you’re seeing. Again, many of our clients have lots of alphanumeric stuff, which is cryptic, it could be invoice numbers, product numbers, standards numbers, regulation numbers, you name it. And so understanding what you’re seeing and grouping those things, especially these alphanumeric, complex alphanumeric problems, is really, really important.

Steve Zakur: Those subject matter experts are going to help you understand that. So the finance guys, they’re going to know what an invoice number looks like, every product team is going to know what their product numbers look like, the customer success teams might know customer numbers and those sorts of things. So they’re going to know exactly what those keywords mean and help guide your efforts. And also they’re going to be great partners because they often own the systems that will help you with some programmatic fixes. So, for example, if there’s a system out there that just spitting out garbage titles, the person who owns that system is probably going to be your subject matter expert or know the person who’s owns that system. They can help you make those programmatic changes. So focusing on these keyword groups and the subject matter experts who control them or at least are informed about them, that’s going to be a critical element to helping you improve your search results.

Steve Zakur: The final tip is Know Your Vendor. The great thing about site search as opposed to Google or Bing or any of those guys, is that you have the ability to play with the knobs and the dials on your site search engine. That’s also the biggest downside because sometimes it’s hard to discern what a feature or what a configuration is actually going to yield in production.

Steve Zakur: So one of the great challenges is that sometimes you’ll make a tuning change. While it makes perfect sense that you’re going to get a desired outcome to you and everybody who’s working with you, sometimes you get exactly the opposite. We did have this a couple of weeks ago with one of our clients. They turned on a feature and it was nonsensical to us. And when we finally went and talked to the vendor, they said, “Oh yeah. That’s the way it works.” Okay good. Our bad. We didn’t understand that. But having that instinct or that bias to reach out to your vendor sooner vs. later can certainly save you a lot of heartache in your site search success efforts.

Steve Zakur: So again, lots of tips in here, lots of ideas for that journey of improving site search success. I can’t stress enough the importance of having that A/B set up. It has allowed us to work faster with a lot of clients and even for those clients that didn’t make rapid progress. The B engine really is the sandbox where you can test things out. The amount of understanding and insight you get into your search journey is just immeasurable. Mike, back to you.

Mike Moran: Well thanks Steve. I’m sure our attendees have a much stronger idea of how to A/B test their site search, but you didn’t answer every question. I’ve got several good questions from our audience teed up for you. And I’d like to remind our audience that it’s not too late to ask your own question by typing it into the Q and A tab. People who use your site search are your best prospects. They know what they want and they think you have it. If you’re like most companies, your site search lets them down just at the moment they’re ready to buy.

Mike Moran: If you’re ready to improve your site search return on investment, but you don’t know where to start, SoloSegment has a special offer just for listeners of this webinar. The first three listeners to contact will receive 50% off their site search inspector for their website for six months. You can start immediately tracking search metrics plus you can give your team access to a knowledge base that tells them exactly what to do to improve those metrics.

Mike Moran: So it’s and tell them you were at the Biznology webinar to get your half price discount while it lasts. If you’re too late to be one of the top three, you can still get 20% off by mentioning this webinar. We’re about to start firing questions at Steve, but we need to thank our sponsors once again. Gerris Corp: a full service digital strategy firm that reaches deeper into the conversation than any other agency anywhere. MountainTop Data: A B to B marketing intelligence company finding marketing lists as well as data cleaning, data appending, and data maintenance services. And SoloSegment: Revenue is trapped in your site search. SoloSegment’s Site Search Inspector can help you set that revenue free.

Mike Moran: Now on to your questions. Steve, the first question is a really good one, I think. “Why can’t people use an A/B test tool to do this like the one that they already use for other marketing like Optimizely or some other A/B test tool? Why doesn’t that work for site search?”

Steve Zakur: Well first and foremost, there’s a data volume problem. If you were a marketing department A/B testing landing pages coming out of site search, there’s a pure volume problem. But what we’re really talking about is the engine itself, so let’s back up. What you’re really testing at the end of the day is the engine. So you have to have a separate engine configured. An entirely different stand up. And that’s very different from just having a second version of a web landing page. So again, in the concept it’s identical. The execution is very different because those marketing systems will not stand up a separate instance of your search engine. You have to have that second thing established and then A/B test those two things against each other.

Mike Moran: Got it. So if you’re doing the A/B testing, do you have to be technical to do this?

Steve Zakur: Yeah, certainly whoever’s setting up your engine, you’re going to need the support of the technical team to do this. But at the end of the day, once you’ve read the user manual, I’m going to be a little hesitant here, because it depends on your search engine and how sophisticated its configuration settings are.

Steve Zakur: We have one client who uses a fairly new search engine, which has these beautiful little sliders to help you with configuration. But the short answer is generally no. These are really kind of business owner decisions. How are we going to weight one thing against the other? So again, if you were to establish an A/B system, yeah, you’re going to need a tech team to do that. But once that’s done and as long as the user interface for configuration of your search engine doesn’t require essentially code writing, no, you’re good to go. It’s really a business team function that can be managed by a business management system.

Mike Moran: Terrific, terrific. So we got all the ideas behind how you measure the success and lack of success for searches. So in your experience, when you first start working with a client, what kind of success rate do they have and what kind of success do they have after you’ve been working with them for a while?
Steve Zakur: I’d like to say take a guess at what you think your search success is and cut it in half. That’s kind of our rule of thumb. We find generally, I don’t think anybody was below 20% search success, but I also don’t think anybody was above 30% when they started with us.

Mike Moran: Wow. So about one in four actually work?

Steve Zakur: You bet. And by the way, when you factor out the fact that your top keywords, maybe your top 10, 15, 20, generally do really well because it’s easy for people to focus on those. It means the rest of them do horribly. So that’s where people start. Where do people get to? If in six to 12 months, you can get up to about 50%, you’re doing really well.

Steve Zakur: 50% would be good. I think there are diminishing returns beyond 50%. I think if you’re diligent about focusing on it, you can get up probably into the 70% range, but beyond that there’s really diminishing returns. Now if you can get your top 250 keywords to 100%, now that’s awesome, right, because now you’re satisfying five or 10% depending on the distribution of your searchers.

Steve Zakur: You can satisfy them with 100% success. So that would really be the goal. Overall, 50 to 70%, but for your top keywords, you’re top 5% of searches, if you can get them to 100%, that’s an awesome result.

Mike Moran: Terrific, terrific. And so we have time for just one more question. If you had one piece of advice for something that people should test, the very first thing that they should test, what would that be?

Steve Zakur: Assuming that you’re measuring really, really well, that’s always my first suggestion. The first thing you should test is your titles, the weighting of your titles. We find especially when a new engine goes in, it’s those titles that usually aren’t weighted enough and if you have good titles, that’s going to matter a lot. Part of the downside and the challenge that we see with a lot of clients (especially those who have a lot of programmatic content being delivered–one client has a large standards database), is sometimes the titles are absolutely awful coming out of it. Like every entry has the same title. But again, if there were a low hanging fruit if you had really awful performance, and you were going to change one setting, it would likely be the weighting of titles.

Mike Moran: Excellent. Well hey, thank you, Steve, for all this great advice. Terrific ideas, and we especially want to thank our audience for your participation and your questions. If any of you had questions that we didn’t have time to answer, you can email your question to, and she’ll be sure to get them to Steve for the answer. Later this week we’ll send you a link to the recording of this webinar to listen to again and to share with others.

Mike Moran: And we also invite you to come to our next webinar, “Back to Basics: A Method for B to B Success” with Garrett Mehrguth at 11 a.m. on March 6th. We hope to see everyone back here then. Bye everybody!

Back To Top