What is Split testing?

 

Split testing consists of taking two or more variants of a web page and dividing your website traffic between them. The test's goal is to determine which variation provides the best performance, as defined by the parameters of the test.

Variants are fully developed web pages, which are stored on the server, and are accessed via different URLs. The mechanism used to split the incoming traffic across variants is known as redirection, and for this reason, the technique is also often known as a redirection test. Often, an existing page is used as a control, and the variants are used to gain insights into the various aspects of its performance.

Split URL test vs. A/B test

On the surface, a Split URL test looks just like an A/B test and is, in fact, quite often confused for the same process. Both types of testing are conceptually similar, as they both involve multiple variants of a web page to determine which one converts the best or has more effective user engagement. However, they are most effectively used together, rather than alternatives to each other.

Let’s start by breaking down what an A/B test does. It takes two similar versions of a web page, which generally have a minor change at a page element level. The variable is often a design or content change in a single place. For example, the color of a CTA button or the page's title could be a potential variable. The test is then set up, indicating the different versions. When the test is executed, all the versions are accessed by the same URL. A/B testing is a great optimization tool for individual web pages and is often used to conduct quick tests.

On the other hand, the variants of a split URL test vary at a page level rather than at an element level. As long as the page's overall goal remains the same, the variants can be dramatically different. In fact, a common use case of this CRO technique is to test out redesigned websites, using the existing design as a control.

 

When to use what

As we said before, A/B testing and split testing are not mutually exclusive, but complementary processes: split URL testing is for big changes, and an A/B test is for optimizing the existing page.

We recommend using a split URL test to give your brand new redesign a dry run, page by page. Keep your existing web page intact as a control, and host the variant on a different URL. Once the test has reached its conclusion, use those insights to pick the one with the best performance as a control for your A/B tests. Then, optimize the redesign further by changing individual page elements. Through this process, you can save valuable resources by taking out the guesswork from your redesign at an early stage, and focus on improving only the version that has a proven track record.

How to set up a Split URL test?

As we mentioned before, Split URL tests have a larger scope than A/B tests. They are more flexible because the parameters you can change and test are more diverse. Keeping that in mind, you can look at an existing web page and analyze it beyond just its UI elements. But before we dive into the benefits of this approach, let’s start with how to set up a test:

Build an alternative

Design a new web page using all the information at your disposal, finding solutions to reach your goals better. Since split URL tests can accommodate much bigger changes than A/B tests, you can consider making changes like altering the workflow of a group of pages. For example: can the checkout process be easier? Should the cart open on the side of the page rather than take the user to an entirely new page? And so on.

Gather insights

Put together all the information you have about the web page. Pull together statistics of user behavior, check analytics, and ask for opinions. Correlate with user feedback through various support channels.

Tip: User feedback plays a huge role in designing a great website or product. Implementing the opinions of existing users increases their tendency to identify with your product, and therefore their inclination to use it.

Formulate a hypothesis

Make an assumption based on your insights: if you change X, Y will happen as a result. This hypothesis becomes the framework of your test and how you determine whether or not it was a success.

Tip: Structure your hypothesis in the form of if-then statements. For example: if I change the layout of the web page, then it will be easier to read, and therefore the bounce rate will reduce.

Identify the web page you want to optimize

Make a list of the problems you see with it. Ask questions like: What are my goals for this page? Why do I feel those goals aren’t being met right now? What are the aspects of this page that I feel could be better?

Define success metrics

Identify the KPIs you want to track for the test. Typical KPIs for split URL tests include conversion rate, downloads, and bounce rate, among others. Match these to the relevant parameters for the test.

TEST!

It is important to let the test run for an adequate amount of time. Many one-off occurrences can skew the test results, and time is the only way to counteract any spikes. This is known as allowing the test to reach statistical relevance.

Benefits of split URL testing

A/B tests and multivariate tests are great tools because they enable marketers to test out different variations for UI elements easily. Testing tools used for A/B tests usually have an easy-to-use interface, and the variants are created with just a few clicks. However, the ease of setting up these tests also becomes their limitation. It is not feasible to make radical changes to a web page using just an interface. More often than not, a new web page involves the combined effort of the design, UX, and development teams, thus making it altogether a larger exercise. Split testing allows you the freedom to makes these changes without any limitations and yet leverage the benefits of testing variations. Therefore, split testing is not just a great tool but a powerful one as well. Some of the benefits -

Types of Split URL testing

Since the variants of a split URL test are fully developed web pages, stored on the server, and accessed via unique URLs, there is a lot of flexibility when setting up a test. Here are some of the different ways you can configure your tests to bring up variants:

Simple redirect

Plug in the full URLs for the control and the variants in the test. This approach works well if the test pages are standalone. For instance, if you have a Features page to test, and a couple of variations on the design, this would most likely take the form of:

Control: http://www.mywebsite.com/features

Variant 1: http://www.mywebsite.com/features_vt1

Variant 2: http://www.mywebsite.com/features_vt2

Several pages

Test groups of related pages together, by providing a base URL and indicating that all subpages are part of the test also. This is a particularly useful function of a testing tool, whereby entire workflows can be tested out at once. The insights of these tests would be of greater value, as the entire workflow is taken into consideration. A good use case of this type of split URL test would be the sign up process. At the very least, there is a form and an acknowledgement, which then potentially moves into a marketing page. Assuming that all the webpages to be tested as a part of the workflow are subpages within login, an example of this sort of redirect is:

Control: http://www.mywebsite.com/features

Variant 1: http://www.mywebsite.com/features_vt1

Variant 2: http://www.mywebsite.com/features_vt2

Event-based redirect

This option is a lesser known version of split testing. It is a code-centric approach, and necessarily requires the involvement of a developer. Event-based redirection refers to the kind of test that executes a portion of code, depending on whether or not a user has been redirected to a variant. These makes for more complex test cases.

Metrics for Split URL tests

Metrics, or KPIs (key performance indicators), enable you to create a framework for the results. By defining these at the start of the test, you get more insightful reports to base your decisions on. Here are some of the common metrics used in tests:

Choosing a good Split URL testing tool

There are quite a few factors to think about when choosing the best testing tool for your needs. Consider factors that will save you time in the long run, and prove to be an asset for your productivity:

Easy to integrate

There will need to be some engineering involvement for the initial set up, but each tool varies in its complexity. Look for a service that has solid technical documentation and rapid integration. Get an engineer to weigh in with their opinions too.

Bells and whistles

Choosing an overly complex tool may give the impression that it has more features. This is not always the case. You want a testing tool that helps you get started, and thus get results, quickly. Intuitive interfaces and well-designed dashboards make the testing process more effective.

Statistical significance

A good testing tool will indicate when the test can be stopped. It does this by ascertaining that the test has reached statistical significance. This is an important feature that helps with mitigating the SEO impact of split testing.

Integration with other CRO tools

Can you perform user segmentation easily? Does your split URL test report include a heatmap? These are qualitative insights which make your reports more meaningful and yield actionable results. Read more about how heatmaps visually present insights about your users’ behaviour.

SEO considerations of Split URL testing

From an SEO perspective, there is the factor of duplicate content to consider when conducting a split URL test.

Search engines define duplicate content as exact or extremely similar chunks of content across multiple web pages, with only minor differences in images, design, or text. Undoubtedly, duplicate content does present a challenge that affects search engine user experience, so there are means to account for cases where it becomes necessary.

Let’s look at how a search engine views duplicate content. A search engine indexes pages to maintain relevancy for search results, and as part of its user experience, it will avoid showing pages with similar content. Thus their algorithms will consolidate all the pages with similar content, display the original or best one as per their discovery, and filter out the other pages.

Search engines also have to contend with the repetition of content in the case of deceptive SEO practices. Google has been known to delist websites from their search results if their algorithms have found duplicate content to manipulate search results. However, they rarely impose a penalty, as is commonly feared. Google has explicitly stated that they recognize cases where duplicate content is unavoidable: mobile and desktop versions of the same page, for example. In fact, search engines do a fairly good job of handling duplicates on their own.

However, it is something best not left up to chance, as there are reasons – apart from the rarely imposed penalty – why you wouldn’t want your variant to show up in search results, as opposed to your control. Let’s look at the recommended ways to handle this.

Measures to get the right page indexed by search engines

 

These pointers can help you rank better in search engines and amp up your SEO efforts.

 

Specify which page is canonical

Indicating a canonical web page tells search engines to crawl it more regularly, and in case there are other pages with similar content, to serve this one up in the search results instead of the others. It is important to be aware though that marking a web page as canonical is along the lines of a stated preference. There are several ways search engines decide on which web page is displayed, and the tag merely indicates the preference of the website owner.

Use the correct redirect

There are two types of redirect within a split URL test: a 301 permanent redirect, or a 302 temporary one. Using a 302 redirect instead of a 301 will signal to search engines that the redirect is temporary and therefore not to replace the original URL with the new one in the index.

Don’t use cloaking

Google wants to see the pages that are shown to actual users, because any other behaviour is considered deceptive. It is common advice to use a noindex tag on test web pages, so that search engines don’t index or rank those pages at all. However, this is perceived as “cloaking” to the search engine, and can potentially affect the website negatively. Another version of cloaking is to serve content based on user-agent. Again, the same pitfalls apply.

Reasonable timeframes for tests

Tests require adequate time to complete, and to generate conclusive and valid results. There are several ways to determine whether the test has concluded or not, and it is based on conversion rates and traffic to the website, among other factors. Once the test has reached statistical significance, the changes need to be implemented and all test elements removed.

Best practices

Testing consumes resources, so you want to maximize the output you get from any test you run. Let’s look at some of the things you can consider to achieve optimum efficiency

Keep detailed records of testing history

Insights from previous tests can guide your future tests considerably, allowing you to test smarter each time. Old tests will indicate what works and what doesn’t, so you don’t have to look at those elements again. Store results and what actions were taken based on those results.

Big picture

Optimization only works if the objective is clear, definable and measurable. It is best to consider the big objective – the organizational goal – when designing tests, and not to get absorbed by the individual page goals

Designing good variants

Come up with widely differing variants to cover more ground on an individual test. Once your general direction is established, you can then optimize the page further with smaller tests.

Control stays put

It is tempting to make changes to the control before the test completes. However, this can alter the results of the test. So it is best to wait until after the results are in, and then run a new test.

Account for extraordinary events

One-time events, like a news article, can sometimes drive up traffic, and change results. It is best to disregard the results from this period of transient visitors, and consider only those on either side of it.

Be patient

Running tests take time, especially if you want to get valid insights. People need to participate in the experiment for the test to generate robust and unequivocal results. Letting the test run its course will eliminate surge events, and present a more evened out report of user behavior.

Split testing is a great addition to your conversion rate optimization deck of tools. So now that we’ve covered all the aspects of split testing, all that’s left to do is implement it.