What Are The Features Of A Good A/B Testing Tool?

77

SHARES

Linkedin

Twitter

Facebook

I just carried out a simple test.

Using Google’s search query, I typed this question; ‘how to improve website conversion,’ and I got 370,000,000 search engine results.

That’s a lot of content regarding how to improve a website’s conversion.

I clicked through some of the results, and I saw sections on best practices, the latest design trends, and checklists on how to improve your website.

Note: there’s absolutely nothing wrong with best practices or the latest design trends, but they’ll not always work, and that’s because user needs vary from website to website.

What works is you observing how users respond to design changes on your site and seeing which one they favor and implementing it.

Enter A/B testing.

As a subset of conversion rate optimization, A/B testing is no longer a new field.

It’s been around for a while, and there have been hundreds of tools developed to run A/B tests. You’ll know this if you’re in CRO.

The truth is that not every A/B testing tool out there might have the features you’re looking for to run the type of tests that align with your business goals.

To help you sift through the noise, In this article, we’ll quickly look at what’s an A/B testing tool, the features of an A/B testing tool, and examples of A/B testing tools that have some of these features.

?

To properly define an A/B testing tool, I need to mention what’s an A/B test quickly. 

A/B test is when you put two web pages together with a difference in one element to see which of them performs better.

With this in mind, we can define an A/B testing tool as software that allows you to create different variations of a web page to see which one performs better.

As the web has matured over the years, the approach to A/B testing which improves user experiences on a site has matured too. 

Over time, CRO practitioners have settled on two primary approaches to A/B testing;

Client-side and server-side.

Client-side testing.

A client-side A/B test is when you’re creating a page variation directly on the user’s browser. A/B testing tools that offer client-side testing use javascript to create the test page variations.

Server-side testing.

With the server-side testing, the test is rendered on the web’s server. Here, when a user comes on the page that’s being tested, the server randomly decides which variation to show the user. It could be the control first or the variation.

In selecting the tool you will use for A/B testing, your business requirement will determine if you’ll go for a tool on the client-side or server-side approach.

According to Khalid Saleh, CEO of Invesp; 

“Every business has different needs. That company A is running certain tests doesn’t mean those tests are right for your business. Your business requirements will drive the initial set of features you will look out for in the A/B testing tool you want.”

Having looked at what client-side and server-side testing means, I’ll examine some differences between client-side and server-side testing before going deep into the features of A/B test tools you should look for.

It’s time to examine features to look out for when selecting an A/B testing tool.

The term split URL testing is interchangeably used with A/B testing, but they’re not the same.

A split URL test or split testing is a website optimization technique to test different variations of a website hosted on multiple URLs.

An A/B test, as I’ve already defined, ‘A/B test is when you put two web pages together with a difference in one element to see which of them performs better.’

With a split URL test, website traffic is randomly split between variations, and the variations created are accessed via different URLs. 

The performance of each variation is tracked and analyzed to see which one provided the better experience for users and had the most conversion.

A split test has multiple URLs, multiple variations, and one winner.

If you’re interested in conducting split tests, you’ll want to keep this in mind. You will want a tool that does split URL tests apart from A/B tests.

Multivariate testing is a technique where multiple variables (site elements) are tested, and multiple variations are generated in the process.

Multivariate testing aims to determine which combination of site elements best increases conversion.

Not every CRO agency or consultant is interested in MVTs, but some do, and if you want to carry out multivariate testing, you’ll have to ensure the tool you end up with has this capability.

Not every hypothesis and A/B test plan is built the same.

Some A/B tests are surface level, e.g., moving content around, testing to see if a section is vital in improving conversions, etc. 

These tests can be easily implemented and require no dev time and resources.

Consider these examples; if your hypothesis requires you to reorganize your purchase funnel and product sourcing algorithm, these are a bit more complicated.

You’ll require a lot of development time and manpower, and this isn’t something that a client-side A/B test tool can carry out for you. You’ll need an A/B test tool capable of server-side tests.

With server-side testing, you have many more options to work with, since you can modify all aspects of your site, whether front-end or back-end. 

All of this is possible because you remain in control of the content sent by your server to your website visitors. 

Some A/B test ideas won’t require dev time and resources.

Consider this scenario;

If you want to quickly test a headline against the one you’ve got on your homepage, you don’t need to involve a developer.

Many A/B testing tools have a visual editor that allows you to create a replica of your homepage or whatever page has the headline; then, you can edit it and start the test.

Because you might run tests like this where you want to test certain elements, consider a tool with an intuitive visual editor (drag and drop). An in-house CRO specialist can easily do this.

On the other hand, specific tests will require manipulation of a site page’s code; this is where a tool with a code editor comes in.

This allows your developer to create a variation using javascript and HTML/CSS that will be responsive across devices. 

Not doing this but creating such complex A/B tests with a visual editor will lead to serious response problems from the variation. 

Every business claims to want to do what’s best for its customers. 

To achieve that (customer satisfaction), you need to create personalized experiences.

At the core of running A/B tests is wanting to create the best user experience for site visitors and customers that leads to an increase in conversions (micro and macro).

Not every A/B testing tool has personalization abilities while running tests, and if this is a big deal to you, you’ll need to consider a tool with such capabilities.

Here’s an example of how personalization works in A/B testing;

You can test potential personalizations using an A/B testing tool by setting up an audience for each segment and then A/B testing within that audience.

To do this, first, predefine each target segment based on data you have about them. Then create an audience that matches this definition. Then create an A/B test for each segment.

For each A/B test, assign the corresponding audience that ensures it’s only shown to that segment. You will then find the best page (A or B) to show this segment. 

Everyone has a bias. From the choice of music we listen to, the movies we love, the food we eat, our favorite authors, etc.

The same can be said of CRO specialists and their statistical approach. 

There are two fundamental ways CRO specialists approach statistics; the Bayesian and Frequentist models.

That said, every tool has a way they record and compute its statistics. 

For some, it’s Bayesian, for others, it’s Frequentist.

When selecting an A/B test tool, you must consider your favorite statistical approach.

If you’re Frequentist, you don’t want to pay thousands of dollars for a tool whose analysis is Bayesian.

No A/B testing tool is perfect and foolproof. 

This means you’ll need assistance sooner or later, but it’ll happen. 

A distinguishing feature of an excellent A/B test tool to keep in mind is your access to support staff when you encounter an issue using their product.

Another perspective to look at support from is access to a knowledge base and documentation that make it easy to sort out significant issues yourself.

Customer needs are constantly changing; this means to stay on the top of your game, you need to evolve to satisfy customer needs.

This will require you to update service offerings, rollout features, make improvements, etc.

Many businesses routinely release new features without testing to see if it’s something customers are interested in.

This leads to a drop in interaction and engagement, which can impact revenue.

This can be avoided by running A/B tests with a feature flagging capability.

Not every A/B testing tool has a feature flagging capability. Still, if your business needs to roll out features to meet customer desires, you’ll need to look for an A/B testing tool with a feature flagging capability.

Many businesses today have many tools that make up their tech stack.

Before picking your A/B testing tool, you need to ensure the tool can integrate with a range of relevant technologies. 

This way, you’re not stuck, or you don’t have to spend a lot of cash on dev to rectify problems created. 

New tools are released every time.

Before paying a huge amount of money for a tool, and connecting it to your system, you want to ensure it has been around for a while.

There are different ways you could know this information; asking the founders directly, looking in CRO forums, review sites, etc.

You don’t want to jump on new tools though they might be cheaper because they’ve got a long way to go in features and capabilities.

Figpii isn’t just an A/B testing platform; it’s a suite that provides many tools for CRO as a whole.

Within this platform, you can find A/B tests, heatmaps, session recordings, polls, etc.

Pros

1. FigPii has both AB testing and Split URL testing.

2. It offers native integration with Shopify and WordPress, which includes revenue reporting.

3. Figpii offers advanced targeting, custom JavaScript targeting, unlimited Subdomains, targeting by campaigns (query), visitors targeting (New vs. Returning), source targeting, etc.

4. It has several goals set up like custom events (JavaScript), Page visits, cross-domain tracking, and domain-wide goals.

5. FigPii has 2 options when it comes to statistical models like;

• Bayesian (A winner will be decided after a variation has a significant enough chance to win, variation weights are not changed).

• Multi-armed bandit algorithm (Variation weights are continuously redistributed to give more weight to better-performing variations).

6. It has IP targeting for quality assurance services. 

7. It helps you launch and pause your tests at specific timings according to your preference.

8. It has integration with Google Analytics, where you can use custom dimensions to analyze your data further on Google Analytics. 

9. FigPii offers unlimited concurrent tests.

Cons

1. Figpii doesn’t offer tests on mobile applications.

Trusted by top businesses such as Klaviyo, Disney, and L’Oreal, the AB tasty tool provides an omnichannel experience (desktop, mobile, IoT, etc.).

With the AB tasty platform, you can run both client and server-side tests, personalization (AI-based segmentation and audience builder), audience activation, AI and ML, etc.

Pros

1. The interface is clear to understand.

2. Test setup is easy.

3. Many integrations are available, like Google analytics, kissmetrics, etc.

4. Pricing is mid-range, unlike other competitors.

Cons

1. The statistical significance calculator is somewhat basic.

Adobe target is a product in the Adobe experience cloud solution that provides everything you need to tailor your customer experience.

It provides A/B and MVT tests, multi-armed bandit testing, server-side optimization, mobile optimization, on-device decisions, connected device optimization, etc.

Pros

1. It features in-depth personalization that can help create relevant experiences for your site visitors.

2. Neat user interface.

3. It has recommendation modules for advanced optimization.

Cons

1. Integrations are complex and require a subscription to the other tools in the Adobe ecosystem.

VWO is an AB testing and conversion rate optimization tool for enterprise brands.

Trusted by brands like Hyundai, Wikijob, and Microfocus, VWO is the choice of thousands of businesses for ther AB testing needs.

Pros.

1. You can run multiple types of tests on different URLs or via Javascript.

2. VWO’s interface is laid out in a user-friendly format. It makes it great for UX teams to onboard stakeholders outside of the research department.

3. It thas a visual editor for those tests that don’t require a developer. 

4. VWO support team is always available via chat in the web app, and they’re very responsive (within minutes).

5. Surveys, heatmaps, and session recordings are built-in.

Cons.

1. The platform tends to consume a large visitor allotment to find a result. 

2. It’s not easy to run a test on multiple pages at once. For instance, to test something on 10 product pages at once.

3. The observation collection sample rate is a little low (for the price) – it takes a bit longer to collect a meaningful amount of data.

4. Regarding AB test analysis, instead of giving you a specific number on the lift, it gives you a range and calculates “likeliness to beat control.” 

5. Google Optimize

This tool from Google lets you test and tailor different variations of your site.

Suppose you want to start with testing; I’ll recommend this tool.

It has a visual editor, code editor, and native integration with GA, A/B, and multivariate testing.

Pros

1. It has a free version that allows you to run five tests simultaneously.

2. It’s easy to analyze results in Google analytics.

3. Seamless integration with other Google products like tag manager and analytics

Cons

1. It’s lacking in features compared to other tools.

2. Using Google analytics segments for targeting is only possible in 360. 

3. No customer support is offered, only used in the premium version.

Google Optimize Is Going Away. What Are The Alternatives?

Depending on when you see this article, Google Optimize and oPTIMIZE 360 is going away.

What started as a rumor in 2022 is finally becoming a reality.

This announcement has been met with mixed reactions from the conversion optimization community.

Some AB testing tool vendors are ecstatic; they’re touting the slogan everywhere that they’re the perfect replacement for Google Optimize. All these are ploys to get a piece of the pie (Google Optimize clients).

On the other hand, some experts are sad, believing the CRO community is going to lose out on businesses that relied on Google Optimize to power their experimentation efforts.

As regards this information, we’ve got something to share with you.

Yes, there are alternatives out there, but there’s only one tool we can recommend, and that’s Figpii.

Figpii: A Google Optimize Alternative

The truth is this – Figpii isn’t just a Google Optimize alternative; it’s also better than any other alternative.

Here’s a test conducted using Google Lighthouse.

The longer it takes for variations to load on these A/B test engines, the more the tool is prone to flicker effect.

A flicker effect in A/B testing is when the original page shows up before the variation loads.

This is bad for tests because if your site visitors see the flicker effect, it increases their distrust levels go up, which leads to higher bounce rate and longer time for your test to reach statistical significance.

Listed below are some more reasons why Figpii stands shouler and neck high above Google Optimize.

Google Optimize 
Figpii

1. Variations load at 550+ milliseconds which increases the chances of a flicker effect.

Variations load at 180 milliseconds, drastically lowering the possibility of a flicker effect.

2. 10,000 total visitors over a 14 day trial period

Test an unlimited number of times with up to 75,000 monthly visitors at no cost.

3. Zero to no expert support. Use forums for any help you need.

24/7 expert support

4. No such calculators to help with your A/B tests.

Built-in sample size calculator and duration calculator.

5. No built-in session recording engine.

Has a built-in session recording tool and heatmap to help you have unparalleled clarity and precision in understanding user behavior.

6. Uses Bayesian methods.

Scales to include multi-armed bandit testing. Figpii’s flexibility allows you to explore beyond traditional Bayesian methods.

There you have it. Some tools can meet your requirements depending on your stage running A/B tests.

This way, you’re running tests to find out what works for your audience and improving your site’s conversion simultaneously.

Additional Resources

1. Google Optimize vs Optimizely

2. Google Optimize: The good, the bad, and the ugly.

77

SHARES

Linkedin

Twitter

Facebook