A/B testing

Why

A minor change in a design may alter user behaviour in ways that are hard to detect in a usability test. An A/B test allows you to compare real-world user behaviour across different versions of a product.

How

In an A/B test, the user is presented with one of two versions of an interactive product. Remote software records metrics for user behaviour. These metrics are compared between the two versions to see which alternative is better. Sometimes, more than two alternatives are tested: A/B/X testing.

Practice

Google does a lot of A/B testing on its search services. The tiniest details can matter to Google because they have so many users. Some companies also use the A/B experimental design for testing concepts, rather than finished products.

Ingredients

• Specialised software.
• Reliable metrics.
• A large number of users.

Phases

• Realisation

Trade-offs

Read more about the trade-offs: Inspiration or Data / Expertise or Fit / Overview or Certainty.