Everyone wants to have the best and that applies to business as well. We want the best landing pages, the best marketing campaigns, the best employees, etc. Bottom line is that we are addicted to having the best.

That’s not a bad addiction to have. A lot of good things come with being the best. However, how can we become the best and furthermore, how do we know that we are doing the best?

That’s where A/B testing can come into handy. It’s a general framework that can be applied to just about any business out there. The rest of this post is going to go over what A/B testing is.

If you much rather watch a video about this subject, I made a video to go along with this post:

What is A/B Testing?

To keep it simple, A/B testing is essentially an experiment where all independent variables are the same except for one. You change just one variable and see if it effects the dependent variable.

Independent variables are basically your inputs. Things like color of a product, number of sales people, and price of product. These are all things you control, so they’re your independent variables.

On the other hand, dependent variables are your outputs. Things like number of sales, number of leads generated, and number of customers. These are all things out of your direct control, so they’re your dependent variables.

You want to see what set of inputs yield the desired outcome. For example, how does the number of salespeople (independent variable) effect the total revenue generated (dependent variable) for a company. You want to find the optimal number of salespeople that produce the highest amount of revenue for your business.

Let’s go step by step to go from finding a problem all the way to analyzing the data and taking meaningful action.

Step 1: Identify a Problem

You want to find a problem in your business that you would like to fix. Not just any problem however. The problem has to have defined actionable tasks that you can control and have an outcome.

For example, when doing a sales call, there are multiple defined actionable tasks that you must do: the greeting, the rapport building, the soft pitch, the question asking, the actual pitch, etc. Each one of these tasks can be done in a different way.

There is also a clear outcome that we can define. Either the sales call was successful and you got the close or you didn’t and the sales call was a failure.

Once you’ve identified a problem of this type, it’s time to move on to the next step, which is identifying the independent variable that you want to control and your dependent variable.

Step 2: Identify Your Variables

So in every A/B experiment, you identify one independent variable that you would like to change. If you’re running social media ads, this could be your ad image. If you have a sales pitch, this can be the words you use at the end of your pitch.

Next, you want to identify your dependent variable and define what success means for the experiment. For example, if you’re doing a cost analysis for a business, the dependent variable could be total expenses of the business and a success for this experiment means that total expenses go down.

Step 3: Designing the Experiment

Now that you’re clear on what input you want to change for the desired outcome, it’s time to design the experiment. If you’re running social media ads, this could be creating identical ad sets. If you’re doing sales calls to warm leads, this could be doing the same sales pitch except for the ending and then recording whether or not the lead closed.

You want to make sure that you have a solid mechanism for gathering data. If you run the experiment and you get no data from it, you might as well have not even ran the experiment. The data you collect will be the gold that you’re looking for.

After you’ve designed your experiment and you know how you will collect data, it’s time to let the experiment run.

Step 4: Knowing When to Stop

Now that the experiment is running, how do you know when to stop it? If you stop it too early, then you don’t collect enough data and if you let it run for too long, then you’re wasting time and resources.

Before we answer that, let’s take a quick trip to the world of statistics. In statistics, there is a principle called the Law of Large Numbers. Basically what it means is that, given a large number of trials, the probability of an event will reach its true value.

Although an extreme example, imagine if you flipped a coin only three times and you get two heads and one tail. In this experiment, the probability of heads is 66% and the probability of tails is 33%. From common sense, we know that is not the case.

This is the same case in running your A/B tests. Most of the time, things are not common sense and so if you stop your experiments too quickly, then you’re not getting the true values.

On the other hand, you could flip that coin a million times and the probability of heads is near 50% and the probability of tails is 50%. If you flipped the coin another million times, the numbers aren’t going to change that much, you would just be wasting your time.

That can happen in business. You can let the experiment run for too long as well. You’re wasting time, you’re wasting energy, and you’re wasting money.

So how do we know when to stop an experiment?

There’s a method to doing this that involves some knowledge of calculus. If you remember back to calculus, there’s something called the derivative. In physics, the derivative of the position function gave the speed of an object. In our case, if we take the derivative of the probability function over time, we can get how fast the probabilities are changing.

In simple terms, how fast are the numbers or probabilities changing?

For example, if on day 3 you measured the clickthrough rate (CTR) of Ad A to be 3% and then on day 5 it was 2%, then the numbers moved quite a bit. If 3% was considered the “new 100%”, then a full 1% drop means that the CTR changed by 33%. That’s a lot. If that was the case, you would keep the experiment running.

If on day 7, you see that the CTR is only 2.1%, then the probabilities have only changed by 5%, that’s much less and it seems like that the probability of a clickthrough is stabilizing, or in mathematical terms, it’s converging.

Once you see the probabilities or numbers converging like that, it’s time to end the experiment. You’re starting to reach the point where the results are starting to converge to their true values.

Step 5: Analyzing the Data

After you’ve stopped the experiment, it’s time to analyze the data and see which inputs create the desired outputs.

This is a common place to make a mistake at. You don’t want to use absolute scales to compare things, you want to use relative scales.

Comparing something on absolute scales can mislead you to bad business decisions. For example, if Ad A has a 2.8% conversion ratio and Ad B has a 2.7% conversion ratio, on absolute scales, Ad A is better than Ad B because 2.8% > 2.7%.

However, if we made 2.7% the “new 100%”, then 2.8% is only 3.7% better. That’s not much at all, so we would say that Ad A and Ad B perform about the same. This test would be inconclusive.

So when you’re analyzing your data, be sure to use comparative statistics to avoid falling into the trap of using absolute scales.

Step 6: Implementation and Adaptation

If your test was not inconclusive, then there was a conclusion that you can draw from it. Perhaps Ad A was better than Ad B. Once you know that, cut out Ad B and give all of the ad budget to Ad A.

You can even run another test against Ad A and see if you can find an even better ad. In this case, you’re continuously adapting your ads and always trying to find an even better performing ad.

This cycle of identifying a problem, running an experiment, and implementing the results can become one of your best friends in business, where many things are unknown.


A/B testing is a powerful, yet simple, tool that allows you to systematically improve your business and it’s not limited to just marketing. You can A/B test for site speed. You can A/B test for customer happiness. A/B testing is a general framework you can apply anywhere.

The following are the six steps in summary:

  • Step 1: Identify a Problem— You want to find a problem that has defined inputs and a defined outcome.
  • Step 2: Identify Your Variables — You want to figure out what the one independent variable is that you’re going to change.
  • Step 3: Designing the Experiment — Make sure you design the experiment with a solid mechanism in place to record data.
  • Step 4: Knowing When to Stop— Measure the data in regular intervals and see how fast the numbers are changing
  • Step 5: Analyzing the Data— Do not use absolute scales to compare, use relative scales to compare your data.
  • Step 6: Implementation and Adaptation— You want to take action on your conclusions and keep adapting with the market.

If you still have any questions about A/B testing, feel free to contact me on social media, it’s @SunnyChopper on basically everything out there, even Xbox Live.

For more content like this, book summaries, free courses, and much more, be sure to check back soon. Constantly updating it with more and more content to help work smarter than the competition with the power of technology.

Written by Sunny on Dec 21 2018