Traditional A/B testing splits incoming traffic evenly (50/50%) for design versions A and B. Then, half of the incoming traffic will receive the original version, and the other half will receive the new design variant during the experiment period. This fixed equal distribution of traffic produces more statistically accurate results with unbiased data, but is slow, and negatively impacts conversion rates during the experiment because users who would receive a poorly performing version of the design may not convert.

Multi-strategy helps perform A/B, A/B/n, or multivariate testing faster by dynamically dividing traffic based on the winning version. Let’s understand how multi-strategy handles A/B testing faster without affecting conversion rates by comparing it with traditional A/B testing traffic distribution methods.

What is multi-strategy?

In UI/UX A/B testing, a multi-strategy (MAB) strategy is a dynamic inbound traffic distribution method that redirects more traffic to the current winning version of the design. Initially start with a traditional 50/50% split, but then dynamically increase traffic to the highest performing version to stabilize selected test metrics and traffic percentages to complete tests faster:

MAB isn’t limited to A/B testing — you can use it to optimize traffic for A/B/n and multivariate tests.

The slot machine example is the simplest way to understand MAB:

Assume there are two slot machines with unknown payoff rates. How do you find the most profitable machines without losing more money?

• First, you play equally with A and B
• If machine B performs well, you will use it more often
• Still use machine A occasionally and start playing it more often if the rewards are better
• Choose the machine that gives the most prizes

How does multi-strategy work in UX?

Let’s go over each common step that MAB A/B testing takes to optimize traffic:

---

---

1. Fair initialization: Initialize A/B testing with a 50/50% traffic distribution to provide a reasonable amount of traffic on each version. In A/B/n testing, the algorithm initially distributes traffic based on the number of versions, for example 25% for a four-version A/B/n test
2. Monitoring: Monitor metric changes and identify the best performing versions for A/B testing and rank all versions for A/B/n and multivariate testing
3. Traffic distribution adjustments: Adjusts traffic distribution percentages based on monitoring results. For example, if version B has 5% conversions, but version A only has 4%, the algorithm will increase traffic for version B because it is more successful
4. Stabilize: The goal of this algorithm is to stabilize the metric to stop the process. This usually occurs when the new traffic does not significantly change the percentage distribution of the current traffic
5. Stop: The MAB process will stop when the traffic distribution stabilizes (the dominant version can stabilize the traffic percentage at 80%-95%) and no significant changes in the metrics are observed. For example, if the conversion rates for versions A and B stabilize at 6% and 5%, respectively, the process will stop

Benefits of using multi-strategy

Using MAB over traditional fixed traffic distribution has the following advantages:

• Shorter test: MAB testing stops when a particular version dominates, so testing is typically faster than traditional A/B testing that waits until results reach statistical significance.
• Better conversions during the experiment: The best performing version gets more traffic, so the conversion rate will increase, even if a particular version performs poorly, unlike a fixed, equal share of traffic
• Less wasted traffic: MAB optimizes incoming traffic during the experiment by sending less traffic to underperforming versions, so that incoming traffic is not wasted
• There is no fixed initial sample size requirement: No need to calculate and meet statistical significance requirements for incoming traffic before starting testing — flexible start-up and progression as traffic flows

Limitations and pitfalls of multi-strategy

MAB has its benefits, but the following problems are making designers rethink its use over classical statistical methods:

• Depends on the MAB algorithm: The accuracy and trustworthiness of the results depend on the quality of the implementation of the MAB algorithm that divides traffic dynamically. Weak algorithms can produce inaccurate results
• There is no definite stop time: You don’t know exactly when the MAB test will end, because it waits until the traffic distribution stabilizes, unlike classic A/B tests that start with a predetermined duration
• Statistical bias: MAB starts with a fair distribution of traffic, but then dynamically adjusts the percentages, so the results are biased towards the initial traffic
• Statistical significance is not guaranteed: You cannot present MAB test results with the usual 95% statistical significance because the test does not use a hypothesis-based statistical basis for decision making.

Multi-strategy vs. Multi-strategy traditional A/B testing

Here’s a summary of the comparison of MAB vs. traditional A/B testing:

Comparison factors	Multi-armed appeal	Traditional A/B testing
Traffic distribution	Dynamic, starting with 50/50%	Still, always 50/50%
Speed	Faster, usually ends within a few days	Slower, usually ending in a few weeks
Stop	Unknown, stops when the distribution of metrics and traffic is stable	Predetermined
Statistical significance	Low	Tall
Target	Prizes during testing	Find the real winner

Example use case

MAB tests work best when you need to optimize traffic while quickly evaluating design versions. Here are some examples:

• CTA: CTAs determine product conversion rates, so MAB helps test design versions without losing traffic
• Onboarding flow: Testing two onboarding flows with a fixed 50/50% traffic split can increase product abandonment rates if one version performs poorly. Using MAB prioritizes the best performing streams without running experiments for a long time
• Recommendation system: Imagine you need to do full-stack testing for two e-commerce product recommendation algorithms on your home page. Using MAB helps you find good algorithms, as well as maximize revenue, unlike traditional A/B testing

Tips for designers

Here are some practical tips for running an effective MAB test:

• Avoid premature conclusions: Don’t forcefully stop testing even if you see rapid improvements in the initial phase, always wait until traffic and metrics stabilize
• Monitor continuously: Monitor top-performing version changes, seasonal traffic effects, and terminate them appropriately without overload
• Combine with human insight: The results depend on the quality of the multi-armed appeal algorithm and the initial traffic, so analyze the results yourself without blindly trusting the algorithm results

Conclusion

A multi-strategy strategy may be preferred over traditional traffic distribution with A/B, A/B/n, or multivariate testing if you care more about wasted traffic and less about statistically significant results

FAQs

Is the multi-strategy method better than traditional A/B testing?

Depends on the scenario. MAB is better if traffic loss is critical, and traditional A/B testing is better if you prioritize statistical significance

Should I often calculate the traffic percentage and notify the developer?

No, MAB-enabled A/B testing tools take care of everything — you just need to initialize the test