top of page

On the importance of A/B testing for retailers

by Anh H. Nguyen

"I have not failed. I've just found 10,000 ways that won't work." - Thomas Edison, on never giving up in his quest for the construction of the electric light bulb.

Imagine a world without light bulbs. Or cars. Or airplanes. Or medicine. Or movies. Or the Internet. Virtually all the modern perks and utilities that we take for granted today, from the humble toilet paper to the mighty spacecraft, are the results of continual, incessant experimentation. For everything that works, there are hundreds, thousands, even more, that have failed. Without experimentation going on in every aspect of life, we may never have stepped foot in the Stone Age.

In the previous articles, we have stressed over and over again the utmost importance of gathering data. But having data without literally putting them to the test is like throwing all your prime ingredients in a crockpot and hoping for the best. Business owners may analyze all the data they possess, but jumping straight to actions without testing them is often hasty, short-sighted, and wasteful. So why do retailers keep doing it?

The first reason is the perceived cost of doing experiments. Many companies fear putting money, time, and labor towards "failures" - experiments that do not yield the outcomes they desire. But in reality, companies have more to lose by not experimenting. If seeing is believing (data is God), then testing is knowing (you do not truly know your data until you test them). At the most preliminary level, doing experiments helps companies quickly learn what works and what does not, thus allowing them to eliminate inferior options. This saves them from further investing resources in suboptimal endeavors and is a great example of optimization over saving.

At a higher level, testing could reveal unexpected insights to retailers. For example, when Amazon released a tougher version of the game Air Patriots by accident, they discovered a negative correlation between the game's difficulty and players' engagement. From that knowledge, Amazon tested four different versions of the game and ended up with the most winning one. Like the inimitable Bob Ross once said, there are no mistakes, only happy accidents.

At the most sophisticated level, companies that weave testing into the core of their business model learn not only what works for them, but also why. It could be the most startling discovery, the re-starting point, the axiom from which companies extrapolate all their future business plans. It could also be unsettling and scary, since it is likely to challenge their preconceived notions and prompt a complete overhaul. This leads us to the second reason why companies are adverse to doing experiments. It is a combination of fear of the unknown, corporate inertia, resistance to change, hesitation to upheave the status quo, and a desire to keep things the way they always have been.

A/B testing: only one variant is tested at a time, all other things are held constant.

No matter what their stance, in this increasingly swift-moving world, companies that do not innovate resign themselves to dying a slow death. And testing is integral to innovation. Among many methods, A/B testing is a well-known methodology where two different versions of the same variant are tested to see which one is more effective. It is also the most popular and streamlined one. In fact, a running joke is that A/B testing is a shorthand for "Always Be Testing." However, offline retailers are often at a distinct disadvantage when it comes to testing compared to their online counterparts. Digital companies could lend themselves to running many tests at once, owing to their ability to easily switch between options., for example, is estimated to run over 25000 tests annually. So how do offline retailers, with their vast stores, copious amounts of merchandise, and numerous staff, stop guessing and start testing?

A more thorough visualization of A/B testing.

There are countless parameters pertaining to retail, but here are a few fundamental ones: store, staff, customer, and products. Each category contains many subsets, each of which in turn includes many metrics, like a nesting doll. For example, in the customer category there would be customer demographics (age, gender, income, ethnic, etc.,) customer dwell time, customer preferences, customer journey, buyers versus non-buyers,... Customer dwell time, then, would be compared across different stores and opening hours. No metric exists in a vacuum; as such, they need to be combined and contrasted for interpretations to make sense. So the first step to testing would be to digitize as many relevant parameters as possible. Only then could retailers alter them to see what fits best into their business plans. The most ideal way to do that would be via camera footage - the truthful imprint of a retail store history. Palexy In-store Analytics products have reached great heights in identifying and grouping retail parameters at no additional hardware cost to retailers, taking advantage of existing surveillance cameras. Here are a few case studies with A/B testing done via AI Analytics:

1 - Promotion: A fashion chain ran the same promotion (buy one get one free) at two different locations. Their expectation: the one with the younger customer base would respond to it more enthusiastically, since they had more limited means. Reality: the older, more affluent customers took to it much better. It turned out the price range was still out of most youngsters' reach. After several more tests confirming the hypothesis, the brand introduced a more affordable line targeted towards young people with great success.

Quickly measure the effectiveness of a marketing campaign by comparing traffic during-campaign vs. before-campaign.

2 - Training: A big jewelry retailer wanted to find the best approach to staff training. They divided staff at their nationwide stores into two groups: one group was coached intensively on sales skills, another was encouraged to maximize customer interaction. The results were surprising: it turned out sheer volume of communication actually trumped expertise. From then on, staff training at the company focused more closely on staff proactivity. The brand also made sure to assign adequate staffing to stores so customers never had to wait for help.

3 - Opening hours: During the crisis of Covid-19, retailers were facing the difficult choice of closing their stores. Such one company had an advantage: months ago, they did a split test to see which hours worked best for their main demographics: stay-at-home moms, whose schedule changed little during the pandemic. Instead of closing some of their stores, they simply reduced the opening hours by half to cut staff pay and still maintained their margin.

An example of A/B testing at similar stores in order to fine tune store layouts.

In summary, doing A/B tests for retailers may be challenging, but not impossible. Retailers looking to stay ahead of their peers by adopting a testing mentality may find AI softwares using computer vision like Palexy perfect for the job. For more information, please contact:



bottom of page