Data is important when making changes to your Shopify store

The past week I've been testing our veggie seeds to see how well they germinate. All of the types had 80-90% germination within 2-6 days which is much faster and a higher percentage that many store-bought seeds.

We can conclude that the seeds were grown well and will be quality seeds to plant later.

The testing collected the data to support the conclusion that the seed are viable. It just took about a week and a few minutes of work.

Similarly I was helping someone who made a bunch of changes to their website and then noticed they had a significant drop in sales a couple of weeks later. Digging into it, their rankings in a specific search engine changed.

Since they made a bunch of changes at once, we don't know what caused the drop. Was it the first change, the second, the third...?

We've started to go back and undo the changes and make one at a time to see what impacts it has on their rankings.

Was the first change the cause of the drop? Was the second? ...

Or did something else in the search engine change and this drop in sales and rankings is the new rankings?

That website didn't test their changes as they went so now we have to go back to collect the data to find the cause.

An opposite situation to the seed testing.

Luckily the search engine is quick at updating rankings (unlike Google which can take months and by that time, there's been a dozen more algorithm updates...)

Sometimes full statistical testing (e.g. A/B testing) is impossible or improbable. For example, the majority of Shopify stores don't have enough traffic to A/B test their homepages, let alone their product pages or their best-selling product page.

When you don't have a lot of traffic you have to look at other ways to test but there's almost always some way to get a result. You might never know for sure, but you should be able to get it close enough to feel comfortable accepting a change.

With longer-term data you might need help keeping track of everything. For example, if you're seeing how a change influences customer behavior you'll likely need to compare months of behavior at once. The Cohort reports in Repeat Customer Insights are designed to help analyze that sort of change.

There's always external influences (e.g. seasonality) so you won't be perfectly sure even with as detailed of an analysis as the Cohort Report, but it can help boost the confidence that the impact is from your change.

Testing is hard, but it can pay off.

Eric Davis