Don’t Use Data-driven Decision Making
It might seem strange for a blog all about optimizing conversion and experimentation to recommend not using data-driven decision making. But the truth is that sometimes what worked well in one isolated experiment might not be right for your overall product or business strategy.
Data Informed Decisions Making
Instead, I always recommend data “informed” decision making. Data-informed decision making incorporates agency and a human element when deciding which experiment variation to roll out. The data “informs” the decisions we make about our product, it doesn’t dictate what we do.
This was something we did a lot at Acorns. Our mission was always to look after the financial best interest of the up-and-coming. This meant that even if the data of one experiment won, the experience should always be viewed through this lens. If you hide pricing information, you will get better conversion at that step. But you’ll also have a lot of pissed off customers who will never trust your product again.
With every experiment and product change, use the data you’ve collected to inform your decision and always put yourself in the users shoes. This is the core of customer centric design. Ask, did this change perform the best because it made the experience more clear or did it out perform the control because it actually hid important information from the user that will come back to bite you later in customer support costs or churn?
Will this come back to bite you later in customer support costs or churn?
Example from Facebook
Data informed decision making can also be used to help inform the type of product you want to be building. An example of this I remember reading about, but can’t find the article now, was from a Designer at Facebook, now Meta. They had designed and tested several variations of notifications on their tab bar icons. The one that got the most engagement displayed the number of notifications in a given section. Based on the data, that seems like the obvious choice to roll out, right?
But, the designers decided that wasn’t the experience they wanted for customers, they felt it would create too much anxiety to see the number of notifications staring at you in the face all the time. So, contrary to the data, they instead made a data informed decision and went with the second best variation, a simple red dot by a tab bar icon that had unread notifications.
Meta has since rolled out the numbered notifications, but they do let you disable these notifications entirely. But most users never change default settings, I'll be writing an article about this soon so be sure to sign up for my newsletter to get that in your inbox when it's ready.
I hope you enjoyed this article and see the benefits of data-informed decisions over data-lead decision making dictating what you do. We are the ones in-control of the products and experiences we build. Use data to inform your decisions, don’t let it dictate what you do and remove your agency!
If you enjoyed this post, please share it with a friend. I also do funnel optimization consulting, if you are interested in this for your product or your company please get in-touch 👉 Funnel Growth Agency.