I used to think: if I master analytics and quantify every design decision in a product, success would be guaranteed — I'd have a clear understanding of where to go next! So, for a couple of years, I studied the topic, took courses, implemented analytics into products, and followed a data-driven decision-making approach.
It turned out that analytics and data are a consequence of the decisions. They are simply a snapshot of the current state of the product, the result of changes that have already happened. In other words, first comes the decision, then the data about it. And the data won’t be perfect: not all of it will make it into a specific analytics system (there will be discrepancies between systems) due to privacy settings; it will be hard to attribute the data to a specific product update & feature; the time window for retention measurement might be too short and inconsistent; or there simply might not be enough data to make a confident decision.
In the end, I believe, the key remains the entrepreneur's vision, an understanding of how the world works, where the industry is headed, insight into what a specific feature brings to the product, and intuition. In this aspect, analytics isn't useful — you will still be in the vacuum of your own worldview.
Of course, analytics can be useful in other aspects. But it's crucial to be 100% sure that you are asking the right question, measuring it correctly, and it's a good time to invest resources in it. From another side: data must be interpreted carefully and consciously, you need to question your conclusions — this is something any good analytics course will tell you.
A/B Tests?
In my experience, founders often say, “Let's do A/B testing and it will become clear”. Conducting A/B tests is a special case of what I described above. They won't help make the right decision in a global sense. Yes, if everything is done correctly (both in framing the question and in the technical implementation of measurement), they can help find a local maximum, but they won't capture the full picture of the product. The product's whole picture (for now) can only be gathered, analyzed, and felt by a human.