What are the Inputs to a Decision?

A few weeks ago I posted about the nature of “data driven decisions” and how there is a hierarchy of how much the data actually drives the decision. Or more specifically, in my view you can only really call something data driven if the data is in fact driving. I.e., you have a model that goes from data as an input to decision as the output. Otherwise, at least from the outside, it is impossible to tell the difference between data being a key input to the decision or being entirely ignored in the decision-making process.

However, when I have brought this idea up, people sometimes point out that not everything needs to be that level of data driven. Which is true! In a theoretical proof kind of way, matters of opinion should definitely not be based on data. But in a much broader way most decisions are too low-stakes to really be worth building a complete data-driven model anyway.

I think what I have had trouble communicating is the difference between the following two ideas. My technical definition:

data driven should mean the data drives

and the hype:

decisions are better when driven by data.

The question, and I think the gap between these two ideas, is if you don’t have a data driven decision, how do you know if the decision you are making is good? It feels very natural to assume that data driven decisions are better than the alternative.

As long as we keep a wishy-washy definition of data driven, the conflict is not obvious. People talk about the improvements possible if only they had better data. When I work with a client, because of the investment good analytics usually takes, I ask them to work through a thought experiment with me. What different actions might they take depending on what they knew? When working through the exercise I also use a dash of skepticism to interpret the things they wish they knew and what they would do about it. If I knew what the lottery numbers were going to be, I would buy the winning ticket. But is it realistic that I could have that information before the drawing?

More importantly, I try to understand how the data would inform the decision. Part of the inspiration for this article is a post on LinkedIn from Elena Verna. She challenges her readers to identify important metrics that are drivers of success and that leadership will review weekly. I don’t disagree with her on the value of metrics. But the way she describes using them has more to do with “model thinking” rather than data driven decision making.

The reason tracking KPIs can be mistaken for data driven decision making is that we have a hypothesized causal model from the driver to the outcome. We expect that if we have more customers, we’ll have more sales dollars. If we have less customers this week than last week, our mental model says “there’s nothing stopping us from having more sales dollars except whatever we did wrong this week that we did right last week. Therefore, we have reason to believe we can increase the number of customers which will increase sales dollars.”

But should that chain of reasoning count as data driven? In my opinion, no. Unless we have some sort of experiment, we don’t have data that tells us more customers means more sales dollars. We just have a model of the world that tells us more customers should help if we want more sales. And that should be enough! This is why to me it makes sense to be really strict in our definition of “data driven” without believing that they are necessary for every decision.

To summarize, data is not the only input into a decision, and frequently it is not even needed. A model of how the world works can be all you need to make good decisions. Once you have an actual tradeoff to consider (how much does each extra customer cost to acquire compared to the additional sales?), you typically do need some kind of data driven model. Quantifying one factor versus another is often an incorrect guess unless you have measurements to back them up.

And this is why I like keeping the definition of data-driven technical. If we are clear that the right decision will only come from analyzing a tradeoff, we need data driven. If we just need to be in the ballpark of right, that level of sophistication is overkill and we should ignore the hype around data driven decision making.