Is it Better?

I have had a favorite thought experiment for a long time about how models interact with human decision-making. Suppose you are a realtor trying to figure out how to price a home. You use a tool which gives you an estimate based on your input data, and in general you have learned that the tool underestimates the value a few thousand dollars and compensate. The house estimation software one day could publish an update to be more accurate. Unless the realtor knows about that update though, they may in fact make worse decisions with the improved model than they did before.

This conflict between humans and the data that supports them is hard to navigate. Most businesses say that they want to ensure they are making data driven decisions. However, there are a million gotchas that can cause relying on data to be unhelpful or even counterproductive. Consider the following common situations:

• You may have incorrect data.

• You may not understand the data or have the context you need.

• The data may be hard to work with.

• The data may not be relevant to the business.

When we give people data and simply ask them to make “data driven decisions” a lot of these issues can stay hidden. If the person corrects for the gaps in the analysis and data, it will appear that everything is working well. However, if one day they are given data that is worse than usual because of those uncorrected errors, they may make a really terrible decision. Or if they are replaced by a new person who doesn’t know how to fix the issues.

Compare that situation to how things go when you code a decision model. If you have a bad data set and a model of the decision which assumes good data, you will quickly see a bad decision. If you have a bad model of the decision, you will similarly get a bad output and realize that you need to consider other information. In effect, decision models allow you to validate that you are incorporating all the data you should into that decision. It’s also easy to validate that everything is working as expected when you ask people to review the suggested decisions.

Anyone who says they are working on making data driven decisions should take a long hard look at how they know the data is right both in the numbers and in the way it supports the decision. Prescriptive analytics like what Operations Researchers use is the simplest way to guarantee that you have done things correctly. Even then, there is an important place for human intuition and adjustments. I think of good (validated) data and good (accurate) decision models as the rising tide that lifts all boats.