How decision trees can help you make better product decisions

I recently came across a blog post discussing the use of decision trees in football from Sumer Sports. It is a brilliant post of a 3 part series that I recommend you check out, even if you are not interested in football. I recently used a decision tree to help me make a product decision and was inspired to write about it by the blog post.

---

As product managers, we are always faced with tough choices that can affect the success of our products and the satisfaction of our customers. Sometimes, these choices involve deciding whether to invest in a new feature, a new product, or a new technology. But how can we make sure that we are making the best decision? How can we evaluate the pros and cons of each option? How can we communicate our decision to our teams and stakeholders?

One tool that I have found helpful for making these kinds of decisions is decision trees. Decision trees are diagrams that show the possible outcomes of a decision, based on different scenarios or events. They help us to:

• Simplify complex decisions into smaller and easier steps

• Compare and evaluate different alternatives and their consequences

• Measure the expected value and risk of each alternative

• Choose the best option that maximizes the expected value and minimizes the risk

Decision trees also help us to explain our decisions clearly and transparently to others, as well as to test how our decisions change with different assumptions.

How do decision trees work?

There are different ways to create decision trees, but the basic steps are:

• Define the problem and the goal

• Identify the possible options and outcomes

• Assign probabilities and values to each outcome

• Calculate the expected value of each option

• Choose the option with the highest expected value

Let me show you an example of how I used a decision tree to make a decision about the future of one of my products.

Example: To integrate or build a new product?

One of the products I work on is a data collection app that allows users to collect almost every conceivable data point in a football game. My North Star metric is time to collect one game. We recently developed another app using better technology that could collect location data up to 3x faster than the current offering. But it did not collect all the data points our customers needed. I had three choices:

• Integrate part of the data collection experience with another app and maintain two data collection products (option A)

• Build a new combined data collection experience in one app (option B)

• Do nothing (option C)

Option A would take 3 months to do, while option B would take 9 months. The goal was to reduce the data collection time per game as much as possible. And the sooner we could do this, the sooner we could offer a greater inventory to our customers.

In an ideal world, option B would be the choice, but because timing was important, we had to consider option A, even though it had some gnarly user experiences.

I categorized the possible outcomes into five buckets:

• Options A and B:

  • Reduces collection time by more than 50%

  • Reduces collection time by 26% to 50%,

  • Reduces collection by up to 25%,

  • Does not reduce collection time, and

  • Increases collection time

• Option C: Does not reduce collection time

  

  Decision Tree

Then, I estimated the probabilities of each outcome based on the results of tests using the new application as well as user feedback and expert judgment. Notice that the positive options for option A will produce higher value because we get the value from option A in 3 months vs 9 months for option B:

• Option A:

  • 30% chance of reducing collection time by 50% or more (+15 points),

  • 30% chance of reducing collection time by 26% to 50% (+10 points),

  • 20% chance of reducing collection time by up to 25% (+5 points),

  • 10% chance of not reducing collection time (0 points)

  • 10% chance of increasing collection time (-5 points)

Option A probabilities and expected values

• Option B:

  • 50% chance of reducing collection time by 50% or more (+10 points),

  • 20% chance of reducing collection time by 26% to 50% (+5 points),

  • 20% chance of reducing collection time by up to 25% (+2.5 points),

  • 0% chance of not reducing collection time (0 points)

  • 10% chance of increasing collection time (-5 points)

Option B probabilities and expected values

• Option C:

  • 100% chance of not reducing collection time (0% value)

As you can see, what we are calculating is a weighted average of expected value. Using these probabilities and outcome points, I calculated the expected value of each option by multiplying the probability and point value of each outcome and adding them up:

• Expected value of option A = (30% x 15) + (30% x 10) + (20% x 5) + (10% x 0) + (10% x -5) = 7.5 points

• Expected value of option B = (50% x 10) + (20% x 5) + (20% x 2.5) + (0% x 0) + (10% x -5) = 6 points

• Expected value of option C = (100% x 0) = 0 points

Which option gives the best value?

The option with the highest expected value was option A, as it had the highest potential to reduce the data collection time per game given that we needed to make an impact sooner than 9 months. But it was close! In the product context, we need to make sure that we acknowledge the limitations of the decision tree and supplement it with qualitative feedback as well.

How to use decision trees effectively?

Decision trees are powerful tools for product managers but have some limitations and challenges. Here are some tips on how to use decision trees effectively:

• Use decision trees as a guide, not as a rule. Decision trees can help us to structure our thinking and explore different scenarios, but they cannot capture all the details and complexities of real-world situations.

• Validate and update your assumptions. Decision trees are based on assumptions about probabilities and values, which may not be accurate or reliable. We should validate our assumptions with data and evidence, and update them as new information becomes available. We should also test how our decisions change with different assumptions, and be ready to revise our decisions if needed.

• Communicate your decision process clearly. Decision trees can help us to communicate our decision process clearly and transparently to others, such as engineers, designers, executives, etc. We should explain how we created our decision trees, what assumptions we made, what options we considered, and what trade-offs we faced. We should also ask for feedback and input from others, and incorporate them into our decision trees.

Conclusion

Decision trees are diagrams that show the possible outcomes of a decision, based on different scenarios or events. They help us to simplify complex decisions into smaller steps, compare and evaluate different options, measure expected value and risk, and choose the best option. They also help us to explain our decisions clearly and transparently to others, as well as to test how our decisions change with different assumptions.

Thanks for reading. If you liked this, please share it with others who might benefit from it. And if you have any questions or comments, please leave them below.