Ask for digital twin info?
Debrief
Theories
- More minutes new phones: equipment days, refurbished… Look at the comparative differences between equipment days and minutes of usage. Equip days matters far far more than MOU
- So if we know that equipment is the driver, then new phone is a big deal.
Options for Creating a Business Case
- Predictive model only: Confounds; what does our promotion assume, all the model knows is that if you go to the store, we know churn drops, but we don’t know what moves someone to the store
- Experiment: Only 10% go to the store. And of the control group at 13%, it drops to 11%, which is only 2 of the 5% we predicted.
- Always ask, what does this experiment tell me?
- Option 3: Why is it different to build from experiment 2?
- We learned when we offer you the in-store sign-up, do you actually churn. We learned who goes to the store. Build a model the likelihood of going to the store. I only go after people who are likely to go to the store and not churn.
- You need the data to build the personalization.
His breakdown of the case
- It’s not a predictive or causal model; it’s how you apply it.
- For > 10 employees on vacation, they’re not random assignment.
- Driver = vacation (hunting season); scheduled downtime (reverse causality b/c the downtime drives vacation)
- Double driver of summer + hot; so summer itself can’t increase downtime, but hot can increase downtime.
- Again, the confound is something that could alternatively drive the outcome.
- The hunting example doesn’t pass the smell test.
- If you want to use all your historical data then you must A.) have a process of finding confounds, B.) find those confounds, and C.) then plug those confounds in
- But when you don’t have data for a potential confound, you can’t control for them.
- Also, confounds aren’t yes or no. So less binary and more strength.
Risk-management and storytelling
- Acknowledge the risk of the presentation, how it’s contextualized, and then your plan in light of that risk