- Observationally equivalent: Similar across some subset of characteristics
- Matching = find twins across sets of variables
- Eliminate the customers that don’t match.
When you have small samples, you don’t want to randomize. Match to create observationally equivalent groups instead.
Once you create those twins and match, then you randomize the matches
The confounding variable of sex
Difference-in-differences (diff-in-diff)
- I got the test group for free. And I’m missing a control group.
- What is a guess, it’s a model. We’re using a model to fill in the control group.
- I have to assume the trend in Region 1 carries to Region 2 and that they covary. Look at longitudinal data to validate or adjust that accordingly. So they can have different absolute values, but proportionally vary together.
Simulation Debrief:
- Think through the possible confounds and then match on the confounds (ask about this)
- Godhart’s law
- Don’t understand the pairing things. Market potentials were higher. Matched on Size of potential revenue, growth rates, and location.
- The confound in this simulation growth. You’re gonna misattribute growth to your investment. How do I control for growth?
- You have to plan if you want to measure. Can’t measure accurately if you don’t have a plan.
- Learn and earn.
- Marketing as buying options
His analysis
- Growth is happening naturally in the market. Base rates matter b/c we want incremental revenues in absolute numbers
- Test group is free (just data); the counterfactual is an educated guess you must make.
- Models always generate an answer
- Confounds are alternative stories that can explain the data
- “What’s the business assumption behind that number? Do you trust it? Why or why not?”
His different analysis approaches
Pre-post comparison and trend analysis
- longitudinal, time analysis growth (how are these two different???)
A/B Comparison
- Why its future growth is different than pre-post: group one and group two start at 8k each and they float at similar; they would both “floating dock”; contemporaneous growht assumption. twin market assumption.
Diff-in-diff
So we’re trying to isolate vector where we can find who will continue accelerating and who needs us to help them.
Use some form of matching in each category: create the largest partners, then w/in try and create matches of those, randomize. By going into the high revenue segment, I then split that into the test and control. The option value of investing.