r/datascience • u/WhatsTheImpactdotcom • 1d ago
Discussion Applying CUPED to Clustered Experiments
If you've worked on experimentation in tech roles, you've probably heard of CUPED. If you've been in product/marketing data science interviews, you've probably been asked about variance reduction techniques. (Hint: They're typically expecting you to name-drop CUPED.)
Thanks to some folks at Instacart, there are some improvements that can be made from common CUPED techniques when working with clustered experiments (think: marketplace geo-XPs when facing SUTVA violations for user-level testing.) This can help you both on the job and also landing senior/staff-level data science roles.
A common way CUPED can be applied is adjusting each randomized unit by its own pre-treatment outcome mean. However, that pre-period average may not appropriately reflect the experiment population. Instead, the IC economists predicted order-level outcomes during the experiment window using features assumed unaffected by the treatment, and averaged those predictions up to the randomization level.
Using my favorite approach to building personal intuition, I simulated a market-level Airbnb summertime experiment where the XP population has more families; families both have different cancellation rates and target different markets. Standard CUPED using each market's pre-XP cancel rate removed about 28% of the variance, while booking-level CUPED removed about half, even though the booking model itself was weak (AUC around 0.6).
CUPED is very likely to come up in an experimentation or case study round in data science interviews. You should nail the foundations first, check that box. Then, describing recent research like this can help you stand out and get that offer!!
Subtack article and Colab notebook to play along linked here: https://whatstheimpact.com/tutorials/booking-level-cuped/




