← List all talks ...

Interference in Experimental Design in Online Platforms

Ramesh Johari – Professor, Stanford University

Thu, 19-Nov-2020 / 4:30pm / Zoom: https://stanford.zoom.us/meeting/register/tJckfuCurzkvEtKKOBvDCrPv3McapgP6HygJ

Talk

• Video

Abstract

Many experiments (“A/B tests”) in online platforms exhibit interference, where an intervention applied to one market participant influences the behavior of another participant. This interference can lead to biased estimates of the treatment effect of the intervention.

In this talk we first focus on such experiments in two-sided platforms where “customers” book “listings”. We develop a stochastic market model and associated mean field limit to capture dynamics in such experiments, and use our model to investigate how the performance of different designs and estimators is affected by marketplace interference effects. Platforms typically use two common experimental designs: demand-side (“customer”) randomization (CR) and supply-side (“listing”) randomization (LR), along with their associated estimators. We show that good experimental design depends on market balance: in highly demand-constrained markets, CR is unbiased, while LR is biased; conversely, in highly supply-constrained markets, LR is unbiased, while CR is biased. We also introduce and study a novel experimental design based on two-sided randomization (TSR) where both customers and listings are randomized to treatment and control. We show that appropriate choices of TSR designs can be unbiased in both extremes of market balance, while yielding relatively low bias in intermediate regimes of market balance. (This is based on joint work with Hannah Li, Inessa Liskovich, and Gabriel Weintraub.)

Time permitting, we will conclude the talk with some discussion of other experimental designs used by online platforms to address interference, including adaptive designs such as switchback experiments, and clustered experimental designs. The goal is to provide an overview of some of the open challenges that arise in this domain. (This part of the talk is based in part on joint work with Peter Glynn and Mohammad Rasouli.)

Bio

Ramesh Johari is a Professor at Stanford University, with a full-time appointment in the Department of Management Science and Engineering (MS&E), and courtesy appointments in the Departments of Computer Science (CS) and Electrical Engineering (EE). He is a member of the Operations Research group and the Social Algorithms Lab (SOAL) in MS&E, the Information Systems Laboratory in EE, and the Institute for Computational and Mathematical Engineering. He is also an Associate Director of Stanford Data Science. He received an A.B. in Mathematics from Harvard, a Certificate of Advanced Study in Mathematics from Cambridge, and a Ph.D. in Electrical Engineering and Computer Science from MIT. He served as co-chair of the ACM Economics and Computation (EC) program committee in 2019, and he is an Area Co-Editor of the Revenue Management and Market Analytics Area for Operations Research, and associate editor for Management Science (in the Stochastic Models and Simulation area) and Stochastic Systems. His research interests are in online platform and marketplace design, experimentation and data science for online platforms, and (more recently) application of these techniques to personalized health care via telemedicine.