Spillover effects in experiments

In experiments, a spillover is an indirect effect on a subject not directly treated by the experiment. These effects are useful for policy analysis but complicate the statistical analysis of experiments.

Analysis of spillover effects involves relaxing the non-interference assumption, or SUTVA (Stable Unit Treatment Value Assumption). This assumption requires that subject i's revelation of its potential outcomes depends only on that subject i's own treatment status, and is unaffected by another subject j's treatment status. In ordinary settings where the researcher seeks to estimate the average treatment effect ( ${\widehat {ATE}}$ ), violation of the non-interference assumption means that traditional estimators for the ATE, such as difference-in-means, may be biased. However, there are many real-world instances where a unit's revelation of potential outcomes depend on another unit's treatment assignment, and analyzing these effects may be just as important as analyzing the direct effect of treatment.

One solution to this problem is to redefine the causal estimand of interest by redefining a subject's potential outcomes in terms of one's own treatment status and related subjects' treatment status. The researcher can then analyze various estimands of interest separately. One important assumption here is that this process captures all patterns of spillovers, and that there are no unmodeled spillovers remaining (ex. spillovers occur within a two-person household but not beyond).

Once the potential outcomes are redefined, the rest of the statistical analysis involves modeling the probabilities of being exposed to treatment given some schedule of treatment assignment, and using inverse probability weighting (IPW) to produce unbiased (or asymptotically unbiased) estimates of the estimand of interest.

Examples of spillover effects

Spillover effects can occur in a variety of different ways. Common applications include the analysis of social network spillovers and geographic spillovers. Examples include the following:

Communication: An intervention that conveys information about a technology or product can influence the take-up decisions of others in their network if it diffuses beyond the initial user.^[1]
Competition: Job placement assistance for young job seekers may influence the job market prospects of individuals who did not receive the training but are competing for the same jobs.^[2]
Contagion: Receiving deworming drugs can decrease other's likelihood of contracting the disease.^[3]
Deterrence: Information about government audits in specific municipalities can spread to nearby municipalities.^[4]
Displacement: A hotspot policing intervention that increases policing presence on a given street can lead to the displacement of crime onto nearby untreated streets.^[5]
Reallocation of resources: A hotspot policing intervention that increases policing presence on a given street can decrease police presence on nearby streets.
Social comparison: A program that randomizes individuals to receive a voucher to move to a new neighborhood can additionally influence the control group's beliefs about their housing conditions.^[6]

In such examples, treatment in a randomized-control trial can have a direct effect on those who receive the intervention and also a spillover effect on those who were not directly treated.

Statistical issues

Estimating spillover effects in experiments introduces three statistical issues that researchers must take into account.

Relaxing the non-interference assumption

One key assumption for unbiased inference is the non-interference assumption, which posits that an individual's potential outcomes are only revealed by their own treatment assignment and not the treatment assignment of others.^[7] This assumption has also been called the Individualistic Treatment Response^[8] or the stable unit treatment value assumption. Non-interference is violated when subjects can communicate with each other about their treatments, decisions, or experiences, thereby influencing each other's potential outcomes. If the non-interference assumption does not hold, units no longer have just two potential outcomes (treated and control), but a variety of other potential outcomes that depend on other units’ treatment assignments,^[9] which complicates the estimation of the average treatment effect.

Estimating spillover effects requires relaxing the non-interference assumption. This is because a unit's outcomes depend not only on its treatment assignment but also on the treatment assignment of its neighbors. The researcher must posit a set of potential outcomes that limit the type of interference. As an example, consider an experiment that sends out political information to undergraduate students to increase their political participation. If the study population consists of all students living with a roommate in a college dormitory, one can imagine four sets of potential outcomes, depending on whether the student or their partner received the information (assume no spillover outside of each two-person room):

Y_0,0 refers to an individual's potential outcomes when they are not treated (0) and neither was their roommate (0).
Y_0,1 refers to an individual's potential outcome when they are not treated (0) but their roommate was treated (1).
Y_1,0 refers to an individual's potential outcome when they are treated (1) but their roommate was not treated (0).
Y_1,1 refers to an individual's potential outcome when they are treated (1) and their roommate was treated (1).

Now an individual's outcomes are influenced by both whether they received the treatment and whether their roommate received the treatment. We can estimate one type of spillover effect by looking at how one's outcomes change depending on whether their roommate received the treatment or not, given the individual did not receive treatment directly. This would be captured by the difference Y_0,1- Y_0,0. Similarly, we can measure how ones’ outcomes change depending on their roommate's treatment status, when the individual themselves are treated. This amounts to taking the difference Y_1,1- Y_1,0.

While researchers typically embrace experiments because they require less demanding assumptions, spillovers can be “unlimited in extent and impossible to specify in form.” ^[10] The researcher must make specific assumptions about which types of spillovers are operative. One can relax the non-interference assumption in various ways depending on how spillovers are thought to occur in a given setting. One way to model spillover effects is a binary indicator for whether an immediate neighbor was also treated, as in the example above. One can also posit spillover effects that depend on the number of immediate neighbors that were also treated, also known as k-level effects.^[11]

Exposure mappings

The next step after redefining the causal estimand of interest is to characterize the probability of spillover exposure for each subject in the analysis, given some vector of treatment assignment. Aronow and Samii (2017)^[12] present a method for obtaining a matrix of exposure probabilities for each unit in the analysis.

First, define a diagonal matrix with a vector of treatment assignment probabilities $\mathbf {P} =\operatorname {diag} \left(p_{\mathbf {z} _{1}},p_{\mathbf {z} _{2}},\dots ,p_{\mathbf {z} _{|\Omega |}}\right).$

Second, define an indicator matrix $\mathbf {I}$ of whether the unit is exposed to spillover or not. This is done by using an adjacency matrix as shown on the right, where information regarding a network can be transformed into an indicator matrix. This resulting indicator matrix will contain values of $d_{k}$ , the realized values of a random binary variable $D_{i}=f\left(\mathbf {Z} ,\theta _{i}\right)$ , indicating whether that unit has been exposed to spillover or not.

Third, obtain the sandwich product $\mathbf {I} _{k}\mathbf {P} \mathbf {I} _{k}^{\prime }$ , an N × N matrix which contains two elements: the individual probability of exposure $\pi _{i}\left(d_{k}\right)$ on the diagonal, and the joint exposure probabilities $\pi _{ij}\left(d_{k}\right)$ on the off diagonals: