More than two years into the COVID-19 pandemic when people everywhere are working hard to summon the energy for another chapter, many questions remain unanswered. For example, what was the source of this pandemic? Will there be new breakthrough variants? And, more importantly, did we use the right strategies to mitigate this pandemic, and have we gained sufficient knowledge to do better next time? 

Evaluating the effectiveness of health policies during a pandemic is critical for informing future efforts. The question we often want to ask is whether the policy can mitigate the impact of the pandemic. For example, did a particular policy reduce transmission of the virus or decrease COVID-related deaths? If we imagine a policy as a switch and the pandemic as a light, it is similar to asking whether the switch can turn off/down the light. 

However, evaluating the impact of health policies is not as simple as flicking the switch, as just because we see changes following the start of a new policy, it does not necessarily mean that policy caused the change. The change could also be due to other unobserved factors that occurred around the same time that the policy was enacted. It is like a power outage happened in the neighborhood at the same time we flicked the switch; it’s hard to conclude whether the switch can control the light even though the light was off after we flicked it. 

In statistics, to answer this type of question we often need to use special study designs or analysis methods, called causal inference, given that regular association analysis could lead to biased results that are confounded by other factors. In the power outage example, flicking the switch is the exposure, to turn off the light is the outcome and the unobserved factors are called confounders. To learn whether the switch can turn off the light, we can apply causal inference.

In health-focused research, randomized controlled trials (RCT) have long been considered the gold standard for causal inference because the randomization, which randomly assigns study units to intervention categories, is believed to break any links between intervention and confounders so that the estimated effect is not impacted by other factors. However, to evaluate the effect of a health policy, particularly a policy to mitigate a pandemic, it is not practical to randomly assign the population to different policies. Therefore, we often try to emulate a RCT to study the effect of health policies. 

The key here is to make or find similar and comparable study units under different policies. Two general types of statistical methods to use are “matching” and “weighting.” 

• What is “matching”? Matching is used to group study units with similar characteristics that may be associated with the outcome by pairing, sub-classification or sub-setting the study units. 
• What is “weighting”? It is to replicate study units based on their characteristics.

In our newly published Health Affairs study, we used a matching method to study the impact of mask mandates on COVID-19 case incidence at the county level between March and October 2020. Since mask mandates are most likely to be enacted when cases are surging, the timing may align with changes in other factors, such as the public starting to reduce social activities and increase personal hygiene practices. Any change in case incidence before and after a mask mandate is put in place could actually be due to (confounded by) those other factors. 

Moreover, a mask mandate may have different effects depending on the characteristics of the community in which it is enacted (e.g., population density, acceptance of mask wearing). Therefore, for every county we included in our study, we used a matching strategy to find a control county that had similar population density, acceptance towards mask wearing, and number of cases, but no mask mandate. We ended up with two groups of counties that were comparable in demographics and pandemic severity, but one group had mask mandates during our study period while the other did not. We then followed the two groups for a few weeks to evaluate the effect of the mandate by comparing the difference between paired counties. 

It’s worth noting that if all the counties had mask mandates at the same time, we would not be able to do such matching. The variation in timing of the enactment of masking mandates at the county level during the early months of the pandemic offered an opportunity to assess this impact via our matching study design. The figure below is a visual representation of our matching procedure.

What we found from this study was that counties that introduced a mask mandate early on in the pandemic experienced lower county-level COVID-19 case incidence in the six weeks following enactment as compared to similar counties without a mandate. How much the mask mandate benefited communities varied across counties and over time, with the strongest effects seen in more crowded communities or those with a reticence toward voluntary masking. We also observed that case incidence declined the most between the third and fourth weeks after the mandate started and the effect of the mandate waned between weeks six and eight in some counties. 

While we detected a benefit of mask mandates, there are questions that remain unanswered since we were not able to identify the exact factors that make a mask mandate beneficial. We can’t say whether other factors—like local enforcement of mandates and individual fatigue over time—modified the effect. We also did not disaggregate the degree to which the benefits of mask mandates were the direct result of more uniform protection with masks in public or the signaling to a community of enhanced risk, which may have led individuals to use other mitigation measures (i.e., social distancing) in the weeks following the start of a mandate.  

In addition to contributing to our scientific understanding of the benefits of a particular health policy, we hope this study informs local leaders’ decisions regarding time-limited mask mandates should we unfortunately experience more periods of high COVID-19 community transmission and strained hospital capacity.