mrb's blog

Evidence of the effectiveness of stay-at-home orders in the US

Keywords: epidemiology covid-19 coronavirus lockdown

A semi-popular Twitter thread by @boriquagato argues that stay-at-home orders are ineffective at reducing mortality in the US: “there is just no evidence that this incredibly expensive and harmful policy has any effect at all.

This is, of course, preposterous. @boriquagato attempted to rank US states by cumulative deaths per capita, and observed how the ranking changed over ~4 weeks. I will explain the flaws in their application of this methodology. But beforehand, I will present a superior methodology: examining the trend of daily deaths per capita.

Trend of daily deaths

When analyzing daily deaths, be aware there is always a lag between the moment a stay-at-home order goes into effect and the moment a reduction of daily deaths is observed, for many reasons:

  • delay between infection and death
  • stay-at-home orders may be initially advisory or poorly enforced, and eventually made mandatory or more strictly enforced
  • infected persons staying at home may infect other household members over time
  • etc

The lag is approximately 2-4 weeks. So I charted daily deaths per million inhabitants for each US state, marked the date of stay-at-home orders with a dashed line, and overlaid a shaded area 2-4 weeks later:

Daily COVID-19 deaths per million, for each US state

In 33 states—plus DC—that implemented stay-at-home orders, we expect to see and do see daily deaths either peaking or reaching a plateau at some point in the shaded area: Alaska (8 deaths,) California, Colorado, Connecticut, District of Columbia, Florida, Georgia, Hawaii (17 deaths,) Idaho, Kansas, Kentucky, Louisiana, Maine, Maryland, Massachusetts, Michigan, Missouri, Montana (16 deaths,) Nevada, New Jersey, New York, North Carolina, Ohio, Oklahoma, Oregon, Pennsylvania, South Carolina, Tennessee, Texas, Vermont (53 deaths,) Virginia, Washington, West Virginia, and Wisconsin. Some states, as noted in parentheses, have recorded very few deaths so the curve exhibits random noise in the form of peaks and valleys.

In 3 states that implemented stay-at-home orders, we see the curve is neither peaking nor reaching a plateau around the expected time. However the sources of these anomalies are easily identified, and once corrected the curve does peak or reach a plateau when expected:

  1. Arizona: the spike in early May is caused by a delay in reporting deaths from weeks prior Other than this artifact, Arizona has reached the plateau when expected.
  2. Indiana: the spike around late April is due to a reporting artifact in my data source for COVID-19 deaths. The state’s official COVID-19 dashboard shows deaths peaking on 22 April, as expected (between 08 April and 22 April): Indiana deaths by day
  3. Mississippi: the spike around early May is, again, caused by a reporting artifact. The state’s official COVID-19 dashboard shows deaths peaking on or about 27 April, as expected (between 17 April and 01 May): Mississippi deaths by day

In 4 states that implemented stay-at-home orders, we see anomalies with no obvious explanation:

  1. Alabama: daily deaths appear to continue to rise past the shaded area, however there is not enough data to tell if the trend is significant or if it will reach a plateau.
  2. Delaware reached a plateau around 23 April, which is 31 days after the stay-at-home order, beyond the expected delay of 2-4 weeks. The state borders Maryland who implemented a stay-at-home 7 days after Delaware did. Delaware is a geographically small state, so one possible but unverified explanation could be people commuting across the state line who continued to import new cases into Delaware for longer than expected.
  3. Illinois: daily death growth slows down in the shaded area, but not significantly enough to plateau. It is unclear why. Most deaths occur in the Chicago metropolitan area. Lack of enforcement in the city?
  4. New Mexico: a cursory search did not reveal any obvious reason why deaths are still climbing. Perhaps the state’s “grossly substandard nursing care” is to blame. 45% of COVID-19 deaths in the state occurred in long-term care facilities.

In 3 states that implemented stay-at-home orders, daily deaths continue to increase past the shaded area. However these states are the top 3 states in the nation with the highest shares of long-term care facility deaths according to Kaiser Family Foundation (Minnesota: 81%, Rhode Island: 77%, New Hampshire: 77%) well above the nationwide average of one-third. So daily death growth in these states does not reflect a failure of the stay-at-home policy, but a failure of keeping their long-term care facilities safe:

  1. Minnesota
  2. New Hampshire
  3. Rhode Island

In 7 states that never issued stay-at-home orders, deaths generally follow an upward trend, with no signs of stopping. However these states are so small and have so few deaths that their curves are very noisy (except the largest state, Iowa):

  1. Arkansas (98 deaths)
  2. Iowa (318 deaths)
  3. Nebraska (118 deaths)
  4. North Dakota (40 deaths)
  5. South Dakota (43 deaths)
  6. Utah (76 deaths)
  7. Wyoming (7 deaths: the curve is pure random noise)

Trend of daily cases

We can produce the same charts with cases instead of deaths. I expect the lag between stay-at-home orders and daily cases peaking or reaching a plateau to be 2-15 days, represented by the shaded area:

Daily COVID-19 cases per million, for each US state

For many states, we do indeed observe daily cases peaking or reaching a plateau 2-15 days after the stay-at-home order.

However daily cases sometimes continue to increase beyond the shaded area. A likely explanation is that states are increasing the daily test rate over time, which improves the case ascertainment rate. For this reason, charts of daily deaths in the previous section are a more reliale indicator of the effectiveness of stay-at-home orders.

Flaws in @boriquagato’s methodology

@boriquagato ranks US states by cumulative deaths per capita and observes how the ranking changes over approximately 4 weeks, between 11 April and 08 May. Blue represents states that implemented stay-at-home orders, and red those that did not:

@boriquagato's states ranking

Firstly, the scatterplot is bogus. I recreated it using census.gov 2019 population data and the New York Times COVID-19 data repository and I ended up with a significantly different scatterplot. Also, trying to place a line of best fit on this scatterplot is pointless as it simply approximates y=x when the time span is as short as 4 weeks. Among the states without stay-at-home orders, according to @boriquagato’s version 4 of the 7 states saw their rank worsen, but in reality it happened to 6 of the 7 states. This is evidence stay-at-home orders are effective at reducing mortality:

Corrected ranking

Secondly, @boriquagato selected a premature time period that failed to fully capture the effectiveness of stay-at-home orders. For example look at the District of Columbia:

Daily COVID-19 deaths per million for D.C.

In the case of DC the stay-at-home order was clearly effective because daily death growth peaked then started declining. However the time period 11 April (red dashes) through 08 May (blue dashes) is too early to capture the decline in full; it should have been shifted by minimum ~9 days. So I recreated @boriquagato’s scatterplot over 20 April–15 May [Update 2020-06-20: I updated the chart to cover the longer time period 20 April–20 June]:

Better ranking

On average stay-at-home states gained 1.2 ranks, while states without the policy lost 7.7 ranks. In fact, every red states moved up above the diagonal. Can we evaluate the statistical significance of this? I assumed the null hypothesis that these states did not lose ranks, and computed a p-value of 0.0047! This is statistically significant evidence that the lack of a stay-at-home policy is strongly correlated with increased deaths.

However a scatterplot of state rankings is a crude tool to validate the effectiveness of stay-at-home orders. A more refined visualization is to produce the same scatterplot but with the axes reflecting deaths/million instead of rankings:

Plotting deaths per million

Again, no surprises here: the states witout stay-at-home orders deviate upward from the linear regression curve (dashed line.) Deaths in states without stay-at-home orders universally increased faster than the national average.

This chart is a different representation of the same data as the scatterplot above, showing deaths/capita evolving through time:

Cumulative COVID-19 deaths per million, for each US state

Summary

The scatterplot created by @boriquagato is incorrect, probably because of errors in their (undocumented) data sources or data input. A corrected scatterplot shows that states not implementing such orders fared noticeably worse than states that did. This is evidence that stay-at-home orders are effective.

Nonetheless, a scatterplot of states ranked by cumulative deaths per capita remains an inferior tool to validate the effectiveness of stay-at-home orders, because of the huge 100-fold difference in death rates, even before states started implementing stay-at-home orders.

A superior methodology is to look at the trend of daily deaths per capita. And indeed, among the 43 states—plus DC—that implemented stay-at-home orders, 36 states and DC (84%) demonstrate a decrease in daily death growth correlated with the timing of the order, while 4 states exhibit unexplained anomalous trends, and in 3 states unexpected daily death growth can be attributed to external factors (deaths in long-term care facilities.) Conversely, the 7 states that did not implement stay-at-home orders exhibit daily death growth that persists to this present day. This is further evidence that stay-at-home orders are effective.

This analysis could be improved if states reported deaths by date of death, like Indiana and Mississippi. Most states do not provide such data, which creates anomalies in daily death curves, as states sometimes report unreported deaths from weeks prior.

Another improvement would be possible if states provided separate statistics for deaths in long-term care facilities. Deaths in such facilities account for a significant number of total deaths, and are largely uncorrelated with whether or not a state implemented a stay-at-home order.

Finally, the findings in this post are largely supported by third-party research showing that social distancing does reduce the daily growth rate of the epidemic:

Responses from @boriquagato

@boriquagato made false counterclaims, did not understand my data, largely ignored my replies, and abruptly withdrew from the discussion by blocking me on Twitter:

  1. He finally provided the data source for his scatterplot, and as I suspected his data is bogus. He used data from 2020-04-17 instead of 2020-04-11. He never replied.
  2. He claimed 3 of 7 states not implementing stay-at-home orders improved their rank, but only 1 of 7 did. And again his data is bogus.
  3. He wrongly claimed states not implementing lockdowns changed by 1 position in the scatterplot, but in reality the mean rank move is -5 and statistically significant.
  4. He thought my analysis of daily deaths per capita was based on raw daily death figures, not realizing I compute a 7-day average to smooth out the spikes (the “date-of-report vs date-of-death” issue.)
  5. He mistakenly thought daily deaths were supposed to decline in the entirety of the shaded areas. I re-explained they either peak or reach a plateau, as expected. I marked the exact location of peaks and plateaus. He never replied.
Comments

pb wrote: "or reaching a plateau" is doing a LOT of work! 27 May 2020 22:59 UTC

BS wrote: This is such a BS analysis; almost states with stricter orders had much higher overall deaths per capita; you are basically claiming that getting hit really hard and peaking out is somehow a success over much milder statistics spread out over a longer period. This is not even science; just lies. 23 Jul 2020 16:31 UTC

mrb wrote: BS: you have got cause and effect mixed up. The states enacted lockdowns *because* they saw deaths climbing the highest.

And by the time they enacted lockdowns, it was already too late (you can't resuscitate the dead)... For example NY locked down on 3/22; deaths lag cases by 3 weeks so daily deaths peak on 4/10; and they go down quickly after that (thanks to the lockdown); so another 2 and a half weeks later on 4/27 they are cut in half; but by that time it's already too late as NY is already at 1100 deaths per million inhabitants.

You say this isn't science, and yet it is: my Summary links to 9 research papers confirming lockdowns work.
28 Jul 2020 18:03 UTC

Mike wrote: I think your analysis is highly flawed as it depends on states accurately reporting “COVID” deaths.
As is clear now a lot of states have been reporting anyone with a cough or any heart disease as a “COVID” death, often without any tests. Even with tests there is a big difference between “death with COVID” and death from “COVID”.
I would suggest you re-do your analysis using all deaths in the particular states, and then about a month into lockdown you would have to start taking away excess deaths due to suicides, cancer etc, which all went up due to people losing their jobs, not being able to get their surgeries etc.

Best Regards
Mike
18 Sep 2020 02:15 UTC

mrb wrote: Mike: ah yes, dying "with" covid instead of dying "of" covid. It is a popular idea being spread on Twitter and other social media. I have heard it a hundred times.

Unfortunately it is bollocks.

US CDC excess deaths since 01 Feb are already at 202k-263k (https://www.cdc.gov/nchs/nvss/vsrr/covid19/excess_deaths.htm#dashboard) So something extremely abnormal never seen in decades is happening in 2020 (hint: it's covid)

And death certificates tell us 92% of covid deaths are cases where covid is the (single) underlying cause of death. IOW they die "of" it, not "with" it: https://twitter.com/zorinaq/status/1291594000437145600

So, yeah, bollocks.
22 Sep 2020 20:26 UTC