How Effective is Exposure Notification in the United States?

A: The short answer is: not nearly as effective as it could be. And should be.

A new preprint assesses the effectiveness of WA-Notify, the Exposure Notification Express solution implemented in the State of Washington. While the preprint makes a number of assumptions that exaggerate the extent of its success, it is nevertheless clear that the technology has done some good. 

However, it is possible to do much better: the UK app had a much larger impact on disease transmission. And it is possible to do much better again: the U.K. manuscript contained a sensitivity analysis exploring expected increases in impact given different kinds of improvement.

Inventing then implementing exposure notification technology only after the pandemic began has been like building an airplane while flying it. All exposure notification solutions were rolled out poorly and in a hurry. Each of them operated under different local constraints, and each of them made different decisions on the fly - as a result, they succeeded and failed to different degrees. What is remarkable, given the time pressure, is that they had any success at all in stemming disease transmission. This should be celebrated by all the teams involved.

But we should not be so enthusiastic in declaring victory that we impede further improvement. Doing better still matters. The pandemic isn’t over, and there is still time to improve the technology itself, its marketing, and its integration with other public health processes. Because the technology has been based since its origins on privacy by design, it is safe enough to have great potential beyond this pandemic, whether for endemic disease such as influenza, measles, pertussis and continued COVID-19, or simply as preparedness so that when the next pandemic comes, the airplane will have already been built.

To know how to improve, we first need to use all available data in an honest comparison of what worked better vs. worse. A good place to start is to identify the solutions that worked best at stemming transmission - in my read of the data, these are the U.K. and Germany - and copy best practices from them. That means copying their strategies to get higher adoption by giving other functionality to the app. These include using the app to scan QR codes rather than write down one’s name to enter a venue, using the app to order tests, and to receive back test results more rapidly as a perk from having the app. They also include better models of infection risk under the hood. Both kinds of improvement are beyond EN Express and require a custom app, but that’s the easy part - code can be copied. The hard work is integrating well with test providers and with local public health. But that hard work is worth it. 

In order to compare what worked better vs. worse, and to receive feedback on how well we are doing, we need data. The privacy-by-design nature of exposure notifications makes this hard, but not impossible. We need to follow best practices not just for implementation, but also for assessment. Otherwise, we end up with assessments merely of #downloads/#people and app store reviews, with no data on how many downloads represent active users, how often cases enter codes, whether notifications go to the right people, whether they arrive fast enough etc. The WA-Notify study does better than this, enabled by ENPA metrics (similar to metrics collected in the UK except restricted to the opt-in subset of users), counting hits to a website landing, and to a limited degree also surveys taken from that landing page. Why not follow the UK’s lead to use something like ENPA to privately collect data from all users? Or if data collection is to be restricted to those who opt in, why not use that fact to collect more information than the UK did? Germany has done an amazing job with surveys, we should all copy them in this space. Key insights from German survey data so far include the facts that notified individuals in Germany tend not to quarantine but do get tested, and tend not to call public health but do call their own doctor.

Given how rushed this technology has been, with more time and modest investment we could do dramatically better than the best implementation of the technology available today, both in reducing transmission and in assessing that reduction. How do we get there before the next pandemic? This is a political question as well as a scientific one. The scientist in me winces when victory is declared on hurried and flawed efforts to date, but I can see that a victory narrative might be important for gaining political buy-in to the technology. But if we go that route, it’s important not to take our own spin too seriously, and make sure that the declaration of victory doesn’t mean that we end up so satisfied with results so far that we get stuck there rather than dramatically improve.

If your implementation had a low impact, that's fine. This tech is new. We'll get it wrong at first, and then we'll fix it. But we need to assess success accurately and copy tips and tricks from those who have the greatest true success, not the most exaggerated analyses. We need to improve how we do assessments, as well as how we do implementations. I’m happy to collaborate on this, please reach out to me!