A well-intentioned but unproven app could reinforce biases and create confusion and stress, something developers must take more time to consider.

(This article has been translated into French.)

We are all participating in an unprecedented global experiment aimed at figuring out what is the best way to confront the COVID-19 pandemic. And according to the latest data, one well-established strategy seems to be working; the messaging around social distancing seems to be motivating most Canadians in just the right way: we’re flattening the curve.

But, in theory we can do better, which is why governments around the world are considering additional curve-flattening strategies, including a new app that uses people’s cell phone location data to trace COVID-19 cases. But even if the app generates perfect contact-tracing data, there are reasons to expect it might not produce the desired outcomes. In fact, it could conceivably make things worse.

This week we learned that the Quebec and Canadian governments are in discussions with Yoshua Bengio, a leading Canadian artificial intelligence (AI) researcher, to launch an AI-powered COVID-19 contact-tracing app within the week.

Although details are scant, an article describing the app states that it “works as a sort of COVID-19 roadmap,” using Bluetooth to share each user’s anonymized COVID-19 risk profile with other phones within 10 metres, “helping its users navigate around higher-risk people and locations.”

The app, based on voluntary participation, will be aware of the COVID-19 profiles, including infection status of those with whom you come in relatively close contact while out and about. Based on your daily movements, it could send you a notification suggesting more handwashing. It could suggest that you stay at home. It also informs users about “people or places that carry a higher risk of infection.”

Bengio told The Logic that he hopes the app will “allow [us] to focus stronger confinement on the most at-risk people and make it easier for those less at risk to go back to activities outside, work, etc, until they cross paths with high-risk people (which would then tell them to stay home, etc).”

However, there are five problematic and reasonably foreseeable outcomes (in addition to significant privacy concerns) that Canadians and our governments should consider before unleashing an untested and unproven COVID-19 contact-tracing app on the public. These considerations should be weighed against the likely (not simply the hopeful) benefits that a contact-tracing app will deliver in the Canadian context. It’s important to bear in mind that Canada is far different than the other countries in which these apps have been deployed.

COVID-19 contact-tracing apps can reinforce existing social biases, thus stigmatizing locations and communities. Bengio is quick to point out that his app will use only anonymized data to avoid stigmatizing individuals. However, it will provide users with information about high-risk locations.

Even if it doesn’t tag specific locations, it enables users to “triangulate” those locations based on notifications about their daily movements. In effect, this may lead to singling out individuals, or groups of individuals through an imperfect process of inference and elimination. We are seeing how this disease is disproportionately affecting African American communities south of the border. Would this app feed into existing biases by digitally tagging their communities and establishments as “dangerous”?

We are also hearing about how the disease has resulted in various forms of discrimination towards Asian communities here in Canada. Would vague information about “infected” locations further fuel such biases?

There is a good chance that people will over-trust the app to keep them safe, which could inadvertently increase social contact. There is a well-documented psychological effect called “automation bias,” according to which users treat a technology as much more authoritative than it actually is. Good design principles and good ethical principles suggest that we should err on the side of caution here and expect a large number of users to fall victim to automation bias when using this app. Those who do could misinterpret it as a sort of COVID-detector, capable of alerting them to the disease before and after they come into contact with it.

Unfortunately, automation bias could cause some users, those who are falsely confident that the app is looking out for them, to lower their guard when it comes to social-distancing practices. It would be a double whammy if this effect were unevenly distributed throughout society. For example, it could have a greater impact on users who have a harder time interpreting the app’s design or have trouble using it.

Notifications could inadvertently overload certain aspects of the healthcare system. Without rigorous real-world testing, it’s hard to know with certainty how people will interpret, and respond to, the notifications they get from this app.

If the surprising level of confusion that has erupted around public health messaging in the past weeks is any indicator, it is entirely possible that these app-based notifications could trigger increased confusion and stress among individuals who are unsure of what to make of them. That could translate into a sudden increase in unnecessary phone calls to telehealth or public health, or worse, unnecessary visits to healthcare facilities. At the very least, healthcare providers should be prepared for this kind of response.

A COVID-19 contact-tracing app could just do psychological harm to its users. Following the previous concern and recalling that this app is largely untested and unproven, the only effect it might have is to stress people out. That would be a shame, but more importantly an increase in general anxiety levels could trigger an increase in related harms such as domestic violence, depression and suicide.

COVID-19 app notifications could contribute to desensitizing users to those and other COVID-19 public health messaging. Many of us have experienced notification overload – the negative effect that too many notifications can have on our lives.

But a recent study in the Netherlands suggests notifications could actually dull the motivational link between the notification and the actions the notification is requesting you to perform. In other words, a notification to wash your hands might actually make it less likely that you’ll wash your hands in a timely manner.

In addition, there is anecdotal evidence that the app the South Korean government is using seems to be contributing to a desensitizing effect, causing people to tune out public health messaging. Again, these effects could actually contribute to an increase in infection rates.

It’s important to acknowledge that although it seems the messaging around social distancing is working, we don’t know exactly why it’s working. It could be that people are motivated by concerns about getting the disease, or spreading the disease. They might also be motivated by the thought of doing their civic duty, of taking one for Team Canada. It could be some combination of all of those factors.

Without proper testing, we also don’t know what effect contact-tracing apps will have on what might turn out to be a delicate balance of motivating factors. Of course, we need not abandon our search for good technological solutions to this pandemic.

But we should proceed responsibly. Just as it would be dangerous to rush an untested vaccine into production, unproven contact-tracing apps, well-intentioned as they may be, won’t necessarily make things better. By requiring that app developers take these five considerations into account when designing the technology, a COVID-19 contact-tracing app will more likely work to our benefit.

This article is part of the The Coronavirus Pandemic: Canada’s Response special feature.

Photo: Shutterstock, by Connect world.