The federal government is wading into the area of predictive analytics to try to help build more effective programs and policies before they are approved and funded.
The Chicago-based data firm Mission Measurement, home of the Impact Genome Project, has already received $324,000 in federal contracts, and is currently working with federal departments to improve and better track the outcomes of Canada’s Youth Employment Strategy.
The pilot project is the latest effort being run out of the Privy Council Office’s Impact and Innovation Unit, which is tasked with seeking out cutting-edge tools to tackle some of Canada’s biggest policy challenges. The unit, which swallowed PCO’s former Innovation Hub, is also in charge of overseeing the Liberal government’s management approach to seeing policy promises to fruition, called deliverology. That has received mixed reviews inside the public service.
Jason Saul, the CEO of Mission Measurement, said the Impact Genome is a new tool that could change policy-making by shifting the mindset from “spending money to buying outcomes.” It’s aimed at getting evidence on which kinds of programs work into the hands of policy-makers when they are designing new policies rather than waiting until they are evaluated years later.
“The ultimate question about government is whether it works or not, and these are the kinds of tools we need to answer those questions,” said Saul in an interview.
A new kind of genome
The Impact Genome is essentially an evidence database. Saul likens it to the human genome, which can help scientists predict health outcomes by allowing them to look at the common genes among people. He developed this social policy genome when he teamed up with musicologist Nolan Gasser from Stanford University, who created Pandora Media’s Music Genome Project. This is a predictive model that is based on hundreds of different musical attributes, such as time, tempo, timbre and instruments. Listeners pick songs they like and Pandora will suggest other songs for them based on the same attributes.
The Impact Genome pulls together and analyzes existing evidence-based research on social programs, such as academic studies, evaluations and impact reports, and zeroes in on the particular “genes” that are common attributes of programs that led to positive social outcomes. Those could be such things as frequency or duration of policy intervention, as well as design features like the use of mentors, financial incentives or hands-on learning techniques.
The meta-analysis of all of these data points helps create standards and benchmarks, and can help extrapolate the likelihood of successful outcomes for a program or project. The project developed various separate genomes targeted at the social impact and outcomes policy-makers are seeking in areas ranging from arts, culture and criminal justice to education, health, economic development and the environment.
Employment and Social Development Canada (ESDC) is leading the genome pilot inside the federal government, starting with youth employment; if it is successful, the initiative could be expanded to other programs. Privy Council Office officials say ESDC and the 10 departments that oversee funding for the various youth employment programs are using the genome to better understand what has worked or not when designing programs.
The government spends about $330 million a year to help youth, aged 15 to 30, to gain skills and work experience for the job market. Last year, it boosted that amount by an additional $450 million over three years.
The ESDC pilot is using Mission Measurement’s workforce development genome, which has bundled and analyzed existing research on the kinds of programs that have resulted in outcomes such as job readiness, re-entry into the workforce and career advancement. A number of Canadian research studies are also being coded and added to the project’s existing inventory, which will be factored into the analysis for the pilot.
Saul said the ESDC pilot is in the early stages, and the data it generates could help develop algorithms and other predictive models down the road.
The first government to use the genome
The Impact Genome Project has so far catalogued more than 10,000 research papers and evaluation studies on social programs, from around the world. Mission Measurement has advised corporations from McDonald’s, Walmart and Starbucks to RBC and Telus. It has also used its methodology with charities and philanthropic organizations, such as Canada’s Peter Gilgan Foundation and the Kielburgers’ We Charity to measure their social impact.
The inner workings of government
Keep track of who’s doing what to get federal policy made. In The Functionary.
The Functionary
Our newsletter about the public service.
Nominated for a Digital Publishing Award.
Mission Measurement has provided some advice to the Ontario government on its poverty reduction strategy. The federal pilot, however, is the first time a government has used the Impact Genome, and the experience is being closely watched by other countries.
Saul singled out Canada’s efforts to measure the social impact of its policies in a TEDx Talk in Chicago last year on how to restore citizens’ trust in their governments. “The dream is that we can use science to produce a better bang for the buck and make government more effective, and not just for Canada, but for governments all over the world,” he said.
“It’s my belief, first, we can start predicting instead of guessing; second, we might create twice the impact for half the costs, and third, someday we can bend the arc of trust and restore faith in our democracy.”
Potential weaknesses?
The PCO says that if the genome project delivers, money would be better spent and reporting would be streamlined. “While not an easy task, by identifying what works, what doesn’t, for whom and why, we can make wiser investment decisions that can have the greatest impact on Canadian citizens,” said PCO spokesman Paul Duchesne in an email.
Still, the use of predictive analytics – and eventually artificial intelligence – in policy-making is a controversial area. Critics say the approach is too simplistic because context, along with other factors and conditions beyond data, can affect a policy’s outcomes: a youth program that works in New York may not work in an Indigenous community.
“It is not perfect, for sure,” said Saul. “We can’t say with 100 percent certainty whether something is going to work, but we can provide a better estimation of success than guessing.”
If the project ultimately moves toward the use of algorithms to analyze data and make predictions, there are likely to be a host of questions about the transparency of the algorithms used, and whether there are any biases embedded in them. The Canadian government last year sought advice on using AI for immigration system purposes. This prospect has set off alarm bells for some experts, who point out that the impact of errors from the use of such algorithms can be life-changing; they call for more oversight around the use of this technology.
Meanwhile, some evaluators also worry governments want to replace the rigour of experiments and evaluations with impact measurement. Evaluations are considered the heart of the empirical evidence that governments need in order to determine which programs work, but they are retrospective and expensive and take a long time to complete.
But Saul said that for the genome project to work, it needs more and higher-quality evaluations. In fact, PCO officials and Saul have been explaining these tools at the annual conference of the Canadian Evaluators Society in Halifax this week.
The PCO argues the way programs are measured has to evolve, especially as Canadians increasingly demand to know what they are getting for their investments. Said Duchesne: “A lot of innovative and unique work is happening on the ground and we must find ways to tap into it so that government can be a more effective partner.”
Photo:Â Shutterstock.By Anastacia Trapeznikova
Do you have something to say about the article you just read? Be part of the Policy Options discussion, and send in your own submission. Here is a link on how to do it. | Souhaitez-vous réagir à cet article ? Joignez-vous aux débats d’Options politiques et soumettez-nous votre texte en suivant ces directives.
