Many of us would like to see a world in which social policies comport with our best understanding of science. Is this possible? Professor Patrick Fafard is skeptical.

the idea is to select among candidate policies or programs in a rational manner using the scientific method. Sometimes this view goes by the name of evidence-based policy making.

In response, my first inclination is to discount if not dismiss the argument by simply asserting that in real life things are not so simple and grumble about the hubris of scientists.

He argues that

policy making should not be reduced to rational problem solving.

and that

most of the truly interesting and non-trivial policy issues do not lend themselves to rational decision-making.

This is puzzling. The Canada Health Act is truly interesting and non-trivial. Was it the outcome of irrational decision-making? We can, do, and should reason about principles and values, even though many other factors will enter into our policy decision-making.

I expect that what Fafard wants to say is that policy making cannot be reduced to scientific problem solving. I agree that policy decision making can’t be reduced to science, even in principle. Fafard correctly observes that much of the conventional literature on knowledge translation is naive about the

complex social, political and values-based dimensions of policy change and research use.

Nevertheless, we would be far better off if we could bring more science to bear on our policy choices.

Scientific evidence is most clearly relevant to decisions about the policy means to pursue agreed upon ends. We sometimes overlook that there is agreement among all parties on many fundamental policy goals — e.g., that “all Canadians should have reasonable access to quality health care regardless of their ability to pay” — and on specific goals toward that end, such as that we should improve infant mortality among the Aboriginal population. But there is often great uncertainty and disagreement about what if any effective means we might have to reduce infant mortality. The only reliable way to determine what works is science. So it seems straightforward that health policy choices should conform to the best available evidence.

In practice, however, this is really hard. In the US, John Bridgeland (former director of the G. W. Bush White House Domestic Policy Council) and Peter Orszag (former director of Obama’s Office of Management and Budget and the Congressional Budget Office) wrote that:

Based on our rough calculations, less than $1 out of every $100 of government spending is backed by even the most basic evidence that the money is being spent wisely.

There are many reasons why the policy decision-making fails to arrive at an evidence-based result.

  1. There may not be enough data to decide the issue. A health policy or a social program may only have been studied in a limited experiment that does not map well to the new setting and larger scale represented by the proposed legislation.
  2. The process of passing a law is long and complex. Often, there is no point where evidence is reviewed in a disciplined and independent way.
  3. Members of Parliament are " of course " partisans. They have convictions that lead them to process evidence through motivated reasoning. Their prior beliefs are impervious to evidence because they find reasons to discount inconsistent data.

In summary, politics doesn’t lead to a disciplined and objective review of the evidence because the decision makers don’t really want to do that.

This seems hopeless. But a new book by Ron Haskins expresses cautious optimism that we can do better. Haskins is a Republican who was a domestic policy advisor in the G. W. Bush administration. But he writes with admiration about the Obama administration’s attempts to change the process so as to generate more evidence-based policies.

The book presents case studies on six programs (e.g., a bill to fund teenage pregnancy prevention). Haskins highlights ways to restructure the policy process to increase the leverage of scientific evidence.

  1. Legislation should do less to specify the social intervention and more to specify a) measurable outcomes that the intervention is supposed to achieve and b) the procedures that will be used to select and evaluate candidate interventions.
  2. The outcomes that should be considered in selecting and evaluating programs should be substantive social and individual benefits (e.g., reduced teen pregnancy). They should not be program or process outcomes (e.g., the number of teens who received sex education).
  3. Programs should be selected based on good quality evidence: the results of randomized clinical trials (RCTs) or strong quasi-experimental designs.
  4. There should be procedures for gathering and synthesizing evidence to score solutions. These procedures need to be specified before the evidence is gathered, so that program operators cannot pick the scoring procedure to rationalize the choice of their desired solutions.
  5. Programs should receive funding on the condition that they collect data on real outcomes. Even more importantly, the implementations should include studies with strong designs that generate good evidence about program effectiveness. We want Learning Health Care Systems and, similarly, Learning Social Services Systems.

So there is a route to evidence-based policy making. It’s a hard route and there’s no certainty that it will succeed. But it’s the way forward.

@Bill_Gardner