Canadians are, rightly, interested in the extent to which government programs achieve their intended objectives. Indeed, the federal government’s approach to achieving results centres around the pursuit of final “outcomes,” and whether programs and policies are improving the lives of Canadians. But along the way from initiating a program to assessing its results, the government needs robust assessment tools. That includes data collected through the life of a program, with a particular emphasis on early indications of whether it is performing as hoped.

Why are these early stages so important? Analysis focused on long-term outcomes may miss the critical step of understanding progress on short-term results – the foundation for final outcomes. By drawing attention to earlier results, problems can be detected more quickly and programs adjusted when it makes the most sense.

The current emphasis inside the federal public service on long-term impacts of programs may be a response to what has been considered a previous over-reliance on program activities rather than a program’s effectiveness. That over-reliance, in turn, stemmed from a lack of the relevant and high-quality data essential to analyzing a program’s impact.

A government program can take 20 years to fully mature. If policy, program and service delivery managers concentrate from the outset on long-term outcomes, they’ll miss out on valuable insights about the program’s effectiveness and whether interventions are necessary. We should not ignore what happens in the initial implementation or early phases of a program’s cycle.

The building blocks of program success

There are three core building blocks of programs that underlie their early and intermediate stages – awareness, understanding and action. The target population for the program must be aware of it, and they must understand it before they can finally act to take advantage of the benefits. For example, a program typically subscribed to by men, such as support for finding apprenticeships and employment in the trades, can also have the potential to help women, but barriers block their awareness and understanding of the program. Ultimately, this lack of awareness and understanding would prevent those women taking action to participate in the program.

Awareness is influenced by context, a person’s exposure to information, his or her capacity to absorb information, an individual’s beliefs, and attitudes – all of which vary among potential program participants.

Analysis of how the program is known and understood by demographic groups such as men, women, persons with disabilities, etc. – known in government as gender-based analysis-plus (GBA+) – can reveal a great deal about the reach of the program and, consequently, where improvements can be made. In other words, what works, for whom and under what conditions.

Understanding is influenced by such factors as the level of education of potential participants, and the type of communication tool that is being used to try to reach them (for example, do they have access to the internet?). These differences within the population need to be monitored so that adjustments can be made to the program.

Action, the third building block, might involve applying for a government benefit or contributing to a retirement or education savings plan. Whether someone takes the action can be influenced by factors including personal motivations, and can be measured by looking at take-up rates and whether there is sustained involvement by the individual. Early action taken by an individual or group is a key step in achieving longer-term positive results, such as improved financial security or well-being.

Risks of focusing only on action

When the building blocks of awareness and understanding are not adequately considered during the early stages of a program’s design and rollout, the final results can be different than expected.

The former Child Trust Fund in the United Kingdom was designed to provide a universal benefit to children for future uses such as advanced education. The UK government enrolled children in the program automatically one year after birth if their parents hadn’t already done so. The government’s initiative in setting up these accounts for parents was seen as a way to incentivize savings early on and potentially be used to finance post-secondary education.

The inner workings of government
Keep track of who’s doing what to get federal policy made. In The Functionary.
The Functionary
Our newsletter about the public service. Nominated for a Digital Publishing Award.

However, evaluation evidence showed that over three-quarters (78 percent) of parents opened accounts for their children, but only 37 percent made a contribution after the initial government benefit was deposited. In cases where accounts were opened automatically by the government, only nine percent of parents made subsequent contributions. The action of opening an account was automated; however, related elements of awareness and understanding of the accounts may have not been sufficiently addressed in program design and delivery in order to achieve the outcome of increased savings.

Evaluation of a similar program, the Canadian education savings program, showed how the building blocks of awareness and understanding are factored into program outcomes. The Canada Education Savings Program (CESP) offers government contributions to a Registered Education Savings Plan (RESP) to support children’s post-secondary education. A CESP program evaluation showed that parental aspirations around post-secondary studies and pre-existing savings habits helped explain why the program reached some potential participants and not others.

Cultural and attitudinal factors were also important, the evaluation evidence showed. For example, CESP contribution levels differed significantly between province of residence. Further to that, data from the Access to Education and Training Survey (ASETS) suggest that higher levels of RESP contributions in British Columbia were likely due to a greater share of Chinese and East Indian immigrants in BC’s population compared with other provinces. Both groups tended to make much higher RESP contributions than families with two Canadian-born parents, even when accounting for other factors such as family income, the data showed.  

Collecting the right data at the right time

It was possible to do such nuanced evaluations of the RESP program because relevant and high-quality data were available. In the past, program management, for operational reasons, focused on collected information on administrative elements such as the number of applications received or the time taken to issue a payment. Similarly, data received from third-party deliverers often concentrated on the activities of the programs they delivered. This type of data is important, but doesn’t draw attention to what is happening more broadly within a program.

Clearly, measuring short-term outcomes to assess program effectiveness requires proactively collected data. Discussions continue on improving data quality and the need for a shift from data ownership to data stewardship within the federal government. The GCInfoBase, developed at the Treasury Board of Canada Secretariat, offers a great deal of information at the program level across many federal departments and agencies, and now includes up to three years’ worth of results information on various programs.

Pilot projects have tested pay-for-performance arrangements in which organizations that delivered programs were paid on the basis of participants’ movement along particular milestones or stages. Such approaches, which can be seen as types of experiments, pay attention to key early indicators and ensure that relevant data are collected for analysis.

To achieve program success and positive long-term outcomes for Canadians, the important building blocks of awareness and understanding deserve attention. The scope of program analysis must be enlarged and data collection improved to adequately take these elements into account, and to see the nuances and incremental changes that take place during the early stages. As such, the necessary course adjustments can be made to improve program management and delivery, and ultimately the well-being of citizens.

The views expressed in this article are those of the author and do not necessarily represent the opinion of the Treasury Board of Canada Secretariat or the Government of Canada.

Photo: Shutterstock / Enkel

(This article has been translated into French.)

Do you have something to say about the article you just read? Be part of the Policy Options discussion, and send in your own submission. Here is a link on how to do it. | Souhaitez-vous réagir à cet article ? Joignez-vous aux débats d’Options politiques et soumettez-nous votre texte en suivant ces directives.

Christine Minas
Christine Minas is a director in the results division at the Treasury Board of Canada Secretariat and a champion of the Federal Evaluation Community.

You are welcome to republish this Policy Options article online or in print periodicals, under a Creative Commons/No Derivatives licence.

Creative Commons License

More like this