Five-Year Plan for Evaluation and Performance Measurement Strategies 2013-14 to 2017-18

PDF Version (395 Kb, 33 Pages)

Executive Summary

This document represents Aboriginal Affairs and Northern Development Canada's Five-year Plan for 2013-14 to 2017-18 for evaluations and performance measurement strategies. The plan adheres to guidance provided by the Treasury Board of Canada Secretariat. It aligns with and supports the departmental Management, Resources and Results Structure and no longer references authorities; ensures the evaluation of direct program spending on a five-year cycle; and implements a risk-based approach for determining the calibration of effort for evaluations.

The number of evaluations has increased from 48 to 58 for three reasons: First, new this year, the forecast spending for salary and operation and maintenance was included in addition to grants and contributions to capture all direct program spending. The inclusion of direct spending has increased the number of activities and amount of spending to be covered by the Plan. Second, the Chart of Accounts was used to examine sub-sub-sub activities and resulted in the division of some cluster evaluations into more manageable groupings of related activities. Third, two evaluations of internal services were added to explore areas deemed high risk in the Corporate Risk Profile.

The Plan was presented to the Directors General Internal Operations Committee and reflects input received from senior management and internal services representatives. There are a total of 58 evaluations scheduled between 2013-14 and 2017-18. Fifty percent of evaluations have a performance measurement strategy in place, which is expected to reduce the level of effort required for these evaluations.

There are 12 new evaluations scheduled for 2013-14 and nine carry-overs from 2012-13. A calibration of the level of effort was conducted for new evaluations scheduled for 2013-14. The analysis of materiality, risk and complexity revealed a high risk portfolio of evaluations for year 1 of the Plan.

1. Introduction

This document outlines a plan for evaluating 100 percent of direct program spending over the next five years at Aboriginal Affairs and Northern Development Canada (AANDC) and the development of performance measurement strategies.

1.1 Purpose of the Evaluation Plan

The primary purpose of the plan is to help the Deputy Head ensure that credible, timely and neutral information on the ongoing relevance and performance of direct program spending is available to support evidence-based decision making on policy, expenditure management and program improvement. The Plan also:

1. Provides an opportunity to align evaluations with information needs of the Department and the information needs of others (e.g. central agencies) as articulated in the Policy on Evaluation;
2. Helps ensure that evaluations supporting program redesign are planned and completed in advance of program renewal;
3. Provides an annual platform for program managers and heads of evaluation to discuss the development and implementation of performance measurement strategies that effectively support evaluations;
4. Allows departmental units responsible for the development of the Report on Plans and Priorities and the Departmental Performance Reports, as well as other groups engaged in strategic planning and reporting activities, to identify when evaluations will be available to inform their work;
5. Initiates regular communication and consensus building on evaluation needs and priorities across the Department; and
6. Provides central agencies with advanced notice of when evaluations will be available to inform their work (e.g. in support of Memoranda to Cabinet, Treasury Board submissions, strategic reviews).

Moreover, the Plan serves as a management tool for the Head of Evaluation by enabling workflow and human resources planning for the coming years.

1.2 Drivers for Evaluation Planning

In the Government of Canada, evaluation is defined as the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance and performance, and to examine alternative ways to deliver them or to achieve the same results. Evaluation serves to help establish whether or not a program contributed to observed results and to what extent. It also provides an in-depth understanding of why program outcomes were, or were not, achieved.

The Government of Canada adopted the first Evaluation Policy in 1977 to inform expenditure management and/or program decision making. A renewed policy, standard and directive introduced in 2009, requires the evaluation of all ongoing grants and contributions (G&C) programs every five years and clarifies the management responsibilities and accountabilities of ministers and deputy heads.

Updates to the 2009 Treasury Board Secretariat Policy on Evaluation came into effect April 1, 2012. The Policy states:

3.2 Evaluation provides Canadians, parliamentarians, ministers, central agencies and deputy heads an evidence-based, neutral assessment of the value for money, i.e. relevance and performance, of federal government programs. Evaluation:

1. Supports accountability to Parliament and Canadians by helping the Government to credibly report on the results achieved with resources invested in programs;
2. Informs government decisions on resource allocation and reallocation by:
   
   
   1. Supporting strategic reviews of existing program spending, to help ministers understand the ongoing relevance and performance of existing programs, and
   2. Providing objective information to help ministers understand how new spending proposals fit with existing programs, identify synergies and avoid wasteful duplication;
3. Supports deputy heads in managing for results by informing them about whether their programs are producing the outcomes that they were designed to produce, at an affordable cost; and
4. Supports policy and program improvements by helping to identify lessons learned and best practices.

In accordance with Section 6.1.7 of the Policy on Evaluation, this Plan aligns with and supports the departmental Management, Resources and Results Structures (MRRS), which is the framework for the systematic collection and analysis of performance information. It also ensures the evaluation of all ongoing G&C programs every five years, as required by Section 42.1 of the Financial Administration Act. In compliance with Section 6.1.3 of the Directive on the Evaluation Function, this Plan identifies a risked-based approach for determining methodologies, the level of effort and the appropriate level of resources required to conduct each evaluation.

1.3 Departmental Context

AANDC Mandate

The vision of AANDC is a future in which First Nations, Inuit, Métis and northern communities are healthy, safe, self-sufficient and prosperous - a Canada where people make their own decisions, manage their own affairs and make strong contributions to the country as a whole.

To this end, the Department supports Aboriginal peoples (First Nations, Inuit and Métis) and Northerners in their efforts to:

• improve social well–being and economic prosperity;
• develop healthier, more sustainable communities; and
• participate more fully in Canada's political, social and economic development — to the benefit of all Canadians.

AANDC is the federal department primarily responsible for meeting the Government of Canada's obligations and commitments to First Nations, Inuit and Métis, and for fulfilling the federal government's constitutional responsibilities in the North. The Department's overall mandate and wide–ranging responsibilities are shaped by centuries of history, and unique demographic and geographic challenges. It derives from the Canadian Constitution, the Indian Act, the Department of Indian Affairs and Northern Development Act, territorial acts, treaties, comprehensive claims and self–government agreements as well as various other statutes affecting Aboriginal people and the North.

The Indian and Inuit Affairs mandate derives from the Indian Act and its amendments over the years, from specific statutes enabling modern treaties, such as the Nisga'a Final Agreement Act or the Labrador Inuit Land Claims Agreement Act, and from more recently enacted statutes such as the First Nations Fiscal and Statistical Management Act and the First Nations Jurisdiction Over Education in British Columbia Act, designed to provide First Nations with jurisdictional powers beyond the Indian Act. A significant amount of the Department's mandate is derived from policy decisions and program practices that have developed over the years; it is framed by judicial decisions with direct policy implications for the Department; and it is structured by funding arrangements or formal agreements with First Nations and/or provincial or territorial governments.

The Northern Development mandate derives from the Department of Indian Affairs and Northern Development Act; from statutes enacting modern treaties north of 60°, such as the Nunavut Land Claims Agreement Act, or; self‑government agreements, such as the Yukon First Nations Self-Government Act, and; from statutes dealing with environmental or resource management, and; is framed by statutes that enact the devolution of services and responsibilities from AANDC to territorial governments, such as the Canada‑Yukon Oil and Gas Accord Implementation Act.

Most of the Department's programs, representing a majority of its spending, are delivered through partnerships with Aboriginal communities and federal–provincial or federal–territorial agreements. AANDC also works with urban Aboriginal people, Métis and non–status Indians (many of whom live in rural areas) through the Office of the Federal Interlocutor. AANDC is one of 34 federal departments and agencies delivering Aboriginal and northern programs and services.

Program Alignment Architecture

AANDC's broad mandate is demonstrated by the Program Alignment Architecture (PAA), which supports four strategic outcomes (SO):

1. The Government - Good governance and co–operative relationships for First Nations, Inuit and Northerners.
2. The People - Individual, family and community well–being for First Nations and Inuit.
3. The Land and Economy - Full participation of First Nations, Inuit and Métis individuals and communities in the economy.
4. The North - Self–reliance, prosperity and well–being for the people and communities of the North.

Appendix A provides a more detailed breakdown of the PAA.

Transfer Payment Programs

According to the 2011-12 Public Accounts of Canada, AANDC is the fifth largest in terms of total ministerial net expenditures (behind Finance, Human Resources and Skills Development Canada (HRSDC), National Defence and Public Safety), and third largest in terms of total transfer payments/G&C (behind Finance and HRSDC). In 2011-12, AANDC's total net expenditures were $7,907,783 and total transfer payments were $6,453,906 [Note 1]. The requirement for direct program spending to be evaluated every five years represents a significant volume of work for the Department.

Planned Expenditures for Fiscal Year 2013-14

2. Planning Methodology

Section 6.2.3 of the Directive on the Evaluation Function states that the Head of Evaluation is responsible for developing and annually updating a rolling five-year evaluation plan. Accordingly, at AANDC, the Evaluation Plan was developed by the Evaluation, Performance Measurement and Review Branch (EPMRB), which is part of the Audit and Evaluation Sector. The approach for the creation of the 2013-14 Plan was as follows:

Define the evaluation universe and scoping

The evaluation universe shows the how each component of the MRRS is to be evaluated and ultimately, how the Plan will achieve full coverage of the MRRS. The evaluation universe also shows the breakdown of forecast spending covered in each evaluation.

The evaluation universe for the 2013-14 Plan no longer uses transfer payment authorities to define the scope of evaluations. Instead, the Chart of Accounts was referenced to identify program activities under sub-sub-activities that could be examined as part of a single evaluation. As a result, a few large evaluations have been split into smaller projects. For example, the evaluation of "Activation of Community Assets" from last year's Plan has been split into five smaller evaluations to better accommodate the variety of unrelated activities contained under this sub-activity.

With the restructuring of the Plan using the MRRS instead of the authorities, some gaps in coverage have emerged. It would appear that evaluations have never been conducted for Northern Governance, Northern Land and Environmental Management and Indian Oil and Gas Canada. Another program that has never been evaluated is the Strategic Partnership Initiative, but this is because it is a new program introduced in 2009. Together these programs represent two percent of total forecast spending for 2013-14. This means that over 98 percent of total forecast spending for 2013-14 has been previously evaluated.

Conduct a risk assessment of units of evaluation

EPMRB has adopted the Audit and Assurance Services Branch (AASB) risk ranking to inform the level of effort and allocation of resources for evaluations. Each year, AASB develops risk rankings for auditable units in the Department. The process followed involves extensive review of corporate documents, workshops and consultations with program representatives and external stakeholders and is an essential component for the preparation of the Risk‑Based Audit Plan.

The AASB risk approach is consistent with guidance from the Office of the Comptroller General, is thorough, and in many cases, the audit and evaluation units are the same. Risk considerations include impact of identified risks on the achievement of departmental outcomes, materiality, scope, potential for public scrutiny, legal risk, prevalence of risk, and the severity of consequences.

The risk rankings can be seen in the Plan and are colour coded to reflect the five levels of risk identified in the Risk-Based Audit Plan. The five levels are defined as follows:

• Very High Risk – The auditable entity is inherently exposed to multiple risks that are expected to remain or get worse over time. Business conditions contain considerable risk factors. If one or more risks were to materialize, consequences would be severe and could include permanent or long-term damage to AANDC's ability to achieve its objectives. Consequences would be felt by the majority of stakeholders, both internally and externally to AANDC.
• High Risk - The auditable entity is inherently exposed to many risks that are expected to remain over time. Business conditions contain many risk factors. If one or more risks were to materialize, consequences would be significant and could be endured by AANDC with significant management attention. Some AANDC activities could be subject to significant review or changed ways of operation. Consequences would be felt by many stakeholders.
• Moderate Risk - The auditable entity is inherently exposed to several risks that are expected to remain over time. Business conditions contain some risk factors. If one or more risks were to materialize, consequences would be moderate and could be managed with a minor level of management attention. Consequences would be felt by a sub-set of stakeholders.
• Low Risk - The auditable entity is exposed to a few risks that may diminish over time. Business conditions contain a few risk factors. If one or more risks were to materialize, consequences would be minor and could be absorbed through normal activity. Consequences would be isolated.
• Very Low Risk - The auditable entity is inherently exposed to few or no risks. Business conditions contain few or no risk factors. If one or more risks were to occur, the consequences would be negligible and could be absorbed through normal activity.

Create the Plan

The 2013-14 Plan builds upon the previous year's plan. As discussed above, the units of evaluation were aligned with the MRRS and forecast direct program spending was added. Deadlines for evaluations were calculated to ensure that an evaluation was scheduled to take place within five years from the date of the previous one.

Planned audits have been added to the Plan to inform the calibration of effort for an evaluation, allow for integration of audit findings, and promote better coordination with programs. Audits generally reveal a considerable amount about the design and management of programs and give early indications on the achievement of program outcomes. Audits scheduled in advance of an evaluation can impact the scope and level of effort needed for an evaluation.

Recent audits are also valuable sources of information for evaluation as they inform the analysis of program design and provide analysis of efficiency and economy. Ideally, audits and evaluations will be scheduled in different years, however, where audits and evaluations are scheduled to occur in the same year, it was agreed with AASB that audit work will be conducted early in the fiscal year, so that programs are not overburdened with requests for information.

In this Plan, 58 evaluations are scheduled between 2013-14 and 2017-18. The table below shows the distribution of evaluations for the five years covered by the plan.

Distribution of Evaluations by Strategic Outcome and by Year

Currently, 29 of the 58 planned evaluations (50 percent) have a performance measurement strategy in place.

Consultation with Senior Management

A draft Plan was presented to the Directors General Internal Operations Committee to collect feedback on the scope and timing of evaluations. Options for integrating Internal Services into the Plan were also presented. A series of subsequent meetings were scheduled with Directors and Directors General of Internal Services and Land and Economic Development (LED) Sector to further define scope and timing of evaluations. As a result of these discussions, two evaluations of Internal Services were added to the Plan and the evaluation of Activation of Community Assets from last year's plan was split into five separate evaluations.

Coverage

Total forecast direct program spending for 2013-14 is $7,904,970,562. Total forecast direct program spending covered by this Plan is $6,983,598,799. In summary, this Plan covers 88 percent of total forecast direct program spending. Not covered by this Plan is the Residential Schools Resolution, which accounts for almost nine percent of total forecast direct program spending. This will not be evaluated because an assessment revealed that it is closely monitored through regular reporting to the court and Treasury Board. The Management of Other Negotiated Settlements and Aboriginal Women's Issues, not covered in this Plan, will be integrated into evaluations in the next Plan. Finally, most internal services, which account for another three percent, are also not identified for evaluation. This Plan only incorporates those internal services deemed high risk in the Corporate Risk Profile. Annual coverage of total forecast direct spending for 2013-14 is detailed below:

Annual Coverage of Total Forecast Direct Program Spending for 2013-14

Calibration of Level of Effort

Section 6.2.1, sub-section c, of the Standard on Evaluation for the Government of Canada gives departments the flexibility to calibrate the nature and depth of each evaluation undertaken in relation to the risks associated with the program and the information needs of the Deputy Head.

An analysis of the 12 evaluations to be started in 2013-14 was undertaken to determine the level of effort required for each evaluation and better align available resources. Four considerations went into the calibration exercise:

1. Materiality – A score of 3 was assigned to evaluations of programs with annual G&C spending over $500 million; 2 was for spending between $100 and $500 million; and 1 was assigned to evaluations covering less than $100 million.
2. Risk – A score of 4 was assigned to evaluation units with a risk ranking of "very high"; 3 for "high", 2 for "moderate" and 1 for "low"
3. Complexity – The scores for complexity were guided by the number of delivery partners, coverage, governance structure, number of delivery mechanisms and number of objectives:
   
   

   High (score=3)	Medium (score=2)	Low (score=1)
   Large number of external delivery partners or delivery through the regions, broad coverage, complex governance structure, multiple delivery structures for multiple objectives	Small number of delivery partners (ie. regions), targeted to a few groups with slightly different characteristics, multiple layers of governance, one delivery structure	Delivery out of Headquarters or one single entity, targeted to one group, simple governance structure, one objective

4. Performance Measurement – The availability of performance measurement data will reduce the level of effort required to complete an evaluation, so evaluations with a performance measurement strategy place for three or more years was given a score of 0. A score of 1 signifies a new performance measurement strategy has been in place for one to two years. A score of 2 means no performance measurement strategy exists and thus, greater effort will be required to complete the evaluation. It is important to note that the above scoring assumes that performance measurement strategies in place have robust indicators of success and have been fully implemented.

Overall, the scoring for the four areas defined above will permit a maximum score of 12. Evaluations with a score of 9 or more will likely require more resources to complete. Results of the calibration exercise are as follows:

A risk ranking was not available for three of the twelve evaluations scheduled for 2013-14 because AASB only develops risk rankings for auditable units. In addition, forecast spending was not available for one evaluation scheduled for 2013-14. As a result, a ranking could not be calculated for three of the twelve evaluations scheduled for 2013-14. Of the remaining nine evaluations, three have a score over 9 and another four received a score of 8, which is still significant. In the end, it would appear that seven of the twelve evaluations would require a high level effort. A closer look at the design, delivery and availability of performance information is needed to better define the resources required.

3. 2013-14 to 2017-18 AANDC Plan for Evaluation and Performance Measurement Strategies

* Evaluations that do not have a coloured risk ranking were not identified as an auditable unit and were thus not ranked by the Audit and Assurance Services Branch.

Appendix A: AANDC Program Alignment Architecture

Appendix B: AANDC Evaluation Universe

Coverage amounts may differ due to the scope of the evaluation, i.e. evaluation covers specific activities and/or a broader range of activities