Archived - Evaluation and Performance Measurement Plan 2009-2010 to 2013-2014

Date: April 4, 2009

PDF Version (526 Kb, 35 Pages)

Table of Contents

• Introduction
• Evaluation Planning Approach
• The Five-Year Evaluation and Performance Measurement Plan
• The Evaluation Process and Protocols
• Consultations
• Updating The Plan
• INAC Evaluation Plan 2009-2010 – 2013-2014
•  Appendix A
•  Appendix B
•  Appendix C

Introduction

Purpose

This document presents Indian and Northern Affairs Canada's (INAC) Evaluation Plan to assess the relevance and performance of its programs and policies, and covers the five year period from 2009-2010 to 2013-2014. It also contains performance measurement strategies to be reviewed by the Evaluation, Performance Measurement and Review Committee at INAC (Terms of Reference for this committee can be found in Annexes).

The plan proposes increased coverage from recent years to address the requirements of the Treasury Board Secretariat's (TBS) policies on Evaluation and Transfer Payments and the Federal Accountability Act, which require that there be a "review"/evaluation, of all direct spending, including transfer payment programs, every five years. As such, this plan provides for full evaluation coverage of all the grants and contributions administered by INAC as well as including direct program spending (non-Grant and Contribution) whenever possible to ensure full compliance with the TBS Evaluation Policy by 2013, as required.

Context for Evaluation

The context for evaluation is best described in the following excerpt from the April 2009 TBS policy on Evaluation:

3.2 Evaluation provides Canadians, Parliamentarians, Ministers, central agencies and deputy heads an evidence-based, neutral assessment of the value for money, i.e. relevance and performance, of federal government programs. Evaluation:

1. supports accountability to Parliament and Canadians by helping the government to credibly report on the results achieved with resources invested in programs;
2. informs government decisions on resource allocation and reallocation by:
   1. supporting strategic reviews of existing program spending, to help Ministers understand the ongoing relevance and performance of existing programs;
   2. providing objective information to help Ministers understand how new spending proposals fit with existing programs, identify synergies and avoid wasteful duplication;
3. supports deputy heads in managing for results by informing them about whether their programs are producing the outcomes that they were designed to produce, at an affordable cost; and,
4. supports policy and program improvements by helping to identify lessons learned and best practices.

The framework for the systematic collection and analysis of performance information is the department's Management, Resources and Results Structure (MRRS). Evaluations are organized along the strategic outcomes of the Program Activity Architecture (PAA).

The plan also includes requirements for performance measurement strategies based on INAC's approach to Performance Measurement Strategies which is derived from the following requirement in Treasury Board's Transfer Payment Policy:

6.5.2 Ensuring that a performance measurement strategy is established at the time of program design, and that it is maintained and updated throughout its life cycle, to effectively support an evaluation or review of relevance and effectiveness of each transfer payment program.

Evaluation Planning Approach

The 2009-2010 to 2013-2014 Evaluation Plan proposes a comprehensive approach to the assessment of the department's programs and activities.

It covers all grant and contribution programs as required by the Federal Accountability Act, and the Treasury Board Secretariat (TBS) policy on evaluation as well as the Policy on Transfer Payments. Where possible programs are clustered to allow for the evaluation function to study and conclude on entire policy areas, using an optimal number of studies focused on broader horizontal issues. This approach also supports annual reporting along the structure of the department's Program Activity Architecture and supports strategic review exercises.

A special annual assessment (The Health of INAC's Performance Information System), a requirement of the TBS evaluation policy, will review performance information being collected by the Department and its usefulness for departmental results reporting purposes. The first report to be tabled in June of 2009 will serve as a benchmark.

Planning Methodology

Documentation of INAC's "evaluation universe" was a critical first step in developing the plan to ensure its comprehensiveness. The department's PAA, its grant and contribution programs, their authorities and expiry dates were all analyzed. The plan reflects INAC PAA elements / strategic outcomes and its grant and contribution programs.

A second step was to identify past commitments to, and central agency expectations for, evaluations, which required a review of departmental Treasury Board submissions' program authorities, management responses to Auditor General audits, and previous evaluations.

In addition, an analysis was done of issues and risks related to achievement of expected outcomes based on management risk discussions, evaluation and audit findings, including those of the Office of the Auditor General, reports, and other documentation of program design and delivery issues facing the department.

The Five-Year Evaluation and Performance Measurement Plan

The INAC Evaluation Plan, 2009-2010 to 2013-2014, proposes comprehensive and sustainable coverage of the Department's programs and activities.

Coverage

The plan includes a total of 54 evaluations and 1 special study (55 in total) over a five-year period, focused on INAC's key policy and program areas. The 54 evaluations provide coverage on all (63) of the department's grant and contribution authorities and also include evaluations of non grant and contribution direct program spending where possible.

The special study assesses factors that cut across programs and policies that affect, either directly or indirectly, achievement of the department's objectives.

Tables in the next section present evaluation projects and performance measurement strategies. Appendix A presents evaluation projects by sector and year. Appendix B presents performance measurement strategies by sector and year.

Risk Considerations

The Treasury Board policy framework for determining risks and the risk assessment approach in the Policy on Transfer Payments were used to assess risks for evaluation and performance measurement planning. These risk assessments have been considered in the identification of evaluation projects and will be further considered in the identification of evaluation issues for individual projects. The risk assessments also factor significantly in the timing and resourcing for evaluations.

The first risk consideration is referred to as Mission-Risk and includes the following elements:

• Size of population affected or targeted by the program
• Consequences if the program or infrastructure risk materializes
• Probability of the program or infrastructure risk materializing
• Degree of control or influence by the department
• Time available to prevent, mitigate, avoid risk

The second risk consideration is that of materiality, which includes the following elements:

• Funding profile
• Public confidence
• Political sensitivity

The third risk consideration is that of performance information availability. This is a key concern in determining the scoping and sequencing of evaluations. Elements included in this risk consideration are:

• Sound underlying program theory
• Robust performance management framework
• Quality evaluation
• Continued relevance of existing performance information
• Time available to generate required performance information

These risk considerations were assessed by the evaluation team in consultation with program managers.

Most evaluations conducted by the Evaluation, Performance Measurement and Review Branch (EPMRB) are summative in nature, focusing on evaluating the relevance and performance of the programs. Formative evaluations, which typically focus on implementation and delivery issues and can be conducted jointly with program managers are also contained in the plan. Such evaluations are completed at the direction of TBS.

Additionally, the Performance Measurement planning included consideration of INAC's Results-Based Management Accountability Framework (RMAF) Special Study, conducted in 2008, as part of its risk assessment.

The Evaluation Process and Protocols

EPMRB is guided in our evaluation work by the TBS policy and standards on evaluation. (April 1, 2009)

For program and senior managers whose programs are being evaluated by EPMRB, the following principles guide the evaluation process:

• EPMRB will consult with the program in developing the Terms of Reference;
• The program Assistant Deputy Minister will sign off on the Terms of Reference, making any financial and human resources available that may be needed to complete the evaluation;
• The program will identify a program contact to facilitate the evaluation process, ensuring that evaluators and their contractors have unrestricted access to documents and other sources of information;
• EPMRB will share all reports related to the evaluation with the designated program contact who will then ensure timely consultation with other program officials involved in the program, including regional staff;
• A comprehensive summary of the findings will be presented to the program for validation by the evaluators at least six weeks prior to the report, management response and action plan being presented to the Evaluation, Performance Measurement and Review Committee (EPMRC);
• The draft final report with recommendations incorporating changes from the validation process will be provided to the program within one month of the report being tabled at EPMRC;
• EPMRB will ensure that the evaluation recommendations flow clearly and logically from the findings. Should the program be unable to accept and/or implement the recommendations, an explanation of their rejection of the recommendations will be provided in the management response and action plan with a justification provided for any alternative course of action;
• Once approved by the committee, EPMRB will finalize the report and submit it for review by INAC's Access to Information Privacy Unit and Communications Branch prior to posting the report, management response and action plan on the departmental website.

(For further details on the process, please see the Critical Path attached to this document as Appendix C.)

Consultations

The plan was discussed with senior management as well as HQ staff and regional staff of INAC and with relevant central agency officials to ensure that it provides for full coverage and that the approach is sound.

Updating The Plan

The plan is updated and tabled annually at the EPMRC, to take account of evolving needs for coverage and strategic information. Because it is a rolling plan, annual updates also identify evaluation and performance measurement activities for the fifth year of the renewed plan. The updated plan reflects work not completed as planned and carried over to another fiscal year.

The following sample table is provided to offer an explanation of how to read the tables presented that outline planned evaluation and performance measurement activities (pages 8-25).

Strategic Outcome: Name of Strategic Outcome as seen in INAC's Program Activity Architecture

Strategic Outcome: The Government

Strategic Outcome: The People

Strategic Outcome: The Land

Strategic Outcome: The Economy

Strategic Outcome: The North

Strategic Outcome: Office of the Federal Interlocutor

Strategic Outcome: Internal Services

Special Studies

APPENDIX A

Evaluations by Fiscal Year by Sector

2009 - 2010

2010 - 2011

2011 - 2012

2012 - 2013

2013 - 2014

APPENDIX B

Performance Measurement Strategies by Fiscal Year by Sector

2009 - 2010

Note: Bolded font with asterisk denotes Performance Measurement Strategies to be brought forward to EPMRC for approval.

2010 - 2011

2011 - 2012

2012 - 2013

APPENDIX C

Critical Path for Evaluation Projects

Footnotes

1. Although Education falls under INAC's Program Activity Architecture Strategic Outcome: The People, this particular authority is being studied within this evaluation as the focus in this case pertains moreso towards measures related to Self-Government. (return to source paragraph)
2. Education falls under INAC's Program Activity Architecture Strategic Outcome: The People. (return to source paragraph)
3. Northern Land and Resources falls under INAC's Program Activity Architecture Strategic Outcome: The North. (return to source paragraph)
4. Métis and Non-Status Indian Organizational Capacity Development falls under INAC's Program Activity Architecture Strategic Outcome: Office of the Federal Interlocutor. (return to source paragraph)
5. Community Infrastructure falls under INAC's Program Activity Architecture Strategic Outcome: The Economy. (return to source paragraph)
6. Governance and Institutions of Government falls under INAC's Program Activity Architecture Strategic Outcome: The Government. (return to source paragraph)
7. Managing Individual Affairs falls under INAC's Program Activity Architecture Strategic Outcome: The People. (return to source paragraph)
8. Claims Settlement falls under INAC's Program Activity Architecture Strategic Outcome: The Government. (return to source paragraph)
9. Responsible Federal Stewardship falls under INAC's Program Activity Architecture Strategic Outcome: The Land. (return to source paragraph)