Archived - Evaluation Plan 2007-2008 to 2011-2012

Purpose

This document presents Indian and Northern Affairs Canada's (INAC) plan to assess the value for money of its programs, policy frameworks and approaches. The Evaluation Plan covers the five year period from 2007-2008 to 2011-2012.

The plan proposes increased coverage from recent years to address the requirements of the Transfer Payment Policy and the Federal Accountability Act. The Act calls for a "review", i.e., evaluation, of all direct spending, including transfer payment programs to be covered each five years.

Purpose of Evaluation

The role of the evaluation function is to provide Ministers, central agencies and Deputy Ministers with regular evidence-based assessments of the performance of programs and other initiatives as well as insights on whether they continue to be relevant. It should be a key source of neutral performance information.

Evaluation is expected to inform decisions about new program and policy approaches, provide guidance on necessary policy and program improvements, and support departmental accountability to Parliament and Canadians by helping deputy heads to manage for results.

The evaluation function is currently guided by Treasury Board's 2001 Evaluation Policy. A revised policy, expected in fall 2007, will increase evaluation requirements to support Treasury Board's renewed expenditure management system. The proposed policy has been respected in the drafting of this plan.

The policy's expected results are:

• neutral, credible and timely evidence-based information for government and Parliament on the relevance and
• five-year cyclical evaluation of all grant and contribution spending (the policy is the means for addressing transfer payment review requirements of the Federal Accountability Act);
• annual assurance from the Head of Evaluation of the adequacy of departmental performance measurement.

With the requirement for additional evaluation coverage, Treasury Board has expanded its definition of evaluation instruments and approaches. Reviews and management consulting studies can now be considered "value-for-money" evaluations.

The 2007-2008 to 2011-2012 Evaluation Plan proposes a strategic, risk-based approach to the assessment of the department's programs and activities.

It covers all grant and contribution programs and in many cases proposes a strategic approach of program clusters. This approach allows for the evaluation function to study and conclude on entire policy areas, using an optimized number of studies focused on broader horizontal issues. This approach also supports annual reporting along the structure of the department's Program Activity Architecture.

The plan also proposes a series of special studies to address horizontal risks and factors that affect, either directly or indirectly, achievement of the department's intended results by assessing some of the complexities of INAC's operating environment and strengths of current practices, instruments and approaches.

Planning Assumptions

A number of assumptions have been made in developing the plan:

• Introduction of a revised Policy on Evaluation;
• Treasury Board Secretariat acceptance of cluster evaluations that are based on the department's Program Activity
• Evaluations supporting Treasury Board Secretariat accountability requirements would be led by the Evaluation Branch.

Planning Method

Documentation of INAC's "evaluation universe" was a critical first step in developing the plan to ensure its comprehensiveness. An analysis was done of the department's Program Activity Architecture, its grant and contribution programs, their authorities and expiry dates, and the department's major policies and policy frameworks.

The plan reflects INAC's Program Activity Architecture (PAA) elements / strategic outcomes and its grant and contribution programs.

A second step was to identify past commitments to and central agency expectations for evaluations, which required a review of departmental Treasury Board submissions, program authorities; management responses to Auditor General audits, and recent evaluation plans.

In addition, an analysis was done of issues and risks related to achievement of expected outcomes based on management risk discussions, evaluation and audit findings, including the Office of the Auditor General audit findings; reports, and other documentation of program design and delivery issues facing the department .

Some major risk factors related to INAC programming are discussed below. A crosswalk has been done to identify links

between these risks and the evaluation projects, these are shown in the plan that begins on page 9.

The INAC Evaluation Plan, 2007-2008 to 2011-2012 proposes comprehensive and sustainable coverage of the department's programs and activities.

Coverage

The plan proposes a total of 38 evaluation projects over a five-year period, focussed on INAC's key policy and program areas. Of these, 33 projects provide evaluation coverage of the department's approximate 55 grant and contribution authorities and five special studies to assess factors that cut across programs and policies and that affect, either directly or indirectly, achievement of the department's objectives.

Tables in the next section present evaluation projects and special studies. Appendix A presents evaluation projects by sector and year.

Risk Considerations

The Treasury Board policy requires that evaluation plans take account of risk. The following bullets suggest risk factors for INAC. They have been considered in the identification of evaluation projects and will be further considered in the identification of evaluation issues for individual projects. It should be noted that many of these risk factors are closely interrelated.

• Clarity of strategic directions and objectives/expected outcomes: In some cases, objectives of programs/policies were not well articulated at the outset, and in others, the focus of INAC programs and initiatives has changed since their introduction, with the result that intended outcomes are no longer clearly defined or there are flaws in design theories or gaps in programming, for example preventive programming.
• Performance, results and reporting: In some areas, as pointed out by the Office of the Auditor General, Treasury Board and Parliamentary committees, the department lacks results information and evaluative work needed to clearly demonstrate a causal link between its investments and the outcomes achieved, thereby jeopardizing its ability to make a successful business case for additional resources, when needed.
• Accountabilities: Authority has been delegated to First Nations for delivery of many Indian and Northern Affairs Canada programs, yet the accountability for these programs remains with the Minister of Indian and Northern Affairs. Although responsible, in both legislation and public expectation, for programs/services to enhance or to ensure the well-being of First Nations, INAC has little control over their quality, timeliness, and effectiveness.
• Inter-jurisdictional relationships: Some programs are delivered in fields where standards and practices are provincially regulated. In some cases, program design / delivery has not kept up with changes in provincial programs or standards, e.g., Child and Family Services.
• Processes and authorities: Some INAC objectives must be achieved through negotiations with First Nations, e.g. comprehensive, specific and special claims and INAC does not control the pace or outcomes. First Nations may choose litigation over negotiation, or decide not to ratify an agreement-in-principle.
• Coordination between federal programs for Aboriginal peoples: A number of federal departments have similar programs targeted to the same population, especially in the social sector. There has been little analysis of the extent to which these programs complement or overlap, or whether coordination can be improved. INAC is often seen as the lead on issues that affect Aboriginal peoples, even when that role more properly lies with other departments.
• Functional guidance and regional delivery: INAC operations are highly decentralized. Decentralization has put pressure on coordination mechanisms to ensure the integration of regional practices with headquarters policies and priorities. Notional budget allocations hinder the tracking of actual expenditures and efforts in relation to outcomes and blur the lines of accountability. Roles and responsibilities of regions in regards to community development, and outcome monitoring and result-based management are not always clear or consistent and there also are some gaps in functional guidance due to weaknesses in program control frameworks.
• Capacity and governance: First Nations that deliver INAC programs are, in some cases, hampered by poor governance regimes, the lack of trained local personnel, and other limitations arising from their small size and the remoteness of their communities. The lack of adequate arrangements to ensure accountability from delivery organizations to First Nations is exacerbated by the absence of an independent complaints and allegations process for First Nations.
• Funding instruments: The government's funding instruments may be poorly suited to the objectives and business of this department, as they focus on short term activity, compliance monitoring and reporting to government rather than creating First Nations governments with a long term vision who are accountable to the communities they serve.
• Comprehensive community planning for First Nations: With few exceptions, the department's financial support has been provided on the basis of approved authorities and formulae rather than comprehensive community plans, although community building and economic development would be better served by integrated and comprehensive programming that responds to local priorities.
• Performance information: Much information is collected from First Nations, but the information collected provides little insight into the performance of the department's funding programs and is not used effectively. Information often tells us only that money has been spent. There is also a lack of performance information on the effectiveness of aboriginal institutions put in place to deliver services outside of the Indian Act and to develop a First Nation public service.
• Financial and legal issues: The department's legal obligations give rise to significant financial obligations, e.g. obligations related to renewal of land claim agreement implementation plans and changes in provincial/territorial legislation and standards.

Socio-Economic Policy and Regional Operations

PAA Program Activity /Policy Area	Risk considerations	Evaluation	Fiscal Year					Coverage -Authority /year of expiry/ -Annual allocation (07/08) -Policy framework
			07/08	08/09	09/10	10/11	11/12
Social development	Federal coordination Interjurisdictional relationships Complex delivery Accountabilities Capacity and governance Financial and Legal issues	1. Income Assistance, National Child Benefit Reinvestment (A program-led review of Assisted Living is underway)	X					Grants to provide income support to indigent on-reserve residents / 2008 / $10,000 Contributions to provide income support to indigent on-reserve residents / 2008/ $767,990,000 Contributions to provide programming for low income reserve residents with children under the National Child Benefit Reinvestment Initiative / 2008 / $52,000,000 Social Development: The Way Forward
		2. Family Violence Prevention				X		Contributions to support culturally appropriate prevention and protection services for Indian children and families resident on-reserve / 2012 / $30,400,000
		3. Child Welfare (includes Child and Family Services)				X		Contributions to support culturally appropriate prevention and protection services for Indian children and families resident on-reserve / 2012 / $487,000,000
		4. Labrador-Innu Comprehensive Healing Strategy (Program-led review involving Health Canada)		X				Funding through following authorities – Policy and Consultation, Education, Social Development, Capital Facilities and Maintenance, Lands and Estate Management
Education	Interjurisdictional relationships Accountabilities Functional guidance Capacity and governance Financial and Legal Issues	5. Elementary and Secondary Education				X		Payments to support Indian, Inuit and Innu for the purpose of supply public services in education / 2008 / $1,424,255,000 (includes elementary and secondary education, special education, cultural centres and youth employment strategy) Grants to Indians and Inuit to provide elementary and secondary educational support services / 2008 / $150,000 Grants to Inuit to support their cultural advancements / 2008 / $45,000 Contributions under the First Nations SchoolNet Program / 2009 / $5,800,000 2005 Education Action Plan
		6 Post Secondary Education				X		Grants to Indians and Inuit to support their post-secondary educational advancements / 2008 / $1,100,000
Community Infrastructure	Clarity and strategic direction Accountabilities Interjurisdictional relationships Funding instruments Capacity and governance Financial and Legal Issues Clarity and strategic direction Accountabilities Interjurisdictional relationships Funding instruments	7. First Nations Water Management (TOR in development)	X					Payments to support Indians, Inuit and Innu for the purpose of supplying public service in capital facilities and maintenance / 2010 / $1,068,505,000 INAC Long Term Capital Plan Federal government Policy Framework on Water Plan of Action on Drinking Water Housing Policy Bill C-12
		8. Long-term Capital Planning			X
		9. On-Reserve Housing (joint INAC /CMHC)		X
		10. First Nations Infrastructure Fund (involving Infrastructure Canada)				X
		11. Housing Policy	X
		12. Emergency Management					X	Bill C-12

Claims and Indian Government

PAA Program Activity /Policy Area	Risk considerations	Evaluation	Timeframe Fiscal Year					Coverage -Authority /year of expiry/ -Annual allocation (07/08) -Policy framework
			07/08	08/09	09/10	10/11	11/12
Claims Settlements and Cooperative Relationships	Performance and results Clarity and strategic direction Processes and authorities Federal coordination Financial and Legal Issues	13. Comprehensive Land Claims Agreements Impacts Evaluation	X	X				Contributions to support the negotiation process for comprehensive, specific and special claims and self-government initiatives / 2010 / $45,349,000 Grants to support First Nations, Inuit, tribal councils, organizations or other level of government for the implementation activities as stipulated in the various agreements / 2010 / $113,666,000 Payments to self-governing Aboriginal organizations, pursuant to comprehensive land claim agreements, self-government agreements or treaty legislation / 2010 / $74,415,000 Contributions to beneficiaries and various implementing bodies for implementing comprehensive land claim agreements / 2010 / $161,955,000
		14 Comprehensive Land Claims Agreements Evaluation of Negotiation and Implementation					X
		15 Specific Claim Agreements		X				Payments to the Government of the Northwest Territories to facilitate the implementation of comprehensive land claim settlements / 2009 / $1,742,000 Grants to support the beneficiaries/organizations for the settlement of specific and special claims / 2010 / $310,644,000 Grant to the Nunatsiavut Government for the implementation of the Labrador Inuit Land Claim Agreement pursuant to the Labrador Inuit Land Claims Agreement Act / 2015 / $17,987,000 Comprehensive Land Claims Policy Inherent Right Policy Outstanding Business: A Native Claims Policy
	Performance and results Clarity and strategic direction Processes and authorities Federal coordination Financial and Legal Issues	16. Self Government Agreements		X				Grants to the Sechelt Indian Band pursuant to the Sechelt Self-Government Act / 2010 / $4,245,000 Grant to the Westbank First Nation to support the implementation of the Westbank First Nation Self-Government Agreement / 2010 / $4,124,000 Payments to Yukon First Nations pursuant to individual self-government agreements / 2011 / $45,096,000 Grant to the Miawpukek Indian Band to support designated programs / 2010 / $9,075,000 (SEPRO) Grants for Mi'kmaq Education in Nova Scotia / 2010 / $32,858,000
		17. Treaty Related Measures			X			Contributions to First Nations, their organizations, provinces and third parties for Interim Measures and British Columbia Treaty Related Measures / 2010 / $7,479,000

Lands and Trust Services

PAA Program Activity /Policy Area	Risk considerations	Evaluation	Timeframe Fiscal Year					Coverage - Authority /year of expiry/ -Annual allocation (07/08) -Policy framework
			07/08	08/09	09/10	10/11	11/12
Governance and Institutions of Government / Co-operative relationships	Capacity and governance Accountabilities Funding instruments Performance and results	18. First Nations Land Management (program-led)	X					Contributions to implement the First Nations Land Management Act / 2008 / $13,600,000 First Nations Land Management Act
		19. Indian Government Support				X		Grant for Band Support Funding / 2010 / $227,843,000 Payments to support Indians, Inuit and Innu for the purpose of supplying public services in Indian Government Support / 2008 / $125,090,000 Contributions to support the building of strong governance, administrative and accountability systems / 2010 / $24,300,000
		20. First Nations Fiscal and Statistical Management					X	Contributions to First Nations institutions for the purpose of enhancing good governance / 2008 / $9,581,000 Grant to the First Nations Financial Authority / 2011 / 850,000
Management of Individual Affairs		21. Registration and Administration			X			Contributions to Indian bands for land and estates management / 2010 / $13,145,000 Contributions to Indian bands for registration and administration / 2010 / $6,519,000
Responsible Federal Stewardship		22. Contaminated Sites	X					Contributions to First Nations for the management of contaminated sites / 2008 / $10,950,000

Policy and Strategic Direction

PAA Program Activity /Policy Area	Risk considerations	Evaluation	Timeframe Fiscal Year					Coverage -Authority /year of expiry/ -Annual allocation (07/08) -Policy framework
			07/08	08/09	09/10	10/11	11/12
Governance and Institutions of Government	Clarity and strategic direction Accountabilities	23. Representative Organizations and Consultation and Policy Development		X				Contributions for the purpose of consultation and policy development / 2010 / $24,824,000 Contributions to support the basic organization capacities of representative Aboriginal Organizations / 2012 / $12,633,881
		24. Gender-Based Analysis	X					Gender-based Analysis Policy
		25. National Aboriginal Achievement Foundation (PCH led evaluation)		X				Contributions to the National Aboriginal Achievement Foundation / 2010 / $817,000

Aboriginal Economic Development

PAA Program Activity /Policy Area	Risk considerations	Evaluation	Timeframe Fiscal Year					Coverage -Authority /year of expiry/ -Annual allocation (07/08) -Policy framework
			07/08	08/09	09/10	10/11	11/12
Economic Development	Clarity and strategic direction Authorities and process Funding instruments Performance and results Unclear accountabilities Processes and authorities Functional guidance	26. Community Economic/ Aboriginal Business Development	X					Payments to support Indians, Inuit and Innu for the purpose of supplying public services in economic development / 2010 / $109,587,000 Aboriginal Business Development Program / 2008 / $38,000,000 Partnering Strategy for Aboriginal Economic Development

Northern Affairs

PAA Program Activity /Policy Area	Risk considerations	Evaluation	Timeframe Fiscal Year					Coverage -Authority /year of expiry/ -Annual allocation (07/08) -Policy framework
			07/08	08/09	09/10	10/11	11/12
Northern Economy and Governance	Interjurisdictional relationships Performance and results Functional guidance	27. Northern Economic Development and Governance	X	X				Contributions for promoting regional development in Canada's three territories / 2010 / $22,457,200 (Strategic Investment) Contributions for promoting the safe use, development, conservation and protection of the North's natural resources / 2010 / $24,416,100 Grant for the advancement of scientific knowledge of the North / 2010 / $1,085,500 Contributions for promoting the Political, Social, and Scientific Development of Canada's Three Territories / 2010 / $329,100
Northern Land and Resources	Interjurisdictional relationships Performance and results Functional guidance	28. ecoENERGY for Aboriginal and Northern Communities					X
		29. Northern Environment International Polar Year component (program-led formative evaluation)		X X
Healthy Northern Communities	Interjurisdictional relationships Performance and results Functional guidance	30. Health Care and Counselling and Healthy Communities		X				Contributions for Inuit Counselling in the South / 2010 / $80,000 Grants to the Government of the Northwest Territories and the Government of Nunavut for Health Care of Indians and Inuit / 2010 / $45,490,000 Food Mail

Office of the Federal Interlocutor

PAA Program Activity /Policy Area	Risk considerations	Evaluation	Timeframe Fiscal Year					Coverage -Authority /year of expiry/ -Annual allocation (07/08) -Policy framework
			07/08	08/09	09/10	10/11	11/12
Management of Métis Aboriginal Rights Issues and Operation of OFI	Clarity and strategic direction Capacity and governance	31. Office of the Federal Interlocutor	X					Federal Interlocutor's Contribution Program / 2008 / $13,504,000 Inherent Right Policy Supreme Court of Canada's Powley Decision
Economic Development	Capacity and governance level Federal coordination	32. Urban Aboriginal Strategy				X		Urban Aboriginal Strategy / 2012 / $10,095,000

Chief Financial Officer

PAA Program Activity /Policy Area	Risk considerations	Evaluation	Timeframe Fiscal Year					Coverage -Authority /year of expiry/ -Annual allocation (07/08) -Policy framework
			07/08	08/09	09/10	10/11	11/12
Departmental Management and Administration	Capacity and governance	33. Intervention Policy			X			Intervention Policy Framework

Special Studies – Chief Audit and Evaluation Executive

Issue	Study	Timeframe Fiscal Year
		07/08	08/09	09/10	10/11	11/12
Funding Instrument Study	An analysis of the effectiveness/appropriateness of funding arrangements available to the department for its programs/initiatives (Alternative Funding Arrangements, Flexible Transfer Payments, Contribution Agreements, Comprehensive Funding Arrangements) in the context of the government's/ department's broad policy objectives for its work with Aboriginal peoples.			X
Regional delivery/functional guidance study	An analysis of departmental processes for coordination and delivery of departmental program/initiatives via regional offices, including performance monitoring and progress reporting.		X
Accountabilities study	An assessment of the accountability provisions in departmental funding arrangements and whether satisfactory arrangements are being made for progress reporting by INAC to Parliament and Canadians and progress reporting by Aboriginal organizations to their communities.			X
Comprehensive community planning study	An analysis of the extent to which there is comprehensive community planning by Aboriginal organizations and an assessment of impact of comprehensive community planning projects now underway.				X
Departmental performance monitoring/results reporting review (to be done annually)	Each year, an assessment of performance information being collected by the department and its usefulness for departmental results reporting purposes will be prepared, with a particular emphasis on performance information for programs due to be evaluated in the following year.	X	X	X	X	X

Evaluation projects are led by Senior Evaluation Managers, with support from other staff of the department's Audit and Evaluation Sector and, in most cases, consulting resources. The project evaluation teams will seek input from other departmental officials, especially departmental officials with management responsibilities related to the program being evaluated, as needed.

The Senior Evaluation Managers will generally call on consulting companies to do fieldwork related to evaluations, e.g., literature reviews, interviews, case studies and expert focus groups.

Costs of consulting resources will vary, depending on project's scope and complexity. A preliminary per project estimate could range from $175,000 to $500,000. Project consulting costs will be borne by the responsible sector and Audit and Evaluation Sector. A detailed cost estimate will be developed by the Senior Evaluation Manager prior to its commencement, a part of the project terms of reference. Once Terms of Reference are approved by departmental senior management, sectors will be expected to arrange for the transfer of their share of project costs.

Costs of special projects that are not sector- or authority-specific will be borne by Audit and Evaluation Sector.

Consultations on the proposed strategic approach have taken place with Treasury Board's Centre of Excellence for Evaluation, which endorses it.

The plan was discussed with the senior management of all sectors in INAC to ensure that it provides for needed coverage and that the approach is sound.

Feedback on the plan was requested from, and discussions were held with, all Assistant Deputy Ministers and their management teams. Meetings were also held with a number of Directors General, and their suggestions were accommodated.

Indian and Northern Affairs Canada Evaluation Plan

The plan will be updated and tabled at the Audit and Evaluation Committee annually, to take account of evolving needs for coverage and strategic information. Because it is a rolling plan, annual update will also identify evaluation activities for the fifth year of the plan.

The updated plan will reflect work not completed as planned and carried over to another fiscal year.

Indian and Northern Affairs Canada Evaluation Plan

Evaluative Work by Year by Sector

2007-2008

Evaluation	Sector
Comprehensive Land Claims Impacts	Claims and Indian Government
Income Assistance, National Child Benefit Reinvestment	Socio-Economic Policy and Regional Operations
First Nations Water Management Strategy	Socio-Economic Policy and Regional Operations
Housing Policy	Socio-Economic Policy and Regional Operations
Gender-Based Analysis	Policy and Strategic Direction
Contaminated Sites	Lands and Trust Services
Community Economic/Aboriginal Business Development	Aboriginal Economic Development
Northern Economic Development and Governance	Northern Affairs
Office of the Federal Interlocutor	Office of the Federal Interlocutor
Departmental performance monitoring/results reporting review (to be done annually)	Special Study - Chief Audit and Evaluation Executive

2008-2009

Evaluation	Sector
Comprehensive Land Claims Impacts	Claims and Indian Government
Specific Claims	Claims and Indian Government
Self-Government Agreements	Claims and Indian Government
On-Reserve Housing	Socio-Economic Policy and Regional Operations
Northern Environment	Northern Affairs
Health Care and Counselling and Healthy Communities	Northern Affairs
Northern Economic Development and Governance	Northern Affairs
Representative Organizations and Consultation and Policy Development	Policy and Strategic Direction
Regional delivery/functional guidance study	Special Study - Chief Audit and Evaluation Executive
Departmental performance monitoring/results reporting review (to be done annually)	Special Study - Chief Audit and Evaluation Executive

2009-2010

Evaluation	Sector
Long term Capital Planning	Socio-Economic Policy and Regional Operations
Comprehensive Land Claims Impacts	Claims and Indian Government
Treaty Related Measures	Claims and Indian Government
Registration and Administration	Lands and Trust Services
Intervention Policy	Chief Financial Officer
Funding Instrument Study	Special Study - Chief Audit and Evaluation Executive
Accountabilities study	Special Study - Chief Audit and Evaluation Executive
Departmental performance monitoring/results reporting review	Special Study - Chief Audit and Evaluation Executive

2010-2011

Evaluation	Sector
Family Violence Prevention	Socio-Economic Policy and Regional Operations
Child Welfare	Socio-Economic Policy and Regional Operations
Elementary and Secondary Education	Socio-Economic Policy and Regional Operations
Post Secondary Education	Socio-Economic Policy and Regional Operations
First Nations Infrastructure Fund	Socio-Economic Policy and Regional Operations
Urban Aboriginal Strategy	Office of the Federal Interlocutor
Indian Government Support	Lands and Trust Services
Comprehensive community planning study	Special Study - Chief Audit and Evaluation Executive
Departmental performance monitoring/results reporting review	Special Study - Chief Audit and Evaluation Executive

2011-2012

Evaluation	Sector
ecoENERGY for Aboriginal and Northern Communities	Northern Affairs
First Nations Fiscal and Statistical Management	Lands and Trust Services
Emergency Management	Socio-Economic Policy and Regional Operations
Departmental performance monitoring/results reporting review	Special Study - Chief Audit and Evaluation Executive